Hybrid surgical navigation system and related methods

Abstract

The present disclosure describes a hybrid navigation system comprising two or more data acquisition apparatuses of different types that addresses limitations of current methods. In some embodiments, the two or more data acquisition apparatuses include at least one structured light three-dimensional scanner-based navigation system and at least one other navigation system (e.g., including but not limited to stereoscopic, ultrasound, electromagnetic, and / or intraoperative imaging). The hybrid approach improves real-time anatomical tracking, reduces the invasiveness of pin-based optical trackers, and maintains compatibility with established workflows. The principles of this disclosure extend across diverse surgical applications, offering enhanced accuracy, flexibility, and efficiency in both traditional and next-generation navigation environments.

Description

Hybrid Surgical Navigation System and Related MethodsCROSS-REFERENCES TO RELATED APPLICATIONS

[0001] The present application claims the benefit of priority from U.S. Provisional Application Serial No. 63 / 724,514, filed November 25, 2024 and entitled “Multimodal Surgical Navigation Systems”, and U.S. Provisional Application Serial No. 63 / 844,018, filed July 14, 2025 and entitled “3D Scanner”, the entire contents of which are hereby incorporated by reference into this disclosure as if set forth fully herein.FIELD

[0002] The present disclosure relates generally to surgical navigation, and more specifically to using intraoperative topological scan data of anatomic structure(s) acquired from structured light three-dimensional (3D) scanning without optical trackers or applied fiducials and interoperative topological scan data of surgical instruments and / or equipment acquired from stereoscopic 3D cameras with optical trackers to intraoperatively modify a pre-operative surgical plan.BACKGROUND

[0003] In modern surgical practice, particularly in orthopedic procedures like Total Knee Arthroplasty (TKA) and Hip Arthroplasty, precise alignment and tracking of anatomical structures are crucial for successful outcomes. Over the past two decades, surgical navigation systems have become standard tools for enhancing accuracy during these procedures. These systems typically rely on optical trackers and stereoscopic cameras (or “SSC”) to guide surgeons through intricate surgical steps, ensuring correct positioning of instruments and implants. The following disclosure generally describes methods currently used in the field of surgical navigation, for example a navigation system using optical markers and stereoscopic cameras. By way of example only, the methods discussed and / or disclosed herein are done so in the context of a TKA, however it should be understood that the systems and methods disclosed herein may be used in any scenario (surgical, medical, or otherwise) to generate a reconstructed 3D model of an object.

[0004] By way of example, an optical marker (OM) is a small reflective marker used in traditional surgical navigation systems to track the position of anatomical structures and tools in real-time via stereoscopic cameras. One common example is a reflective sphere, though multiple types of optical markers can be used depending on the system and application. Generally, OMs are easily segmented from other elements in an image. These markers retro-reflect infrared light emitted by stereoscopic cameras, making them highly detectable to cameras that are located close to the near infrared (NIR) source. OMs are useful for tracking anatomical landmarks, surgical instruments, and equipment during surgery, ensuring reliable visibility even in complex surgical environments.

[0005] OMs are often mechanically grouped into specific rigid geometric patterns or “constellations”, known as optical trackers (OTs). These unique arrangements allow the system to differentiate between various anatomical structures, instruments, or equipment being tracked. By way of example, OTs can be attached to patient anatomy (e.g., bones) near the surgical site, mounted on surgical instruments, (e.g., pointers or measurement tools), and / or fixed to active equipment (e.g., robotic arms, saws, or drills). The distinct geometries of OTs enable the system to identify what each OM represents within the surgical field.

[0006] Stereoscopic cameras are foundational to spatial tracking in surgical navigation systems. These cameras capture images from slightly different angles, similar to human binocular vision. By detecting the reflective OMs in the surgical field, the cameras calculate the 3D positions of all visible OTs.

[0007] Typically, the navigation system compares the relative positions of the detected OMs to the predefined geometric pattern of each OT. Once an OT is identified, it is assigned to its corresponding surgical tool, anatomical structure, or piece of equipment. The identified OTs are then registered in a common coordinate system, or space, providing a comprehensive 3D representation of the surgical field. The system can now track the position and orientation of all OTs in real time.

[0008] Once the positions of the OTs are known in the common coordinate system, they are transformed into a Surgical Coordinate System (SCS), which provides a fixed reference frame for the procedure. The SCS could be equivalent to camera space, CT space, tracker space, orbased on another tracked location, such as the base of a robot. By applying a 4x4 rigid transformation matrix 2 (see Fig. 1), any OT can be positioned within the SCS, ensuring consistency in tracking across the system. To accurately transform the object into the SCS, a sequence of transformations may be applied. The process involves transformations between, for example, an object’s local space, OT space, camera space, and a final SCS, which is a chosen common space for all relevant items for navigation.

[0009] For surgical navigation systems to be effective, they must track rigidly attached OTs in real-time, updating positions multiple times per second. This continuous updating enables precise monitoring of tool positions, implant alignment, and anatomical movement, ensuring accurate and timely adjustments throughout the surgery.

[0010] Despite their effectiveness, these systems present several challenges, in particular for scanning anatomic structure. The setup of optical trackers is often time-consuming and technically complex, requiring precise calibration and positioning of cameras and tracking equipment. Additionally, the physical nature of optical trackers introduces mechanical vulnerabilities — trackers can become dislodged, damaged, or obstructed during surgery (e.g., affecting line of sight between the camera and the OTs), leading to potential inaccuracies in tracking and necessitating recalibration. These issues add to overall procedural time, which may increase patient exposure to anesthesia and complicate workflow.

[0011] Moreover, the reliance on optical trackers and complex equipment adds significant costs and requires highly trained personnel to operate the systems. These challenges have limited the widespread adoption of such systems, particularly in smaller or resource-constrained healthcare facilities. As a result, there is a pressing need for more robust and efficient solutions that maintain or improve the accuracy of surgical navigation while reducing complexity and cost.

[0012] While stereoscopic cameras and optical trackers have become valuable tools for improving surgical precision, they come with several limitations. The reliance on physical optical trackers, complex setup, time-intensive calibration, and susceptibility to mechanical failures all point to the need for a more robust and flexible solution. Additionally, the combination of high financial costs, lengthy training requirements, and the complexity of operating these systems has made broader adoption across surgical specialties challenging.

[0013] The field of surgical navigation encompasses a variety of technologies, each with its strengths and limitations. By way of example, existing navigation systems may include or use stereoscopic cameras, electromagnetic (EM) navigation, ultrasound-based navigation, robotic platforms, and / or intraoperative imaging (CT, MRI, fluoroscopy, etc.).

[0014] For example, stereoscopic navigation systems use multiple stereoscopic cameras to track OTs in three-dimensional 3D space. The OTs can be attached to the patient and surgical tools / equipment. Stereoscopic navigation systems are advantageous in that they are well- established and widely used in clinical practice, and generally provide reliable tracking of instruments and implants. However, as previously mentioned, stereoscopic navigation systems can have undesirable limitations that may include requiring the use of invasive pins to attach optical markers to the patient, a lack of the ability to track anatomy directly, and line of sight issues as personnel move about the surgical suite.

[0015] By way of example, conventional navigation relies on trackers essentially as “proxies” for anatomy. Further, these trackers, for example pin-mounted tracker arrays in TKA, are placed at a distance from the anatomy, which leads to a magnification of angular errors by lever-arm effects. Other conventional approaches often rely on rigidity to establish constant relative coordinate systems, such as robot bases, or Mayfield-type clamps connected to the patient via cranial pins, or intraoperative CT scanners. Like TKA tracker pins that flex or drift in bone, these elements can experience mechanical instabilities like subtle “robot base shifts,”, and small amounts of shifts of “cranial pins” even when the head appears rigidly fixed. Such small errors in the proxy pose, especially angular errors, are magnified at the anatomy by lever-arm effects, where a slight rotational deviation at the proxy yields a larger translational deviation at the more distant surgical target. Mechanical play such as backlash, flex, or slippage at any point in the daisy-chain of navigation further increases this discrepancy. The system’s error model therefore treats each tracked proxy as a potential source of lever-arm amplification and explicitly accounts for navigation chains such as robot base-arm-tool or tracker-clamp-pins-skull when evaluating navigation uncertainty.

[0016] Accuracy issues are present in other technologies as well, which are not based on rigidity and distance. Electromagnetic (EM) navigation systems, for example, use magneticfields to track sensors embedded in instruments. Strengths of the EM navigations systems include pin-free tracking of instruments, effectiveness in soft tissue surgeries where optical markers are less practical. However, EM systems are limited in that they are susceptible to interference from metallic objects in the operating room, have a limited range of operation, and no direct registration of tracking of anatomy.

[0017] For example, with ultrasound-based navigation, ultrasound imaging is used to visualize and track anatomy during surgery. Advantages of this navigation technique can include non-invasiveness, real-time imaging, and a usefulness for soft tissue procedures. However, ultrasound-based navigation techniques often result in lower resolution output compared to CT / MRI-based methods, while requiring significant operator expertise to use the system, and require contact with the patient for the duration of its use.

[0018] By way of example, robotic surgical systems integrate various navigation technologies to enhance precision. They are surgical tool positioning systems that have an origin, and their motion can be tracked in real time and any instruments attached thereto can be assigned a transformation matrix (TM) based on the geometry of the instrument. The main advantage of robotics is that it enables automation and improved control during complex procedures. However, robotic surgical systems typically have a high cost and steep learning curve.

[0019] Intraoperative Imaging (e.g., CT, MRI, and / or fluoroscopy) provide detailed imaging and localization of anatomy during surgery for navigation. The main advantage of intraoperative imaging lies in its high accuracy and detailed visualization of the surgical site. However, there are significant drawbacks to using intraoperative imaging, including but not limited to the time and expense required to use the technology, and the increased radiation exposure (e.g., in the case of fluoroscopy) for not only the patient but for the operating room personnel as well. These devices also provide information one snapshot at a time and are not typically considered realtime imaging modalities.

[0020] In some embodiments, structured light-based 3D scanners (also referred to as “SLS scanners”) may be used to collect topological data of a surgical site, for example as described in U.S. Patent Application Pub. 2024 / 0102795 to Bernstein, et al. (“Bernstein ‘795”). By way of example, these structured light-based 3D scanners may also be referred to as “shadow-castingscanners”. In some embodiments, the SLS scanner utilizes a transmissive LCD screen and a near linear light source to project structured light patterns, capturing detailed 3D topological data of the surgical site. By way of example, key features and innovations of the system and method utilizing the SLS scanner include (but are not limited to) pin-free / OT-free surgical navigation, comprehensive functionality in surgical procedures such as TKA, image-based and imageless workflows, and dual scanner systems for redundancy and expanded view.

[0021] By way of example, traditional optical tracking systems (e.g., using stereoscopic cameras) rely on invasive pins to affix optical markers to the patient's hard tissues. These pins can cause discomfort and complications. In contrast, structured light-based scanning eliminates the need for optical markers and trackers entirely, offering a non-invasive alternative for surgical navigation, enabling a system using SLS scanners to perform all critical steps required for TKA (for example) without reliance on additional technologies.

[0022] While structured light-based navigation systems offer significant advantages, its adoption poses challenges for surgeons, hospitals, and device manufacturers due to its departure from traditional workflows. Most notably, stereoscopic navigation systems, which rely on OTs, are widely accepted in industry. Transitioning to pin-free structured light scanning requires retraining surgical teams and adapting workflows. Surgeons that are familiar with current systems may face a steep learning curve when adopting a SLS system and the methods associated therewith. Existing surgical instruments and implant designs are tailored to work with stereoscopic systems. A SLS system would necessitate modifications to these tools to achieve seamless integration.

[0023] While these challenges may be typical of those faced by market disruptors trying to convince the market to switch from established, conventional systems and methods, finding ways to mitigate the challenges may increase the speed at which SLS systems become the new standard of care in surgical navigation. Hospitals and manufacturers are often hesitant to invest in entirely new systems due to the costs and risks associated with replacing well-established technologies. Incremental changes that retain compatibility with existing systems are more likely to gain traction in clinical settings. Thus, a need exists to bridge the gap between theconventional systems and methods and the SLS systems to ease transition from the former to the latter.SUMMARY

[0024] The present disclosure addresses this need by introducing a hybrid navigation system which for example comprises a stereoscopic camera and a structured light scanner. For example, hospitals and companies that have invested heavily in surgical navigation systems may find it less expensive to introduce a second navigation system to make a hybrid system instead of replacing all equipment required to recognize all benefits of a new system.

[0025] This disclosure builds upon the innovations presented in Bernstein ‘795. More specifically, the present disclosure introduces advanced clinical applications for the previously described 3D scanning system. Bernstein ‘795 unveiled a groundbreaking 3D scanner whose capabilities were not previously available in surgical settings. The present disclosure expands on the utility of that technology, particularly emphasizing its application in orthopedic surgeries such as Total Knee Arthroplasty (TKA) and Robotic Assisted TKA (RA-TKA). By way of example only, Fig. 2 illustrates a sample general workflow of a prior art RA-TKA procedure.

[0026] The enhancements detailed herein expand upon the exemplary clinical uses disclosed in Bernstein ‘795, with the goal of illustrating how the technology can be highly beneficial and tailored in certain arthroplasty procedures. For example, by leveraging advanced scanning methodologies, the system reduces the need for optical trackers, streamlining setup and improving surgical precision. While this disclosure can be applied broadly across multiple surgical domains, particular emphasis is given to its use in orthopedic procedures such as TKA. The subsequent sections will outline the significant advancements this technology brings to the clinical environment, setting a new standard for precision and efficiency in surgical navigation and instrumentation. Moreover, although described herein in this context as a specific example, it should be understood that the system and methods disclosed herein may be used in any scenario (surgical, medical, or otherwise) to generate a reconstructed 3D model of an object, register it to a reference object in a different coordinate space, and track movement of the object in real space with respect to the reconstructed 3D model of the object within the coordinate space.

[0027] The present disclosure describes a 3D structured light scanner specifically designed for surgical applications, capable of collecting visible topological data from the surgical field in real time. This represents a new class of data that was previously unavailable during surgery. By capturing continuous, high-rate 3D topological data during the procedure, this scanner can replace traditional workflows that rely on optical trackers — without the associated costs, setup time, and limitations. Although described within the context of surgical applications as a specific example, the system and methods disclosed herein may be used in a wide variety of non-surgical applications to register and track an object within a coordinate space.

[0028] This disclosure expands on how this topological data can be applied in a surgical setting to perform various procedures. Although the current disclosure emphasizes applications in TKA, the underlying principles can be adapted for other surgeries with minimal modifications by those skilled in the art. While the 3D scanner described herein is the preferred method for collecting this data, the use of the scanner is not a requirement for this innovation, as other technology capable of capturing 3D surface data during surgery may be utilized to achieve similar results.

[0029] When a continuously updated 3D topological data of sufficient accuracy and rate of the surgical site is available, the data can be used for Continuous Anatomical Auto Tracking (CAAT). This innovation eliminates the need for introduced physical optical markers and reduces the complexity of setup, offering a more streamlined, cost-effective, and accurate approach to surgical navigation. CAAT allows for real-time registration and tracking of anatomical structures, instruments and equipment without using OMs or OTs or other fiducials, marking a significant advancement in surgical technology. CAAT promises greater precision, flexibility, and efficiency in the operating room, benefiting a wide range of surgical procedures.

[0030] In some embodiments, this disclosure describes systems and methods for real-time assessment of anatomical structures during surgical procedures. By leveraging topological data collection techniques, the disclosure enables accurate measurement of joint laxity and relative gap changes between bones or other anatomical structures without requiring optical tracking systems. The approach captures continuous, three-dimensional surface scans of relevant anatomical structures, such as bones, enabling precise calculation of joint gaps under variousapplied stresses without requiring attached optical tracking devices. It is particularly applicable to orthopedic surgeryjoint replacement, and other procedures requiring precise alignment and stability assessment, though its principles can be adapted to a variety of surgical and diagnostic applications.

[0031] In some embodiments, a method disclosed herein establishes a baseline “zero-stress” measurement and calculates relative changes in gap size under controlled stresses, such as varus or valgus forces. By determining the change in varus / valgus angles and using a known contact distance between key anatomical points, the system and method described herein calculates lateral and medial gaps with accuracy. The system’s real-time tracking continuously monitors the relative positions of the anatomical structures, capturing peak gap values to provide a comprehensive assessment of joint laxity.

[0032] Applicable to a range of surgical fields, the method described herein offers a versatile solution for measuring relative distances and angles between any two anatomical structures where topological data can be collected and registered to preoperative imaging. Examples include orthopedic procedures, such as knee and shoulder replacements, as well as spinal and hip surgeries.

[0033] In some embodiments, the method described herein provides a way to calculate the hip center of rotation dynamically and with high confidence, using only topological data captured intraoperatively. By leveraging the femur’s constrained rotational movement about the hip, this method enables precise, real-time calculation of the hip center without requiring an expanded scan field or external trackers. This approach not only simplifies the workflow but also enhances intraoperative decision-making by offering accurate anatomical reference points that were previously only estimable with additional equipment.

[0034] This disclosure involves real-time 3D topological scanning to calculate anatomical reference points, such as the hip center of rotation, during procedures. This technology combines aspects of computer vision, structured light 3D scanning, and least-squares fitting algorithms to enhance surgical alignment and implant positioning. Applications include TKA and other joint surgeries where precise alignment and intraoperative reference points are critical to patient outcomes.

[0035] The method described herein is for accurately determining the hip center of rotation in real-time during surgery using a 3D topological scanner (such as the scanner described herein and / or in Bernstein ‘795). By way of example, as the surgeon slowly moves the knee, the scanner captures a series of femoral origin points within its field of view. These points, transformed in camera space, approximate a spherical surface due to the femur’s constraint by the hip joint. The center of this sphere corresponds to the hip center of rotation, an essential anatomical reference for surgical alignment.

[0036] In some embodiments, to estimate this center, a least-squares sphere-fitting method may be applied to the collected data points, minimizing errors to generate the best-fit sphere. The method further validates the precision of this calculation through a confidence ellipsoid, generated by plotting the residual errors of each point, providing a probabilistic boundary that ensures the estimated center’s accuracy to a defined confidence level, typically 95%. This approach enables surgeons to achieve optimal implant alignment without additional tracking devices, streamlining the surgical workflow and improving patient outcomes.

[0037] In some embodiments, this disclosure introduces a method for identifying specific anatomical points that are beyond the scanner’s field of view without performing calculations on visible anatomy or using optical trackers. Rather than relying upon line-of-sight-dependent trackers, this approach leverages a pointer instrument with a distinct “finial” design (or other navigation feature). This finial, which may be part of the handle or positioned above the handle, remains scannable even as the pointer is extended to contact points beyond the field of view. By registering the finial’s position in 3D topological scans, the system can infer the location of anatomical points using the known geometry of the pointer.

[0038] While this method applies broadly to various orthopedic procedures, TKA and locating the ankle center are used as examples here to explain the principles. In TKA, for instance, determining the center of the ankle joint is crucial for alignment. Using the pointer to locate both the medial and lateral malleoli (for example), the system infers the ankle center by registering the finial’s position, providing an anatomical reference without requiring additional tracking devices.

[0039] Procedures like TKA require precise landmark identification, and this method addresses this need while streamlining the surgical setup by eliminating traditional tracking devices. However, this method can be broadly applied to any surgical procedure that requires precise anatomical point localization.

[0040] In TKA, accurate localization of the ankle relative to the tibia is critical for establishing mechanical axis alignment. Current navigation systems such as stereo tracking platforms rely on external cameras and arrays of markers to track instruments. The approach described herein leverages the optical 3D scanner already directed at the exposed tibia to simultaneously localize both the tibia and a specialized pointer. The pointer extends distally to touch the ankle, while proximally it carries a prominent structure optimized for robust localization, which for example may be a cluster of fiducial spheres, or otherwise be a unique finial that uniquely describes alignment and orientation in six degrees of freedom, such as a contiguous shape that is highly visible when illuminated by the wavelengths used by the scanner. Because the scanner captures both the tibial surface and the fiducials in the same frame, the position and orientation of the pointer tip relative to the tibia is established with high accuracy. This avoids additional hardware or reorientation of the scanner and provides a workflow- integrated method for ankle localization during TKA.

[0041] In some embodiments, this disclosure provides a method and device for localizing the ankle during TKA procedures by extending the capabilities of an optical 3D scanning system that is concurrently being used to register the tibia. By way of example, a rigid pointer is designed to contact the ankle at its distal tip, while its proximal end is equipped with a cluster of fiducial spheres (or finial or other navigation feature). The cluster may be optimized for the scanner’s strongest localization performance, positioned near the exposed tibia so that both the fiducials and tibial anatomy are scanned within the same field of view. The rigid body relationship between the fiducial cluster and the pointer tip my be determined through calibration, thereby allowing the ankle contact point to be precisely resolved.

[0042] In some embodiments, calibration of the pointer tip relative to the fiducials could be achieved through a dedicated attachment that accepts the pointer tip in a defined seating geometry. By scanning this attachment with the optical 3D system, the tip position is constrainedwith high certainty and registered into the same coordinate system as the fiducials. During surgery, once the pointer tip is placed on the ankle, the scanner captures both the tibial anatomy and fiducial cluster. Because the transformation between fiducials and pointer tip is known, the ankle position can be localized relative to the tibia.

[0043] The pointer and method disclosed herein therefore integrates tibial surface registration and ankle localization into a single, scanner-based workflow. It reduces equipment complexity and improves efficiency relative to existing systems.

[0044] In some embodiments, this disclosure provides a verification method that replaces traditional checkpoint and osteotomy verification methods by leveraging intraoperative topological data captured through a 3D scanner. By scanning the anatomical site in real time, this disclosure provides a method for precise, imageless verification of anatomical landmarks and resected surfaces without requiring additional trackers or specialized instruments. These verifications can be performed continuously throughout the procedure or as single, on-demand scans at specific stages, depending on the needs of the surgeon. This approach not only reduces setup complexity but also streamlines workflow by eliminating procedural delays associated with OT recalibration, offering a robust, tracker-free solution for accurate and reliable intraoperative verification.

[0045] In some embodiments, this disclosure introduces a method for real-time verification of anatomical structures and bone resections during surgery, leveraging topological data to replace conventional optical tracker-dependent systems. In traditional practices, checkpoints are used to verify OT stability, while OT-based instruments with planar surfaces are employed to confirm resection planes. This disclosure eliminates these requirements by providing continuous registration and tracking through topological data, eliminating the need for physical trackers and checkpoints.

[0046] By way of example, the method operates in two stages: first, topological data is collected to verify anatomical consistency, replacing checkpoint methods; second, post-resection scans are compared to a modified, idealized model of the anatomy to assess resection accuracy. These stages streamline intraoperative workflows by minimizing setup complexity and reducing interruptions caused by tracker recalibrations or alignment adjustments.

[0047] In some embodiments, the CAAT system disclosed herein has the ability to perform all critical steps required for TKA without reliance on additional technologies. By way of example, some of these steps include anatomy registration and tracking (e.g., capturing the patient’s anatomy as a 3D point cloud and registering it in order to bring pre-operative images into the unified Surgical Coordinate System), hip center determination (e g., establishing the hip center for alignment and surgical planning), gap balancing (e.g., ensuring proper soft tissue tension for optimal implant placement), and bone resection and checkpoint verification (e.g., confirming the accuracy of bone cuts and implant positioning without the need for specialized tools or installing checkpoints). These capabilities make the CAAT system disclosed herein a comprehensive solution for TKA procedures.

[0048] By way of example, the CAAT system supports both image-based and imageless modalities. For example, with CT / MRI-based workflows, preoperative imaging data can be utilized to enhance navigation precision. In cases where preoperative imaging is unavailable, the system can operate using real-time anatomical data captured intraoperatively (e.g., “imageless” referring to a lack of preoperative imaging data).

[0049] In some embodiments, the CAAT system may use two shadow-casting scanners simultaneously, for example in tracking of one shadow casting scanner with a second shadow casting scanner. By way of example, in this embodiment, a first wide field of view scanner can scan a second to localize it. The second scanner scans a narrower field of view, which localizes features based on anatomic features or fiducials.

[0050] In some embodiments, the systems and methods disclosed herein integrate two or more navigations systems to create a hybrid navigation system that addresses limitations of current methods. In some embodiments, the two or more navigation systems include at least one structured light navigation system (e.g., the CAAT system described herein) and at least one other navigation system (e.g., including but not limited to stereoscopic, ultrasound, electromagnetic, and / or intraoperative imaging). The hybrid approach improves real-time anatomical tracking, reduces the invasiveness of pin-based optical trackers, and maintains compatibility with established workflows. The principles of this disclosure extend across diversesurgical applications, offering enhanced accuracy, flexibility, and efficiency in both traditional and next-generation navigation environments.

[0051] This concept is presented by way of a specific example, a hybrid surgical navigation system that integrates stereoscopic imaging (e.g., the conventional technique) with structured light-based 3D scanning technologies (e.g., the CAAT system), and may include a robotic system for positioning a surgical tool once the robotic base location has been established in the hybrid navigation system. By combining the strengths of both modalities, the hybrid navigation system disclosed herein enhances surgical precision, reduces invasiveness, and streamlines workflows while maintaining compatibility with established practices. In some embodiments, other hybrid systems, comprising different types of navigation systems (e.g., including but not limited to stereoscopic, ultrasound, electromagnetic, and / or intraoperative imaging), are also discussed. For example, surgical precision is enhanced by the ability of the structured lightbased 3D scanner to acquire data that is so dense that the surface-to-surface registrations constrain them against each other in 6 degrees of freedom. More sparse methods (such as a pointer) that capture only a few hundred data points are not nearly as constrained in how they fit preoperative imaging (or even a model as in “imageless”).

[0052] In some embodiments, at its core, the hybrid navigation system disclosed herein uses a unified SCS to integrate data from different navigation systems, enabling real-time tracking of anatomy, instruments, and surgical tools / equipment. By way of example, transformation matrices (TMs) facilitate localization and alignment of objects across coordinate systems, for accurate navigation throughout procedures.

[0053] Building upon the structured light 3D scanning technology for surgical applications, which eliminates the need for invasive pins and optical trackers, the hybrid navigation system disclosed herein addresses potential challenges associated with transitioning surgeons to nextgeneration technologies (such as the CAAT system described herein) all at once. For example, in some embodiments, the hybrid system disclosed herein retains optical trackers of the conventional navigation systems for instrument / equipment tracking while introducing pin-free methods of the CAAT system for anatomy localization, tracking and verifications. This approach allows surgeons and manufacturers to adopt the benefits of structured light scanning ofthe CAAT system in a stepwise fashion without abandoning familiar workflows and expensive capital equipment used in conventional navigation systems.

[0054] In some embodiments, the hybrid navigation system disclosed herein is adaptable to various surgical applications, including orthopedic (e.g., knee, hip, and shoulder replacements), spinal, cranial, and soft tissue procedures. Moreover, the hybrid system can incorporate additional navigation technologies — such as electromagnetic (EM) systems, ultrasound, intraoperative imaging, and robotic platforms — to create tailored solutions for specific clinical needs.

[0055] For example, in cranial procedures in which the patient’s head is fixed using a Mayfield-type cranial clamp, the hybrid navigation system may recognize that the pressure-based fixation may shift throughout the procedure. In some embodiments, the hybrid configuration supports repeated surface scanning of the head, face, exposed cranial bone, and any trackers attached to the Mayfield-type frame at baseline after clamp fixation, at key procedural milestones such as after draping or skull opening, and on demand whenever drift is suspected. These successive scans may be rigidly or quasi-rigidly registered to the preoperative CT-based cranial model, and deviations between the current surface scan and the original pinned-head reference may be used to detect relative motion between the skull and the clamp pins. This relative motion may be used to calculate a correcting transform to correct the navigation. When such motion exceeds configurable thresholds, the hybrid navigation system may alert the surgeon and provide the optional correction, and, upon approval, update the transform chain so that guidance remains accurate relative to the cranial anatomy, rather than implicitly trusting the clamp and pins to be perfectly rigid over time. In some embodiments, the head frame may be removed altogether, as its function is to immobilize the patient relative to a tracker, primarily.

[0056] In some embodiments, the hybrid surgical navigation system disclosed herein introduces a suite of innovations designed to enhance surgical precision, streamline workflows, and improve ease of adoption. These innovations improve the hybrid navigation system's effectiveness while maintaining compatibility with existing technologies and workflows.

[0057] In some embodiments, the hybrid surgical navigation system of the present disclosure includes pin-free navigation. By way of example, the hybrid system enables real-timeanatomical tracking using the structured light scanner of the CAAT system without requiring invasive pins or optical markers attached to the patient. This reduces patient discomfort, minimizes workflow disruption, and eliminates complications associated with traditional pinbased navigation.

[0058] In some embodiments, the hybrid surgical navigation system of the present disclosure includes compatibility with traditional navigation systems. By way of example, the hybrid system integrates stereoscopic imaging for instrument tracking for minimal disruption to existing workflows. By retaining compatibility with traditional navigation systems, it reduces the learning curve for surgeons and the need for extensive equipment overhauls.

[0059] In some embodiments, the hybrid surgical navigation system of the present disclosure includes real-time tracking with time-stamped data. By way of example, time-stamped transformation matrices (TMs) enable precise, synchronized updates across navigation systems operating at the same or different registration frequencies for continuous real-time tracking of anatomy, instruments, and equipment throughout surgical procedures.

[0060] In some embodiments, the hybrid surgical navigation system of the present disclosure includes interference mitigation. By way of example, the hybrid navigation system may employ non-overlapping spectra, multiplexing, and active filtering to prevent cross-system interference. By operating in separate regions of the electromagnetic spectrum and coordinating system activity, the hybrid navigation system ensures functionality even in challenging environments.

[0061] In some embodiments, the hybrid surgical navigation system of the present disclosure includes the ability to daisy-chain registrations for integration across navigation system components. By way of example, the hybrid navigation system supports the daisy-chaining of transformation matrices, enabling precise spatial alignment across navigation systems, even when their fields of view are non-overlapping. In some embodiments, the daisy chaining may include various transformation matrices and / or the inverse transformation matrix as needed. By way of example, this allows the daisy chaining to go any direction needed regardless of the direction of the calculated transformation matrices.

[0062] In some embodiments, the hybrid surgical navigation system of the present disclosure includes the ability to calibrate shared fiducials. For example, both the structured light scanner (SLS) and stereoscopic camera system (SSC) can scan and register shared fiducials, such as calibration spheres or markers.

[0063] In some embodiments, the hybrid surgical navigation system of the present disclosure includes the ability to transfer transformation matrices. For example, transformation matrices may be exchanged between navigation systems to align their respective coordinate spaces within the unified SCS for consistent anatomical tracking and tool localization across systems.

[0064] In some embodiments, the hybrid navigation system disclosed herein improves robustness by relocating the effective sensing locus from distant proxies like trackers to the anatomy itself via 3D scanning. By acquiring three-dimensional surface data directly at or near the anatomical region of interest — such as joint surfaces, cranial bone, or even sinus cavities in the case of an endoscopic embodiment — the system shortens the lever arm between sensing and action and reduces sensitivity to small angular errors.

[0065] In some embodiments, repeated scans can provide direct measurements of the actual anatomical pose relative to preoperative images and plans, detecting discrepancies between proxy-based pose estimates and anatomy-based pose estimates. When these discrepancies exceed configurable thresholds, transforms can be updated or re-registration is prompted to restore accuracy. Alternatively, transforms may be updated continuously as in a tracking mode. In this way, hybrid navigation using a 3D scanner that can “see” the anatomy not only introduces realtime anatomic tracking, but also reduces the role of lever-arm from its large effects from distant mechanical proxies to smaller effects by measuring the immediate vicinity of the target anatomy.

[0066] In some embodiments, the hybrid surgical navigation system of the present disclosure includes hybrid OTs. By way of example hybrid OTs combine traditional reflective optical markers with unique geometric features detectable by structured light scanners. These advanced trackers enable dual-system recognition, bridge gaps in non-overlapping fields of view, and enhance flexibility in diverse surgical environments. In some embodiments, the hybrid surgical navigation system may be used with existing OTs engineered for stereoscopic systems.

[0067] In some embodiments, in addition to conventional tracked marker arrays, the hybrid navigation systems may support fiducial geometries explicitly optimized for the surface scanner. These fiducials may be non-spherical and non-canonical in shape, chosen to maximize visibility and distinguishability in the scanner’s field of view, minimize occlusion in crowded operating rooms, and improve pose-estimation accuracy for a given geometry and material. Modular fiducial blocks based on these shapes can be rigidly attached to CT gantry housings or support structures, robot bases and arms, ultrasound probes, endoscopes, the base of needles, and other tracked instruments. This approach enables scanner-primary navigation configurations and simplifies calibration between multiple devices in a hybrid system by giving the scanner well- defined, scanner-optimized reference structures on key hardware elements.

[0068] In some embodiments, the hybrid surgical navigation system of the present disclosure includes the ability to maintain consistency in non-overlapping fields of view. By way of example only, shared fiducials or hybrid OTs may bridge spatial relationships between navigation systems, allowing accurate tracking even when only one system can view the reference markers. For example, by having stand-off use of a conventional relatively long-range stereoscopic camera to “see” the structured light 3D scanner (e.g., by placing an OT on the structured light 3D scanner), while the structured light 3D scanner is trained toward a surgical cavity, the navigation visibility of the anatomy is extended “around a corner” to a location not in the field of view of the stereoscopic camera. In some embodiments, for example, after registration of anatomy via stitching, the 3D scanner could be moved away from an OT that is still in view by the stereoscopic camera, but so long as the OT is stationary with respect to the anatomy being viewed by the scanner, the position of the object attached to the OT is known. In some embodiments, for example, the OT (e.g., spheres) on a tool may be visible by the stereoscopic camera, and not by the structured light 3D scanner, but another portion of the tool (e.g., tip) is visible to the structured light 3D scanner, so the position of the tool is known.

[0069] In some embodiments, the hybrid surgical navigation system of the present disclosure enables real-time synchronization. For example, time-stamped transformation matrices synchronize spatial and temporal data across systems, minimizing drift and ensuring robust localization.

[0070] In some embodiments, the hybrid surgical navigation system of the present disclosure enables workflow flexibility and scalability. For example, the hybrid system supports both traditional and imageless workflows, enabling surgeons to adapt the system to their preferred approach. It can incorporate additional technologies, such as robotic platforms or intraoperative imaging, to create tailored solutions for specific clinical needs.

[0071] In some embodiments, the hybrid surgical navigation system of the present disclosure enables an incremental adoption pathway from the traditional navigation system to the innovative and groundbreaking CAAT system. For example, by combining familiar technologies (e.g., stereoscopic imaging) with next-generation methods (e.g., structured light scanning), the hybrid system offers a stepwise adoption pathway. This minimizes the risk, cost, and learning curve associated with transitioning to advanced surgical navigation.

[0072] These key innovations position the hybrid navigation system disclosed herein as a transformative tool for improving surgical precision and efficiency while ensuring practical adoption across clinical settings.

[0073] As additional description to the embodiments described below, the present disclosure describes the following embodiments.

[0074] Embodiment l is a hybrid surgical navigation system configured for registering and tracking one or more anatomic structures of a patient and one or more artificial objects within a common coordinate system in an operating room environment, comprising: a first data acquisition apparatus comprising a three-dimensional scanning device having an image capture component and a first optical tracker rigidly coupled to the first data acquisition apparatus; a second data acquisition apparatus comprising a stereoscopic camera configured to detect optical trackers; a computer system including a memory and at least one processor; and computer readable media embodied in a non-transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a three-dimensional data representation of a surface of an anatomic structure of the patient using a first dataset, the first dataset derived from one or more images captured by the first data acquisition apparatus without optical trackers or applied fiducials, the first dataset being expressed in a first coordinate system associated with the first data acquisition apparatus;compute a first transformation matrix representing a position and orientation of the first data acquisition apparatus in a second coordinate system associated with the second data acquisition apparatus, the first transformation matrix derived from a second dataset comprising one or more stereoscopic images of the first optical tracker captured by the second data acquisition apparatus; compute a second transformation matrix representing a position and orientation of an artificial object having an attached optical tracker in the second coordinate system, the second transformation matrix derived from a third dataset comprising one or more stereoscopic images of the attached optical tracker captured by the second data acquisition apparatus; derive a third transformation matrix by multiplying an inverse of the first transformation matrix with the second transformation matrix; and apply the third transformation matrix to transform positional information of the artificial object from the second coordinate system to the first coordinate system.

[0075] Embodiment 2 is the hybrid surgical navigation system of embodiment 1, wherein the artificial object is a surgical instrument, an article of equipment located within the operating room environment, or an optical tracker.

[0076] Embodiment 3 is the hybrid surgical navigation system of embodiments 1 or 2, wherein the three-dimensional scanning device is a structured light three-dimensional scanner.

[0077] Embodiment 4 is the hybrid surgical navigation system of any of embodiments 1 through 3, wherein the structured light three-dimensional scanner includes a light source and a pattern generator.

[0078] Embodiment 5 is the hybrid surgical navigation system of any of embodiments 1 through 4, wherein the pattern generator is a liquid crystal matrix.

[0079] Embodiment 6 is the hybrid surgical navigation system of any of embodiments 1 through 5, wherein the computer readable media includes further instructions that, when executed by the one or more processors, cause the computer system to: receive a fourth dataset representing a three-dimensional anatomical model of the same anatomical structure represented in the first dataset, the fourth dataset being derived from preoperative or intraoperative medical imaging and expressed in a virtual coordinate system; generate a virtual surgical plan defined inthe virtual coordinate system using the fourth dataset, the virtual surgical plan comprising one or more surgical targets or surgical cut planes; register the first dataset to the fourth dataset to compute a fourth transformation matrix representing a transformation from the virtual coordinate system to the first coordinate system; and apply the fourth transformation matrix to transform the virtual surgical plan from the virtual coordinate system to the first coordinate system.

[0080] Embodiment 7 is the hybrid surgical navigation system of any of embodiments 1 through 6, wherein the computer readable media includes further instructions that, when executed by the one or more processors, cause the computer system to compute a deviation between a position and orientation of the artificial object expressed in the first coordinate system and one or more transformed surgical cut planes expressed in the first coordinate system.

[0081] Embodiment 8 is the hybrid surgical navigation system of any of embodiments 1 through 7, wherein the computer readable media includes further instructions that, when executed by the one or more processors, cause the computer system to provide a control signal to a robotic actuator configured to reduce the deviation and assist a surgeon in performing a surgical procedure.

[0082] Embodiment 9 is the hybrid surgical navigation system of any of embodiments 1 through 8, wherein the artificial object is attached to the robotic actuator.

[0083] Embodiment 10 is the hybrid surgical navigation system of any of embodiments 1 through 9, wherein the medical imaging comprises computed tomography (CT) or magnetic resonance imaging (MRI) scan data.

[0084] Embodiment 11 is the hybrid surgical navigation system of any of embodiments 1 through 10, wherein the one or more images are captured by the image capture device using infrared light or near infrared light.

[0085] Embodiment 12 is the hybrid surgical navigation system of any of embodiments 1 through 11, wherein the one or more stereoscopic images are captured by the stereoscopic camera using infrared light or near infrared light.

[0086] Embodiment 13 is a hybrid surgical navigation system configured for registering and tracking one or more anatomic structures of a patient and one or more artificial objects within a common coordinate system in an operating room environment, comprising: a first data acquisition apparatus comprising a three-dimensional scanning device having an image capture component and a first optical tracker rigidly coupled to the first data acquisition apparatus; a second data acquisition apparatus comprising a stereoscopic camera configured to detect optical trackers; a computer system including a memory and at least one processor; and computer readable media embodied in a non-transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a three-dimensional data representation of a surface of an anatomic structure of the patient using a first dataset, the first dataset derived from one or more images captured by the first data acquisition apparatus without optical trackers or applied fiducials, the first dataset being expressed in a first coordinate system associated with the first data acquisition apparatus; timestamp the first dataset with a first time of acquisition, the first time of acquisition being the time acquisition of the first dataset; compute a first transformation matrix representing a position and orientation of the first data acquisition apparatus in a second coordinate system associated with the second data acquisition apparatus, the first transformation matrix derived from a second dataset comprising one or more stereoscopic images of the first optical tracker captured by the second data acquisition apparatus; time-stamp the first transformation matrix with a second time of acquisition, the second time of acquisition being the time acquisition of the second dataset; compute a second transformation matrix representing a position and orientation of an artificial object having an attached optical tracker in the second coordinate system, the second transformation matrix derived from a third dataset comprising one or more stereoscopic images of the attached optical tracker captured by the second data acquisition apparatus; time-stamp the second transformation matrix with a third time of acquisition, the third time of acquisition being the time acquisition of the third dataset; derive a third transformation matrix by multiplying an inverse of the first time-stamped transformation matrix with the second time-stamped transformation matrix; and apply the third transformation matrix to transform positional information of the artificial object from the second coordinate system to the first coordinate system.

[0087] Embodiment 14 is the hybrid surgical navigation system of embodiment 13, wherein the artificial object is a surgical instrument, an article of equipment located within the operating room environment, or an optical tracker.

[0088] Embodiment 15 is the hybrid surgical navigation system of embodiments 13 or 14, wherein the three-dimensional scanning device is a structured light three-dimensional scanner.

[0089] Embodiment 16 is the hybrid surgical navigation system of any of embodiments 13 through 15, wherein the structured light three-dimensional scanner includes a light source and a pattern generator.

[0090] Embodiment 17 is the hybrid surgical navigation system of any of embodiments 13 through 16, wherein the pattern generator is a liquid crystal matrix.

[0091] Embodiment 18 is the hybrid surgical navigation system of any of embodiments 13 through 17, wherein the computer readable media includes further instructions that, when executed by the one or more processors, cause the computer system to: receive a fourth dataset representing a three-dimensional anatomical model of the same anatomical structure represented in the first dataset, the fourth dataset being derived from preoperative or intraoperative medical imaging and expressed in a virtual coordinate system; generate a virtual surgical plan defined in the virtual coordinate system using the fourth dataset, the virtual surgical plan comprising one or more surgical targets or surgical cut planes; register the first dataset to the fourth dataset to compute a fourth transformation matrix representing a transformation from the virtual coordinate system to the first coordinate system; and apply the fourth transformation matrix to transform the virtual surgical plan from the virtual coordinate system to the first coordinate system.

[0092] Embodiment 19 is the hybrid surgical navigation system of any of embodiments 13 through 18, wherein the computer readable media includes further instructions that, when executed by the one or more processors, cause the computer system to compute a deviation between a position and orientation of the artificial object expressed in the first coordinate system and one or more transformed surgical cut planes expressed in the first coordinate system.

[0093] Embodiment 20 is the hybrid surgical navigation system of any of embodiments 13 through 19, wherein the computer readable media includes further instructions that, whenexecuted by the one or more processors, cause the computer system to provide a control signal to a robotic actuator configured to reduce the deviation and assist a surgeon in performing a surgical procedure.

[0094] Embodiment 21 is the hybrid surgical navigation system of any of embodiments 13 through 20, wherein the artificial object is attached to the robotic actuator.

[0095] Embodiment 22 is the hybrid surgical navigation system of any of embodiments 13 through 21, wherein the medical imaging comprises computed tomography (CT) or magnetic resonance imaging (MRI) scan data.

[0096] Embodiment 23 is the hybrid surgical navigation system of any of embodiments 13 through 22, wherein the one or more images are captured by the image capture device using infrared light or near infrared light.

[0097] Embodiment 24 is the hybrid surgical navigation system of any of embodiments 13 through 23, wherein the one or more stereoscopic images are captured by the stereoscopic camera using infrared light or near infrared light.

[0098] Embodiment 25 is the hybrid surgical navigation system of any of embodiments 13 through 24, wherein the computer readable media comprises a further set of instructions that, when executed by the one or more processors, cause the computer system to: use interpolation to generate one or more intermediate transformation matrices to align the time-stamped first dataset, the time-stamped first transformation matrix, and the time-stamped second transformation matrix to align the first and third datasets within the first coordinate system.

[0099] Embodiment 26 is a hybrid surgical navigation system configured for registering and tracking one or more anatomic structures of a patient and one or more artificial objects within a common coordinate system in an operating room environment, comprising: a first data acquisition apparatus comprising a three-dimensional scanning device having an image capture component configured to reconstruct geometric information of objects within its field of view; a second data acquisition apparatus comprising a stereoscopic camera system configured to detect optical trackers; a hybrid reference object comprising an optical tracking array and a distinct geometric structure configured for detection by both the first data acquisition apparatus and thesecond data acquisition apparatus; a computer system including a memory and at least one processor; and computer readable media embodied in a non-transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a three-dimensional data representation of a surface of an anatomic structure of the patient using a first dataset, the first dataset derived from one or more images captured by the first data acquisition apparatus without optical trackers or applied fiducials, the first dataset being expressed in a first coordinate system associated with the first data acquisition apparatus; compute a first transformation matrix representing a position and orientation of the hybrid reference object in the first coordinate system based on a second dataset comprising one or more images of the hybrid reference object captured by the first data acquisition apparatus; compute a second transformation matrix representing a position and orientation of the hybrid reference object in a second coordinate system associated with the stereoscopic camera system based on a third dataset comprising one or more stereoscopic images of the hybrid reference object captured by the second data acquisition apparatus; compute a third transformation matrix representing a position and orientation of an artificial object having an attached optical tracker in the second coordinate system based on a fourth dataset comprising one or more stereoscopic images of the artificial object captured by the second data acquisition apparatus; and derive a fourth transformation matrix expressing the artificial object in the first coordinate system by multiplying the first transformation matrix by an inverse of the second transformation matrix and by the third transformation matrix, and apply the fourth transformation matrix to transform positional information of the artificial object from the second coordinate system to the first coordinate system.

[0100] Embodiment 27 is the hybrid surgical navigation system of embodiment 26, wherein the artificial object is a surgical instrument, an article of equipment located within the operating room environment, or an optical tracker.

[0101] Embodiment 28 is the hybrid surgical navigation system of embodiments 26 or 27, wherein the three-dimensional scanning device is a structured light three-dimensional scanner.

[0102] Embodiment 29 is the hybrid surgical navigation system of any of embodiments 26 through 28, wherein the structured light three-dimensional scanner includes a light source and a pattern generator.

[0103] Embodiment 30 is the hybrid surgical navigation system of any of embodiments 26 through 29, wherein the pattern generator is a liquid crystal matrix.

[0104] Embodiment 31 is the hybrid surgical navigation system of any of embodiments 26 through 30, wherein the computer readable media includes further instructions that, when executed by the one or more processors, cause the computer system to: receive a fifth dataset representing a three-dimensional anatomical model of the same anatomical structure represented in the first dataset, the fifth dataset being derived from preoperative or intraoperative medical imaging and expressed in a virtual coordinate system; generate a virtual surgical plan defined in the virtual coordinate system using the fifth dataset, the virtual surgical plan comprising one or more surgical targets or surgical cut planes; register the first dataset to the fifth dataset to compute a fifth transformation matrix representing a transformation from the virtual coordinate system to the first coordinate system; and apply the fifth transformation matrix to transform the virtual surgical plan from the virtual coordinate system to the first coordinate system.

[0105] Embodiment 32 is the hybrid surgical navigation system of any of embodiments 26 through 31, wherein the computer readable media includes further instructions that, when executed by the one or more processors, cause the computer system to: compute a deviation between positional information of the artificial object expressed in the first coordinate system and one or more surgical cut planes of the transformed virtual surgical plan expressed in the first coordinate system.

[0106] Embodiment 33 is the hybrid surgical navigation system of any of embodiments 26 through 32, wherein the computer readable media includes further instructions that, when executed by the one or more processors, cause the computer system to provide a control signal to a robotic actuator configured to move at least one of the artificial object or a surgical guide in a manner that reduces the deviation and assists a surgeon in performing a surgical procedure.

[0107] Embodiment 34 is the hybrid surgical navigation system of any of embodiments 26 through 33, wherein the artificial object is attached to the robotic actuator.

[0108] Embodiment 35 is the hybrid surgical navigation system of any of embodiments 26 through 34, wherein the medical imaging comprises computed tomography (CT) or magnetic resonance imaging (MRI) scan data.

[0109] Embodiment 36 is the hybrid surgical navigation system of any of embodiments 26 through 35, wherein the one or more images are captured by the image capture device using infrared light or near infrared light.

[0110] Embodiment 37 is the hybrid surgical navigation system of any of embodiments 26 through 36, wherein the one or more stereoscopic images are captured by the stereoscopic camera using infrared light or near infrared light.BRIEF DESCRIPTION OF THE DRAWINGS

[0111] Many advantages of the present disclosure will be apparent to those skilled in the art with a reading of this specification in conjunction with the attached drawings, wherein like reference numerals are applied to like elements and wherein:

[0112] Fig. 1 illustrates an example of a rigid transformation matrix for transferring a 3D data set from one coordinate system to another;

[0113] Fig. 2 is a flowchart depicting an example of a typical prior art workflow for a robotically assisted total knee arthroplasty;

[0114] Fig. 3 is a block diagram illustrating essential features of a continuous anatomical automatic tracking (CAAT) system of the present disclosure, according to some embodiments;

[0115] Fig. 4 is a perspective view of an example of a surgical environment utilizing the CAAT system of Fig. 3, according to some embodiments;

[0116] Fig. 5 is a perspective view of an example of a 3D scanner forming part of the CAAT system of the Fig. 3, according to some embodiments;

[0117] Fig. 6 is a perspective view of the 3D scanner of Fig. 5 with the cover removed, according to some e embodiments;

[0118] Fig. 7 is a plan view of the 3D scanner of Fig. 6, according to some embodiments;

[0119] Fig. 8 is another perspective view of the 3D scanner of Fig. 6, according to some embodiments;

[0120] Fig. 9 is an perspective view of an example of a rigid scaffolding forming part of the 3D scanner of Fig. 5, according to some embodiments;

[0121] Fig. 10 is an exploded view of an example of a light source assembly forming part of the 3D scanner of Fig. 5, according to some embodiments;

[0122] Figs. 11-13 are perspective, bottom plan, and side plan views, respectively, of an example of a pattern generator forming part of the 3D scanner of Fig. 5, according to some embodiments;

[0123] Fig. 14 is a partial exploded view of an example of a thermal management system forming part of the 3D scanner of Fig. 5, according to some embodiments;

[0124] Fig. 15 is a block diagram depicting an example of a data acquisition process using the CAAT system of Fig. 3, according to some embodiments;

[0125] Fig. 16 is an example of reconstructed point cloud of an exposed knee, simulated with sawbones and using a visible light source generated using the CAAT system of Fig. 3, according to some embodiments;

[0126] Figs. 17A-B depicts a typical prior art stereoscopic camera field of view (on the left) and what the cameras actually capture (on the right), according to some embodiments;

[0127] Figs. 18A-B depicts an image and 3D point cloud from a 3D scanner forming part of the CAAT system of Fig. 3, using infrared light instead of visible light, according to some embodiments;

[0128] Fig. 19 depicts a 3D point cloud from a 3D scanner forming part of the CAAT system of Fig. 3, using infrared light instead of visible light and shown at an oblique angle, which illustrates in particular the 3D nature of the regeneration, according to some embodiments;

[0129] Fig. 20 depicts the 3D point cloud of Fig. 60, with some anatomical structures segmented for easy identification, according to some embodiments;

[0130] Fig. 21 is a block diagram depicting an example of a method of registration of anatomical structures and / or surgical instruments using the CAAT system of Fig. 3, according to some embodiments;

[0131] Fig. 22 illustrates three data sets registered into the same coordinate space using the CAAT system of Fig. 3, particularly CT data from the femur and tibia registered to a 3D topological scan, according to some embodiments;

[0132] Fig. 23 is a block diagram depicting an example of a method of calculating a hip center of rotation using the CAAT system of Fig. 3, according to some embodiments;

[0133] Fig. 24 is a block diagram depicting an example of a method of assessing gaps between femoral condyles (e.g., medial and lateral) and a tibial plateau using the CAAT system of Fig. 3, according to some embodiments;

[0134] Fig. 25 is a block diagram illustrating several steps of an example method of calculating joint gaps based on changes in alignment between the femur and tibia under applied stress using the CAAT system of Fig. 3, according to some embodiments;

[0135] Fig. 26 is a block diagram depicting an example of a user interface of the CAAT system of Fig. 3, illustrating in particular a baseline orientation of a knee joint, according to some embodiments;

[0136] Fig. 27 is a block diagram depicting an example of a user interface of the CAAT system of Fig. 3, illustrating in particular a knee joint under applied varus stress, according to some embodiments;

[0137] Fig. 28 is a block diagram depicting an example of a user interface of the CAAT system of Fig. 3, illustrating in particular a knee joint under applied valgus stress, according to some embodiments;

[0138] Fig. 29 is a block diagram depicting an example of a method of tracking movable anatomy and a movable surgical instrument configured to affect the movable anatomy using the CAAT system of Fig. 3, according to some embodiments;

[0139] Fig. 30A-B is a block diagram depicting an example of a method of anatomy tracking with robot-controlled instrument in the context of a Total Knee Arthroplasty (TKA) using the CAAT system of Fig. 3, according to some embodiments;

[0140] Fig. 31 is a block diagram depicting an example robotically assisted TKA workflow with pre-operative imaging using the CAAT system of Fig. 3, according to some embodiments;

[0141] Fig. 32 is a block diagram depicting an example robotically assisted TKA workflow without pre-operative imaging using the CAAT system of Fig. 3, according to some embodiments;

[0142] Fig. 33 is a perspective view of an example of a pointer instrument configured for use with the CAAT system of Fig. 3, according to some embodiments;

[0143] Fig. 34 is a block diagram depicting several steps in an example method for identifying specific anatomical points that are beyond the scanner’s field of view without performing calculations on visible anatomy or using optical trackers using the CAAT system of Fig. 3, according to some embodiments;

[0144] Fig. 35 is a block diagram depicting several steps of an example method for real-time verification of the positioning and orientation of an anatomical structures using the CAAT system of Fig. 3, according to some embodiments;

[0145] Fig. 36 is a block diagram illustrating several steps of an example method for real-time verification of osteotomies and bone resections using the CAAT system of Fig. 3, according to some embodiments;

[0146] Fig. 37 is a block diagram illustrating essential features of a hybrid surgical navigation system of the present disclosure, according to some embodiments;

[0147] Fig. 38 is a plan view of an example of a pointer instrument configured for use with the hybrid surgical navigation system of Fig. 37, according to some embodiments;

[0148] Fig. 39 is a block diagram illustrating several steps of an example method of locating an anatomical landmark in the surgical coordinate system using the pointer instrument of Fig. 38, according to some embodiments;

[0149] Fig. 40 is a perspective view of an example of a surgical environment utilizing the hybrid surgical navigation system of Fig. 37, according to some embodiments;

[0150] Fig. 41 is a perspective view of another example of a surgical environment utilizing the hybrid surgical navigation system of Fig. 37, according to some embodiments;

[0151] Fig. 42 is a block diagram illustrating several steps of a first example method of aligning data from a structured light 3D scanner and a stereoscopic camera within a unified surgical coordinate system, according to some embodiments;

[0152] Fig. 43 is is a perspective view of another example of a surgical environment utilizing the hybrid surgical navigation system of Fig. 37, according to some embodiments;

[0153] Fig. 44 is a block diagram illustrating several steps of an example method of a second example method of aligning data from a structured light 3D scanner and a stereoscopic camera within a unified surgical coordinate system, according to some embodiments;

[0154] Fig. 45 is a block diagram illustrating several steps of an example method for aligning structured light 3D scanner derived data with stereoscopic camera derived data for synchronized real-time tracking using the hybrid navigation system of Fig. 37, according to some embodiments;

[0155] Fig. 46 is a block diagram illustrating several steps of an example workflow for communication between the structured light 3D scanner and stereoscopic camera system forming part of the hybrid navigation system of Fig. 37, according to some embodiments;

[0156] Figs. 47 is a plan view of an example of a hybrid optical tracker configured for use with the hybrid navigation system of Fig. 37, according to some embodiments;

[0157] Figs. 48 is a plan view of another example of a hybrid optical tracker configured for use with the hybrid navigation system of Fig. 37, according to some embodiments; and

[0158] Figs. 49 and 50 are block diagrams illustrating example computer systems with which any of the devices or systems described herein may be implemented, according to some embodiments.DETAILED DESCRIPTION OF A PREFERRED EMBODIMENT

[0159] Illustrative embodiments of the disclosure are described below. In the interest of clarity, not all features of an actual implementation are described in this specification. It will of course be appreciated that in the development of any such actual embodiment, numerous implementation-specific decisions must be made to achieve the developers’ specific goals, such as compliance with system-related and business-related constraints, which will vary from one implementation to another. Moreover, it will be appreciated that such a development effort might be complex and time-consuming but would nevertheless be a routine undertaking for those of ordinary skill in the art having the benefit of this disclosure. The surgical navigation system and related methods disclosed herein boasts a variety of inventive features and components that warrant patent protection, both individually and in combination.

[0160] By way of example, this disclosure introduces a novel system and method for registering and tracking anatomical structures during surgical procedures by leveraging continuously updated 3D topological data of the surgical field. Unlike traditional systems that rely on physical optical markers (OMs) and Optical Tracker Constellations (OTCs), the system and method disclosed herein utilizes the data captured by a 3D scanner to perform Continuous Anatomic Auto Tracking (CAAT). Hereinafter, the system disclosed herein may be referred to as “CAAT system 10”. While shown and described herein within the specific context of a surgical application (specifically a TKA procedure), the CAAT system 10 including the 3D scanner 12 may be used in any scenario (surgical, medical, or otherwise) to generate a reconstructed 3D model of an object.

[0161] Referring to Fig. 3, and by way of example, the CAAT system 10 disclosed herein may feature a 3D scanner 12, hub 14, user interface 16, and distributed computation 18. By way of example, the CAAT system 10 also includes a software program 20 embodied in a non- transitory computer readable storage medium (e.g., the hub 14) and comprising a set of instructions configured to enable the computer to perform the various tasks described herein. In some embodiments, one or more peripheral computing devices 21 may be associated with the CAAT system 10, for example to integrate use and control of certain medical and surgical equipment that may be used in robotics-driven procedures. Such peripheral computing device(s) 21 may be configured to be controlled by the hub 14 and / or user interface 16.

[0162] In some embodiments, the 3D scanner 12 continuously captures topological data of one or more objects within a field of view at a high repetition rate, providing detailed surface information of anything within the field of view (e.g., anatomical structures, surgical equipment, surgical tools, surgical staff, etc.,) and ensuring uninterrupted tracking during dynamic surgical procedures. While the 3D scanner 12 disclosed herein is optimized for this application, the CAAT system 10 is compatible with any 3D scanner capable of optically generating topological information from the surgical site. In some embodiments, the 3D scanner 12 may be enclosed in a sterile drape to maintain sterility, allowing the 3D scanner 12 to function within the sterile surgical environment.

[0163] In some embodiments, the hub 14 is a dedicated computer or computer system responsible for receiving raw image data from the 3D scanner 12. Equipped with specialized hardware and software, the hub 14 processes this data in real-time, converting the data into a usable format for tracking anatomical structures and surgical instruments. In some embodiments, the CAAT system 10 performs automatic registration of the data to a Surgical Coordinate System (SCS), eliminating the need for manual optical trackers or registration steps. In some embodiments, the hub 14 may comprise multiple computer systems that communicate with one another so that data can be processed serially or in parallel on separate hardware.

[0164] In some embodiments, the hub 14 transmits transformation matrices for each tracked object to a separate or second computer system, where the data is further processed based on the surgical workflow. This information is presented to the surgeon in an intuitive, user-friendlyinterface 16, providing real-time visualization of anatomy and surgical tools. In some embodiments, the interface guides the surgeon through key actions such as implant positioning, bone cuts, and other critical tasks.

[0165] In some embodiments, while the hub 14 plays a central role in data processing, the CAAT system 10 architecture allows for flexibility in where computations occur. For example, in some embodiments, some processing tasks, such as initial data filtering or compression and / or reconstruction of a 3D point cloud, may be performed by a processor within the 3D scanner 12 itself to reduce data transmission loads. Additionally, certain computations may be handled by other computers within the system, either in series or parallel with the hub 14, depending on the complexity of the procedure and the system's overall configuration. In some cases, specific computational tasks, such as visualization rendering or more advanced data analysis, may be managed directly within the user interface (UI) 16 system. This distributed computation 20 approach enables scalable performance and ensures that real-time requirements are met across a range of surgical applications.[00166J 3D Scanner

[0167] Fig. 4 illustrates an example of a surgical environment 22 which may be a sterile field within an operating room at a hospital or surgical center, in which a patient 24 is positioned on an operating table 26 for a surgical procedure such as a TKA. By way of example, a surgical instrument 28 (e.g., drill, oscillating saw, etc.) may be provided at the end of a first robotic arm 30, which itself is attached to a first cart 32. A 3D scanner 12 forming part of the CAAT system 10 is provided at the distal end (or “end effector”) of a second robotic arm 34 which is provided on a second cart 36. It should be noted that this surgical environment 22 is provided by way of example only to illustrate one possible setup and positioning of the 3D camera in relation to a patient during a surgical procedure. In some embodiments, the second robotic arm 34 to which the 3D scanner 12 is attached may be attached to a wall, ceiling, immobile table, or other structure.

[0168] Figs. 5-8 illustrate an example of a 3D scanner 12 configured for use with the CAAT system 10 of the present disclosure, according to some embodiments. In some embodiments, the 3D scanner 12 comprises an optical system 42, a light source assembly 44, a thermalmanagement system (TMS) 46, a cover 48, and a rigid scaffolding 50. In some embodiments, the optical system 42 includes a camera, lens, and filter, and is configured for continuously capturing data including high repetition images of patterns projected onto the surgical field (from which topological data is derived and used in reconstructing a 3D point cloud) and communicating the captured data to the hub 14. In some embodiments, the light source assembly 44 is configured to project a structured light pattern onto the target surface, which the CAAT system 10 may then analyze as described below to generate a 3D model of the surface area with the optical system’s field of view, as described herein. In some embodiments, the TMS 46 is configured to remove heat from the 3D scanner 12. More specifically the TMS 46 removes heat generated by the light source and the optical system 42 by using airflow to transfer the heat through a heat sink and ultimately out of the 3D scanner 12 through ventilation tubes 178, as described in detail below. The TMS 46 shown and described herein is representative of one example of removing heat from the 3D scanner 12. However, the 3D scanner 12 may be modified to incorporate other methods of removing heat in addition to or instead of the TMS 46 described herein.

[0169] In some embodiments, the optical system 42 has a field of view 68 with a central axis Li, and the light source assembly 44 has an illumination field 80 with a central axis L2. In some embodiments, the optical system 42 and light source assembly 44 may be positioned at an angle 0i relative to one another defined as the offset angle between the central axis Li of the optical system 42 and the central axis L2 of the light source assembly 44, as shown by way of example in Fig. 7. This arrangement produces an overlap area (or “working volume”) 69 in which the field of view 68 overlaps with the illumination field 80. Objects within the overlap area 69 may be scanned by the 3D scanner 12. Objects that are not in the overlap area 69 will not be scanned by the 3D scanner 12. In some embodiments, the angle 0i may also be defined as the apex angle in a triangulation between the optical system 42, the light source assembly 44, and the target site. By way of example, the apex location in the triangulation represents the optimal point within the overlap area 69 in which the optical system 42 produces the best imaging results. In some embodiments, the optical system 42 and light source assembly 44 may be positioned such that the angle 0i may be within a range of 10° to 30°. In some embodiments, the optical system 42 and light source assembly 44 may be positioned such that the angle 9i may be within a range of10° to 20°. In some embodiments, the optical system 42 and light source assembly 44 may be positioned such that the angle 0i = 16.5°. In some embodiments, the 3D scanner 12 may be configured such that the angle 9i may be adjustable during use, for example to adjust the working volume 69 of the 3D scanner 12 or if the positioning of the 3D scanner 12 relative to the target site needs to be moved. In some embodiments, the 3D scanner 12 may be configured such that the focus is operable by a motor. In some embodiments, the 3D scanner 12 may be configured such that the focus is automatically adjustable. In some embodiments, the 3D scanner 12 may be configured such that the working volume 69 may be adjusted by changing the lateral distance between the optical system 42 and the light source assembly 44. In some embodiments, the 3D scanner 12 may be configured such that the focus may be adjusted by changing angle 6i between the optical system 42 and the light source assembly 44. In some embodiments, the 3D scanner 12 may be configured to include an adjustable and / or motorized iris to enable a user to fine tune the level of light entering the camera.

[0170] Fig. 9 illustrates an example of an optical system 42 forming part of the CAAT system 10 of the present disclosure, according to some embodiments. In some embodiments, the optical system 42 may comprise a camera 60, lens 62, filter 64, and a data connection 66. In some embodiments, the camera 60 may be any compatible camera available in the market currently or in the future. In some embodiments, the camera 60 may have a silicon sensor configured to collect the light. In some embodiments, the sensor may be selected to have a high sensitivity to a specific range of wavelengths of light, including but not limited to (and by way of example only) a range of 750-850 nm, or 750-800 nm, or 800-850 nm. In some embodiments, the camera 60 has a field of view 68 defined as the area in front of the camera lens 60 and / or filter 64 that is captured by the camera. Points of interest within the field of view 68 may be captured by the camera 60 and transmitted as data to the hub 14. In some embodiments, the data may be sent directly to the hub 14 by way of the data connection 66. In some embodiments, the data may be sent indirectly to the hub 14 by way of an intermediate component, which may process the data before sending the data to the hub 14. Points of interest that are outside of the field of view 68 will not be captured by the camera 60. In some embodiments, the lens 62 extends from the camera 60 and may be configured to have an adjustable and / or motorized field of view range.Alternatively, the lens 62 may have a fixed field of view and changing the lens 62 would then change the field of view.

[0171] In some embodiments, the filter 64 may be a high pass wavelength filter or spectral filter. In some embodiments, the filter 64 may be selected and / or configured to block light wavelengths below a predetermined threshold value from passing through the filter 64. In some embodiments, the predetermined threshold value may be 800 nm such that wavelength values below 800 nm are blocked and wavelength values above 800 nm (e.g., including infrared light) are able to pass through the filter 64. In some embodiments, the predetermined threshold value may be 750 nm. In some embodiments, the filter 64 may comprise a band pass filter configured to block light wavelengths below a minimum predetermined threshold value and above a maximum predetermined value from passing through the filter 64 such that light having wavelength values within a specified range may pass through the filter 64. In some embodiments, the specified range may be 800-850 nm. In some embodiments, the specified range may be 810-840 nm. By way of example, such specified ranges may block visible light (e g., light wavelengths below 800 nm) and higher light wavelengths (e g., light wavelengths above 850 nm) from passing through the filter 64. This helps to maximize the dynamic range of the sensor to capture the pattern projected. For example, in a surgical environment, a NIR wavelength range would not be visible to the surgeon or other staff and the ambient lighting within the operating room would be blocked by the filter, allowing only light reflecting off the target area to be allowed into the camera.

[0172] Fig. 10 illustrates an example of a light source assembly 44 forming part of the CAAT system 10 of the present disclosure, according to some embodiments. In some embodiments the light source assembly 44 includes a linear light source 70, a light shield 72, a light box 74, a transmissive LCD (or pattern generator) 76, and a pattern generator cover 78. In some embodiments, light generated by the light source 70 passes through the light shield 72, light box 74, and pattern generator 76 on its way to the surgical target site. In some embodiments, the light source assembly 44 has an illumination field 80 defined as the maximum possible area in front of the light source assembly 44 that may be illuminated by light emanating from the linear light source 70. In some embodiments, the linear light source 70 comprises an elongated emission component configured to emit light through the light shield 72 and light box 74. Insome embodiments, the emitted light may have a wavelength value within a specified range. In some embodiments, the specified range may be 800-850 nm. In some embodiments, the specified range may be 810-840 nm. In some embodiments, the linear light source 70 comprises a plurality of LED emitters mounted to a chip-on-board (COB) printed circuit board (hereinafter “COB 82”) in a linear orientation. In some embodiments, the linear light source 70 may comprise one or more groups of LED emitters, with each of the one or more groups of LED emitters independently powered with voltage and current. In some embodiments, the linear light source 70 may comprise 10 groups of 21 LED emitters.

[0173] Figs. 11-13 illustrate an example of a pattern generator 76 forming part of the 3D scanner 12, according to some embodiments. In some embodiments, the pattern generator 76 comprises a transmissive LCD screen with individually controllable opacity comprising a liquid crystal (LC) layer 140 positioned between first and second polarizing layers 142, 144

[0174] In some embodiments, the first polarizing layer 142 ensures light entering the LC layer 140 is polarized. The LC layer’s twist (controlled by voltage) determines whether light’s polarization matches the second polarizer, and how much light is allowed through the LC layer 140. The second polarizing layer 144 acts as a gate that only passes light correctly rotated by the LC layer 140. This arrangement enables each energized line 150 within the LCD layer 140 to precisely modulate light intensity, forming a distinct and sharp pattern of edges. In some embodiments, the first and second polarizing layers 142, 144 may be near infrared (NIR) polarizers (e.g., which allow near infrared light through but block most visible light) instead of more common optically transmissive polarizers. In some embodiments polarizers may also allow for transmission of some visible light (though not as well as the usual polarizers), allowing for dual functionality in the event that the 3D scanner 12 is configured to have both NIR and visible lights present on a single unit.

[0175] Fig. 14 illustrates an example of a thermal management system (TMS) 46 forming part of the 3D scanner 12, according to some embodiments. By way of example, the TMS 46 is provided to remove heat from the major heat sources in the 3D scanner 12, namely the linear light source 70 and the camera 60 to prevent overheating of the light source 70 and the camera 60. In some embodiments, the TMS 46 may include a heat transfer assembly 170, a heat sink172, an air box 174, a vent box 176, and one or more ventilation tubes 178. In some embodiments, heat is transferred from the linear light source 70 and the camera 60 to the heat sink 172 by the heat transfer assembly 170. Meanwhile, airflow is forcibly directed into one of the ventilation tubes 178 and / or forcibly pulled out of the other of the ventilation tubes 178 by fans that are at distal ends of the ventilation tubes which may be located outside of the sterile field (e.g., external of the sterile drape). In some embodiments, the forcible pushing and forcible pulling operate in concert so that the airflow is in one direction through the TMS 46. In some embodiments, only pulling fans may be used. In some embodiments, only pushing fans may be used. Heat that is pulled from the linear light source 70 and the camera 60 is directed into the heat sink 172, wherein it is transferred to the air that is directed through the heat sinkl72. This heated air is then directed out of the TMS 456 through a ventilation tube 178.

[0176] Scanning and Data Acquisition Process

[0177] As shown in Fig. 4, by way of example, the 3D scanner 12 may be positioned to continuously capture the surgical field. In some embodiments, as the surgeon manipulates instruments and interacts with the patient’s anatomy, the 3D scanner 12 generates a point cloud of the visible surfaces within the overlap area 69 (e.g., the surfaces that are within both the illumination field 80 of the light source assembly 44 and the field of view 68 of the optical system 42). In some embodiments, assuming data is collected at a rate exceeding 20 Hz, the data stream may be continuously updated in real-time. While faster rates are preferred to reduce system latency, rates above 20 Hz are generally sufficient for functional real-time performance. In some embodiments, the point cloud data is then used to create transforms that can be used to update the position and orientation of 3D reference geometry of the anatomy and any tracked instruments within the Surgical Coordinate System (SCS).

[0178] Fig. 15 is a block diagram depicting a data acquisition process 250 using a 3D scanner 12 described herein. While the 3D scanner 12 disclosed herein introduces several novel features, the general principles of structured light scanning are well established and are described here for clarity. Furthermore, this data acquisition process 250 may be used with other scanners, such as a shadow caster of the type disclosed in in the ‘973 app.

[0179] In some embodiments, a first step 252 in the data acquisition process 250 is to generate and project a structured light pattern onto a surface to be scanned. For example, in some embodiments, the 3D scanner 12 may illuminate the target surface with a predefined light pattern (e.g., commonly grids, stripes, or dots) projected onto the surfaces being scanned. By way of example, this may be accomplished by configuring the pattern generator 76 described above to produce the desired structured light pattern. In some embodiments, the 3D scanner 12 described herein may be configured to use infrared light, ensuring that the operating room lighting does not interfere with the scanning process. In some embodiments, a series of light patterns is projected to encode all the data necessary for reconstructing the 3D point cloud. Multiple pattern series can be used, depending on the specific requirements of data encoding. In some embodiments, variations in intensity, color, or phase may be applied to the pattern to enhance data accuracy or accelerate acquisition.

[0180] In some embodiments, a second step 254 in the data acquisition process 250 is image capture. For example, in some embodiments, cameras or sensors within the 3D scanner 12 (e.g., optical system 42 described above) capture the reflected structured light as it interacts with the target surface. In some embodiments, an image may be taken for each pattern projected in the series. By way of example, the captured images reflect deformations in the light pattern caused by the surface's contours and depth. In some embodiments, these images may be collected by one or multiple cameras, with the relative positions and orientations of the cameras being known to the CAAT system 10. In some embodiments, the first step 252 of generating and projecting structured light pattem(s) onto a surface to be scanned and the second step 254 of image capture may be repeated in an iterative loop until all the required images have been acquired.

[0181] In some embodiments, once all required images have been obtained by way of the previous steps, a third step 256 in the data acquisition process 250 is reconstruction of the 3D surface, for example within the hub 14. In some embodiments, reconstruction of the 3D surface may occur on board the 3D scanner 12 in a main control board rather than in a separate hub 14, in which case the following sub-steps also occur within the main control board of the 3D scanner 12. In some embodiments, the reconstruction step 256 includes sub-steps of pattern decoding, triangulation and 3D point cloud generation. In some embodiments, after image capture, the images are sent to the hub 14 for processing. In some embodiments, a first sub-step 258 of thereconstruction step 256 is pattern decoding. By way of example, data can be transferred from the 3D scanner 12 to the hub 14 continuously as images are captured, or in bulk once an entire pattern series is complete. In some embodiments, the reconstruction step 254 cannot be completed until all captured images are transferred to the hub 14, but some computation can be completed on each image as it arrives in the hub 14. In some embodiments, the CAAT system 10 is configured to decode the captured light patterns by analyzing their brightness history to create camera pixel correspondences to light angle projection. This, together with known pixel observation ray, enable via triangulation in-depth information, allowing the CAAT system 10 to reconstruct the 3D surface as a 3D point cloud.

[0182] In some embodiments, a second sub-step 260 of the reconstruction step 256 is triangulation. By way of example, the CAAT system 10 may use triangulation to calculate the 3D coordinates of each point where the light interacts with the surface. With the known positions of the linear light source 70, camera 60, and pattern generator 76, the CAAT system 10 determines a 3D point for each pixel in the captured images.[00183J In some embodiments, a third sub-step 262 of the reconstruction step 256 is generation of a 3D point cloud 268 representing the scanned target surface. By way of example, the triangulated points from each frame may then be combined into a 3D point cloud 268, representing the surface in 3D space, with each point assigned x, y, and z coordinates. An example of a 3D point 268 cloud is shown in Fig. 16. In some embodiments, depending on system capabilities, additional data such as color or intensity may be included to enhance detail and visual fidelity.

[0184] In some embodiments, a fourth step 264 in the data acquisition process 250 is data processing and filtering. For example, in some embodiments, the raw point cloud data may undergo post-processing to reduce noise, fill gaps, and improve accuracy. In some embodiments, if multiple scans are taken from different angles, the data may be aligned, merged, or "stitched" together to create a cohesive 3D model.

[0185] In some embodiments, a fifth step 266 in the data acquisition process 250 is to output the data such that the final point cloud 268 is made available for further processes, such as segmentation and registration, which can be used to guide the surgical workflow.

[0186] Differences between optical tracker generated data and 3D scanned topology data

[0187] It is important to distinguish between the data generated by Optical Markers (OMs) and Optical Trackers (OTs) (e.g., as used in prior art system) versus the data produced through 3D scanning of a surgical site, as disclosed herein.

[0188] By way of example, in Figs. 17A-B, the image on the left (Fig. 17A) depicts a typical TKA surgery, showing externally applied optical trackers 270 attached to the robot 272, femur 274, and tibia 276. Each externally applied optical tracker 270 comprises four optical markers 278 in a predetermined arrangement. This image represents the typical field of view for stereoscopic systems. On the right (Fig. 17B), the image shows what one of the stereoscopic cameras captures after segmenting the image to isolate the OMs 278 (which by way of example appear as black markers on a white background). Clearly visible are three OTs 270, each consisting of four OMs 278. When two such images, taken from slightly different angles, are combined via triangulation, the OTs 270 can be located in space, inferring the positions of the bones and robotic instruments registered to their respective trackers.

[0189] In contrast, the data collected by a 3D topological scanner such as 3D scanner 12 presents a very different picture. By way of example, Figs. 18A-B illustrates images from the 3D scanner 12. By way of example, the image on the left (Fig. 18A) is a scanner image 280 captured by the 3D scanner 12 using infrared light, and the image on the right (Fig. 18B is a 3D point cloud 268 generated from the whole pattern series of collected image data.

[0190] For example, one key difference is the field of view. For a stereoscopic system to capture all the necessary OTs 270 during surgery, the field of view must be quite large, often requiring the cameras to be positioned several meters away, typically at an angle from the side and slightly above the surgical area. From this vantage point, the bones are often not visible to the naked eye, as they may be obstructed by surgeons, instruments, other tissues, etc. By contrast, the 3D scanner 12 provides a much narrower field of view, captured from a closer, top- down perspective, for example as shown in Fig. 4. Moreover, the OTs are some distance away from the anatomy and errors localizing the trackers are amplified by that distance, so that the error is greater at the anatomy. In contrast, when the CAAT system 10 registers and tracks theanatomy itself, that distance does not exist, thereby potentially resulting in reduced relative localization errors when compared to tracker-based systems.

[0191] Another key difference is that the data density is markedly different between the two systems. By collecting points one by one, the stereoscopic system provides sparse data, providing around 50 possible data points on the surface of a femur, for example. This must be collected manually by using a pointer to touch a point and then a foot pedal may be activated to capture the point. This is time consuming and can be error prone if the pointer is moving. Also, on the articular surface of the femur and tibia, if the surgeon is trying to register to preoperative CT bone, then the pointer must be sharp and penetrate the cartilage fully without penetrating the bone. Conversely, the 3D scanner 12 of the present disclosure captures a much more robust data set without penetrating patient anatomy. For example, the image shown in Fig. 18B has approximately 4 megapixels, with about 300,000 individual points on the femur and 200,000 on the tibia. Additional points correspond to surrounding soft tissues. In some embodiments, a depth filter may be applied to the data to remove much of the background (such as the blue surgical drape).

[0192] By way of example, to further illustrate the 3D nature of the data, Fig. 19 presents the scan data (e.g., point cloud 268) from an oblique angle, giving a clearer sense of its depth. By way of example, for users familiar with knee anatomy, identifying key anatomical structures is relatively straightforward. However, for clarity, Fig. 20 highlights several major anatomical landmarks, by way of example, including the distal end of the femur 274, the anterior cruciate ligament (ACL) 282, the proximal end of the tibia 276, the patellar tendon 284, and a side view of the patella 286. Other points correspond to skin, muscle, fascia, and fat, though they are not specifically labeled. Note that these images use simulated tissue models commonly used for research, not real tissue.

[0193] By way of example, in real tissue, the visual characteristics will vary due to differences in how materials interact with light. These images are representative, but real tissue tend to introduce more noise, particularly in tissues that are partially transparent to infrared light. In some embodiments, this increased noise can be mitigated by applying more aggressive filtering algorithms, for example. In some embodiments, to help define a more reliable shape atone point in a procedure, a wavelength may be used that has a shorter optical penetration length. That wavelength may overlap with other sources of light in the operating room, but may accommodated for the sake of an initial accurate view for reference with subsequent scans taken at wavelengths that introduce more noise, but that do not compete with other sources of light in the operating room. In addition, it may also be a relatively narrow-band source (e.g. 5 to 10 nm bandwidth) that has a relative brightness at that wavelength much greater than that of the room light, such that spectrally filtering for that wavelength results in high-contrast scans with minimal competition with the ambient light.

[0194] Registration Using Topological Data

[0195] Once topological data is available in the form of a 3D point cloud 268, a wide range of capabilities become possible. In some embodiments, multiple 3D point clouds may be aligned and / or combined within a single coordinate space for a more accurate and efficient surgery. By way of example, this aligning and / or combining may occur by way of a process called “registration”, which aligns two or more 3D point cloud datasets, combining them into a common coordinate space. This registration process allows different datasets — such as preoperative CT scans and intraoperative 3D scans — to be aligned and even combined, enabling more precise surgical planning and execution. By way of example, Fig. 21 illustrates an example registration process 290 that may be performed by the CAAT system 10. In some embodiments, the registration process 290 comprises several steps including but not necessarily limited to preprocessing the data for accuracy 292, segmentation 294, registration of anatomical structures 296, registration of instruments and equipment 298, and making precise measurements 300. Although presented as a block diagram, many of these “steps” may be performed in any order, in series or parallel, and / or independently of one another. For example, anatomic registration and instrument registration may occur in any order, independent of one another, or one may occur without the other occurring.

[0196] In some embodiments, a first step 292 in the registration process 290 using topological point cloud data is pre-processing the raw point data cloud to ensure accuracy and reliability. In some embodiments, this pre-processing step 292 may include several sub-steps including noise reduction 301, smoothing 302, upsampling 303 the 3D point cloud, and / or downsampling 304the 3D point cloud. By way of example, noise in the data can be introduced from various sources, such as light interference or reflections from surgical instruments. In some embodiments, noise reduction techniques 302 like statistical outlier removal and radius-based filtering (for example) are used to clean the dataset, ensuring that only relevant points remain. In some embodiments, smoothing algorithms 303, such as Gaussian fdtering or bilateral fdtering, may be applied to create a more continuous and realistic surface representation. For example, this step may help to reduce local variations in point locations that are not significant for registration, ensuring a more refined dataset without losing critical anatomical details. In some embodiments, the 3D point cloud may optionally be upsampled 304 by increasing the data points in the point cloud. In some embodiments, the 3D point cloud may optionally be downsampled 304 by decreasing the data points in the point cloud. In some embodiments, further data processing techniques such as outlier removal 305, normal estimation 306, and / or feature extraction 307 may be used.

[0197] In some embodiments, a second step 294 in a registration process 290 using topological point cloud data is segmentation. Segmentation enables the CAAT system 10 to handle multiple objects simultaneously during registration while reducing the computational load. By way of example only, once the point cloud data has been pre-processed, the segmentation step 294 can be applied to isolate different anatomical structures or surgical tools. Segmentation is an optional step but provides significant benefits for both data handling and computational efficiency. One such benefit is that segmentation allows for the identification of different objects within the dataset. For example, all points belonging to the femur can be isolated from the main dataset, while points belonging to the tibia can be similarly separated. Any remaining data can be segmented or discarded if unnecessary. Another benefit of segmentation is related to speed and latency. For example, registering large point clouds can be computationally intensive. By segmenting and reducing the dataset to only the objects of interest, computational requirements are lowered, improving processing speed and reducing overall system latency. In some embodiments, the segmentation step 294 may occur before the pre-processing step 292 to perform segmentation using raw, unfiltered data.

[0198] In some embodiments, various methods of segmentation 294 may be used, including but not limited to manual segmentation 308, traditional algorithms 309, and artificial intelligence(AT) 310 powered segmentation. By way of example, manual segmentation 308 may be useful in complex cases, as operators can manually label objects in the point cloud to isolate key structures. In some embodiments, traditional algorithmic techniques 309 such as edge detection or region-growing may be used to identify structures based on their shape or intensity. In some embodiments, Al-powered segmentation 310 may be used so that pre-trained deep learning models can automatically segment point cloud data in real-time, offering high accuracy and minimal manual intervention. In some embodiments, the Al-powered segmentation 310 may operate within a 2D image space (e.g., a 2D image or 2D depth-map representation) or a 3D space (e.g., point cloud space).

[0199] In some embodiments, a third step 296 in a registration process 290 using topological data is anatomic registration. In most cases, registration involves aligning two datasets, such as a preoperative CT dataset and a 3D topological scan from the surgical site. By way of example only, the alignment may be performed using algorithms such as RANSAC (Random Sample Consensus) 312 for initial, coarse registration and ICP (Iterative Closest Point) 314 for finetuning. These algorithms compare corresponding points between datasets and compute a transformation matrix that minimizes alignment errors. For example, RANSAC 312 selects key features in the point cloud, filtering out outliers, and computes an initial transformation matrix by focusing on valid, inlier points, and ICP 314 iteratively refines the resulting transformation matrix by minimizing the distance between corresponding points in the two datasets, leading to precise alignment. The resulting transformation is a 4x4 rigid matrix that translates, rotates, and scales one dataset to align with the other, for example similar to the transformation matrix shown in Fig. 1. In the context of a TKA procedure, for instance, this could align a CT scan with a realtime topological scan of the femur 274 and tibia 276

[0200] In some embodiments, the anatomic registration step 296 can be repeated for multiple objects, such as the femur 274 and tibia 276, allowing simultaneous registration of more than one object assuming the computational horsepower is available. Fig. 22 illustrates the registration of CT data from the femur 274 and tibia 276, along with a 3D topological scan or point cloud 268, into a common coordinate space 316. By using registered datasets, the CAAT system 10 provides surgeons with accurate measurements that aid in making critical decisions during surgery. By way of example only, Fig. 22 shows the registration of three datasets — CT datafrom the femur and tibia, and a 3D topological scan — into a common coordinate space for this purpose.

[0201] The registration step 296 is a critical step in aligning two 3D point cloud datasets, such as preoperative CT data with a real-time topological scan in a surgical setting. This method achieves the same results as registration done with Optical Markers (OMs) 278 and Optical Trackers (OTs) 270, but without the need for external tools anchored to bones or an additional transformation to account for the indirect determination of a transformation matrix via the trackers. Additionally, the topological dataset is far denser and more detailed than what a surgeon could collect using a pointer with OT.

[0202] In some embodiments, a fourth step 298 in a registration process 290 using topological data is instrument and equipment registration. By way of example, this step is optional and may occur before, during, or independently of anatomic registration in the previous step 296. In addition to anatomical structures, instruments and surgical equipment can be registered using topographical data. By way of example, each instrument has a unique shape that can be used for registration. For instance, a bone saw has a distinct geometry compared to a scalpel, making identification possible in the topological data. In some embodiments, the same registration methods used for anatomy can be applied to instruments, but some challenges exist. For example, instruments with reflective surfaces may not be well-captured in the 3D scan. In some embodiments, treating surfaces to scatter light can reduce specular reflections and improve scan quality. Another challenge is that some materials absorb light, causing poor visibility in the scan. As with specular reflections, in some embodiments treating surfaces to scatter light can mitigate this issue. By way of example, another challenge with instrument and equipment registration is occlusions, in that instruments and / or equipment may be partially obscured by the surgeon's hand. In some embodiments, by extending the handle or otherwise introducing distinct geometry, the CAAT system 10 can still register the instrument even when partially blocked. In some embodiments, distinct geometry may include (but is not limited to) portions of the tracker that are not optical markers, patterns printed on the instruments (e.g., particularly on their most distal end) and / or attachment of a fiducial to the instrument enabling identification and understanding of the instrument position and / or orientation at the surgical site itself.

[0203] In cases where an anatomical point is not directly visible (e.g., the center of the ankle), a pointer instrument can be used. In some embodiments, for example, registering an end of the pointer to the scanned data, the anatomical point can be inferred accurately. In some embodiments, this could be done by registering a proximal end (e.g., that would extend into camera view) of the pointer in the common coordinate system and, based at least in part on the known attributes or the pointer, the relative positioning of the distal tip (e.g., that would be positioned near the target out-of-view anatomical point) would be known or inferred. In this way, moving the scanner in order to change its field of view to see anatomy at the ankle (for example) may be avoided.

[0204] In some embodiments, a fifth step 300 in a registration process 290 using topological data is measurements. For example, once the objects are registered within the same coordinate space, precise measurements can be made, providing crucial information during surgery. These measurements may include, but are not limited to distance measurements, angle measurements, and / or gap measurements. By way of example, measurements made after registrations are relative measurements between two registered objects within the coordinate space or a single registered object and the scanner reference frame in the coordinate space. In some embodiments, the CAAT system 10 can measure the distance between two points on anatomical structures or instruments. For example, measuring the width of the knee’s sulcus is as simple as selecting two points (either manually, or with the points identified by Al or by registration with a model, or with CT), with the system computing the distance in real-time. In some embodiments, more complex measurements, like assessing the varus or valgus angle of the knee, can be performed by identifying the sagittal planes of the femur and tibia in the registered datasets. Once identified, the system calculates the angle between the planes, providing vital information for implant alignment. In some embodiments, if two objects are registered, measurements between the two objects can be computed. For example, in TKA surgeries, the medial and lateral gaps between the femoral condyles and tibial plateau can be measured to assess ligament balance. Surgeons often aim to balance these gaps to ensure even tension in the knee, which is critical for the success of the implant.

[0205] Also, by measuring known anatomic features of the knee, careful fitting of a parametric model of the knee may be possible directly for implant selection or even real-time manufacture.

[0206] Tracking with Topological Data

[0207] By way of example, tracking and registration are closely related, but they serve distinct purposes in a surgical setting. As previously described, registration is the process of aligning two or more 3D point cloud datasets, bringing them into a single coordinate system. This is typically a one-time event or a relatively infrequent operation. In contrast, tracking involves continuously registering these datasets over time, monitoring changes in position or orientation, and keeping objects aligned within the same coordinate system. Tracking updates the spatial alignment of registered objects multiple times per second, which allows for dynamic monitoring of anatomical structures and surgical tools as they move throughout the procedure.

[0208] Real-time tracking is essentially registration performed in a continuous loop, where the system rapidly processes new data to reflect movements or changes as they occur. The faster this loop is executed, the more accurately the system can represent the current state of the surgical environment. In some embodiments, and by way of example only, registration uses a dense point cloud, while tracking may use “sparse” scans, which are not the highest pixel density scans, but are much faster to take. The goal of real-time tracking is to create a seamless experience where movements of anatomical structures, instruments, or other objects are captured and reflected without noticeable delay.

[0209] Defining what constitutes "real-time" is challenging because it varies depending on the context and application. In the world of cinema, for example, a standard film is displayed at 24 frames per second (fps). At this rate, the human brain perceives the sequence of still images as continuous motion, which most viewers consider to be real-time. Computer monitors typically operate at higher refresh rates, such as 60Hz or 120Hz, offering smoother visuals. Gaming monitors push this further, with refresh rates of up to 240Hz, though this is generally only advantageous in specific, fast-paced applications.

[0210] From a physiological standpoint, studies suggest that the human brain begins to interpolate between frames at speeds around 24 fps, making discrete images appear as a fluid, continuous motion. For the purposes of this disclosure, we define real-time tracking as performing at least 20 registrations per second (rps). This threshold provides a sufficiently smooth experience, making the data appear continuous to the surgeon. Faster registration speeds are always preferable, as they can further reduce latency, but they come with significant tradeoffs in terms of the system’s requirements. Higher speeds increase the demands on light intensity, the speed of image sensors, and computational power to process the data quickly enough to maintain real-time performance.

[0211] In some embodiments, speed may be increased by bypassing the coarse or global registration step (e.g., RANSAC as described above) during tracking, and instead loop through the fine registration (e g., ICP) step. This is because typically the tracked object does not move very much or very far in the short time between scan registrations, it is already close enough for successful ICP operation. In some embodiments, the exclusion of a coarse registration step may provide a significant increase in tracking speed. In some embodiments, tracking may be lost during ICP-only operation (e.g., the tracked object moves faster or farther than ICP can successfully register). In such cases, by way of example only, the system 10 may revert to a global registration step which may be RANSAC as described above, or in some embodiments an Al-driven analysis of captured images (e.g., from the 3D scanner 12) may be used to quickly determine the location of the tracked object.

[0212] In some embodiments, the CAAT system 10 may increase tracking speed by using a previously registered object position to constrain the considered data points of where the object may move to next for fine registration. In some embodiments, this would reduce the points for consideration during fine registration (e.g., ICP), and fewer points to analyze generally leads to increased processing speed. By way of example, in some embodiments, this data may be constrained based on velocity and / or acceleration. In some embodiments, an object can’t go faster than a maximum determined velocity in any direction, which limits or constrains the potential data points needed for fine registration. In some embodiments, the data points could be constrained base on likely acceleration of the object. In some embodiments, object velocity may be used to extrapolate an initial guess location for the fine registration.

[0213] In some embodiments, the CAAT system 10 may constrain the considered data points of where the object may move to next for fine registration by limiting points of consideration in the acquired topological scan(s) that are a certain distance away from the previously registered object points. In some embodiments, this may include points located within an axis-aligned bounding box in every direction by an amount that is limited by practical physics, for example a set amount (e.g., a guess without first measuring velocity). In some embodiments, the CAAT system 10 may be configured to select a direction for the bounding box growth based on its previous velocity.

[0214] In some embodiments, the CAAT system 10 may constrain the considered data points of where the object may move to next for fine registration by considering only points within a radius from each of the relevant points (and optionally in a particular direction).

[0215] Single object tracking (Hip Center Example)

[0216] The simplest case of tracking is the real-time monitoring of a single object, where its position is continuously updated in the Surgical Coordinate System (SCS). This type of tracking is critical in various surgical procedures where accurate positioning of key anatomical landmarks or surgical tools is required. In this example, the scanner is kept stationary, or its motion is itself tracked relative to the operating room via some other navigation device.

[0217] A practical example of single object tracking in a TKA procedure is the calculation of the hip center of rotation. In some TKA methodologies, such as the Mechanical Alignment approach, determining the precise location of the hip center is essential for establishing the mechanical axis of the leg, which guides the alignment of implants. By way of example, Fig. 23 presents a block diagram depicting steps of a method 320 of determining the hip center of rotation using the CAAT system 10 described herein. As an initial pre-operative (“pre-op”) step 321, a 3D model of the femur may be generated from a pre-op scan, for example by way of a CT scan, MRI, and the like, and uploaded to or otherwise made available to the CAAT system 10. By way of example, the method 320 for calculating hip center of rotation described herein uses dynamic, real-time topological data to calculate the hip center of rotation from a series of transformed femoral origin (or “chosen anatomic”) points. By way of example, the chosen anatomic points may be any given point on the pre-op 3D model of the femur that is registeredinto camera space. In some embodiments, for example, during surgery, as the surgeon rotates the knee within the scanner’s field of view, the scanner captures multiple scans in real time. Each scan records the femoral origin, which is transformed in camera space based on the femur’s position. Since the femur is constrained by the hip joint, these transformed origin points lie on a spherical surface, with the center of this sphere representing the hip’s center of rotation.

[0218] In some embodiments, the first eight steps of the method may be repeated in a (rapidly occurring) loop until sufficient data has been accumulated to accurately determine and report a hip center of rotation and femur length. By way of example, a first step 322 in the method 320 of determining the hip center of rotation is using the 3D scanner 12 described herein to acquire topological scan data of the femur 274. For example, as the surgeon moves the patient’s knee within the field of view 80 of the 3D scanner 12, multiple scans of the femur may be taken in real time. By way of example, this continuous tracking captures multiple high-density data sets within a few seconds, providing numerous transformed femoral points. In some embodiments, a second step 324 of the method 320 of determining the hip center of rotation is registering the 3D model of the femur 274 to the acquired topological scan data. In some embodiments, a third step 326 in the method 320 of determining the hip center of rotation is producing a transformation matrix for each time point. This transformation matrix 2 maps the position of the femur into camera space, allowing the system to track the femur's movements in real-time as the surgeon manipulates the leg.

[0219] In some embodiments, a fourth step 328 of the method 320 of determining the hip center of rotation is to use the generated transformation matrix to compute the location of a chosen anatomic point on the pre-op 3D model for determining the length of the femur in camera space. In some embodiments, a fifth step 330 of the method 320 of determining the hip center of rotation is to send coordinates of the tracked chosen anatomic point to the hub 14 (for example) for sphere fitting.

[0220] As previously mentioned, the first eight steps of the method may be repeated in a (rapidly occurring) loop until sufficient data has been accumulated to accurately determine and report a hip center of rotation and femur length. The first five steps identified above describe the CAAT system 10 producing a data point comprising calculated (x, y, z) coordinates for thechosen anatomic point location on the pre-op 3D model upon registration to each scan, which in some embodiments may be happening at a high frequency (e.g., at least 20 times per second in some embodiments). In some embodiments, by way of example only, the chosen anatomic point may be selected manually or automatically with a registration to a model, or with artificial intelligence. Thus, a sixth step 332 of the method 320 of determining the hip center of rotation is to add a newly determined coordinates data point to the dataset containing the previously calculated coordinates. In some embodiments, a seventh step 334 of the method 320 of determining the hip center of rotation is to calculate the size and location of a sphere to which the accumulated data points were best fit, for example using least squares fitting to find the bestfitting sphere, which minimizes the error between the observed data points — the transformed chosen anatomic points captured in real time — and the theoretical spherical surface they approximate. In some embodiments, the best-fitting sphere may be determined using other methods including but not limited to RANSAC. The center of this sphere represents the hip center of rotation, while the radius of the sphere aligns with the physical distance dictated by the femoral constraints. The least squares sphere-fitting method minimizes residuals between the observed points and the estimated spherical surface, resulting in a highly accurate center calculation.

[0221] By way of example, the mathematical basis for the least squares sphere-fitting method is the general equation of a sphere:(% - x0)2+ (y - y0)2+ (z - z0)2= r2where (xo, yo, zo) are the coordinates of the sphere’s center, representing the hip center of rotation, r is the radius of the sphere, corresponding to the distance from the chosen anatomic point to the hip center, and each transformed chosen anatomic (e.g., femoral) point (x;, yi, Zi) should ideally satisfy this equation if it lies perfectly on the sphere’s surface.

[0222] In real-world data, due to noise and minor inaccuracies, the chosen anatomic points do not lie precisely on the sphere. Thus, the residual error for each data point (xi, yi, Zi) may be defined as:

[0223] The goal is to minimize the sum of the squares of these residuals across all data points. The objective function to minimize is:where n represents the total number of data points.

[0224] In some embodiments, the calculation may be optimized using the Levenberg- Marquardt Algorithm. By way of example, the optimization process involves adjusting the parameters (xo, yo, zo) and r to achieve the minimum sum of squared residuals. Given the nonlinear nature of the problem, the Levenberg-Marquardt algorithm (e.g., a standard for nonlinear least squares optimization) may be used. This algorithm iteratively updates the center coordinates (xo, yo, zo) and r by considering the gradient of the objective function until convergence, minimizing the residuals.

[0225] By way of example, the Levenberg-Marquardt algorithm process may include: (i) initializing (xo, yo, zo) and r with initial guesses, possibly derived from preliminary data or approximate mean positions; (ii) iteratively updating the center coordinates and radius based on the error gradient, adapting step sizes according to the algorithm’s damping factor to balance convergence speed and accuracy; and (iii) converging to the best-fit parameters, continuing iterations until changes fall below a pre-defined tolerance level, signifying minimized residuals and an optimized fit.

[0226] In some embodiments, an eighth step 336 of the method 320 of determining the hip center of rotation is to compare residual errors of the fit of the sphere to the predetermined acceptance criteria to assess whether the generated best fit sphere is a good fit (e.g., decision box 336). In some embodiments, once the optimization completes, the quality of the fit may be evaluated by calculating the residual eLfor each point using the equation:

[0227] These residuals quantify how closely each point aligns with the estimated spherical surface. For example, small residuals indicate accurate fit, with residuals near zero suggestingthe points closely follow the spherical surface, indicating a good fit and that the estimated hip center accurately represents the true center.

[0228] In some embodiments, confidence analysis may be applied to verify the accuracy of the calculated hip center. By way of example, this may be accomplished through a statistical confidence ellipsoid, calculated based on the residual errors of each fitted point. This ellipsoid offers a measure of confidence, typically set to 95%, indicating that the true center likely falls within this ellipsoid, ensuring a reliable reference point for surgical adjustments.

[0229] For example, to assess the accuracy of the calculated center of rotation, a confidence ellipsoid can be constructed around the estimated center (xo, yo, zo). In some embodiments, this confidence ellipsoid defines a probabilistic boundary, delineating the region within which the true center is likely to reside at a specified confidence level, such as 95%. In some embodiments, the confidence ellipsoid is derived from the residual errors, which represent the difference between each observed data point and the surface of the fitted sphere. In some embodiments, the covariance matrix of these residual errors is calculated to reflect the spread of deviations from the fitted center in the x, y, and z dimensions. By way of example, to construct the confidence ellipsoid, this covariance matrix may be scaled by a factor derived from a chi-square distribution (for a 95% confidence level with 3 degrees of freedom, this factor is approximately 7.815), adjusting the size of the ellipsoid to define a region within which there is a 95% probability that the true center lies. The resulting semi-axes of the ellipsoid reflect the precision along each axis (x, y, z). When these semi-axes are small (e.g., less than 10 mm), it can be concluded with 95% confidence that the true center lies within 1.5 mm of the estimated center. In some embodiments, a small ellipsoid denotes high confidence in the fit, while a larger ellipsoid may suggest the need for further data points or refinement of the fitting process.

[0230] If the sphere is not a good fit, then the workflow reverts to the first step 322 of acquiring topological scan data of the femur 274. In some embodiments, if the generated best fit sphere is determined to be a good fit, then a final step 338 in the method 320 of determining the hip center of rotation is to report the hip center and femur origin length (e.g., based upon the position of the sphere origin and the radius of femur).

[0231] While some noise is inevitably introduced due to factors such as movement artifacts in the pattern images or reconstructed object, or slight variations in scan accuracy, the CAAT system 10 can compensate for these by averaging or applying noise-reduction techniques. Once the hip center is identified in camera space, it can easily be transformed into any other coordinate system (e.g., SCS), ensuring that this critical anatomical point is available for accurate surgical planning and alignment, including but not limited to intraoperative real-time modification of the surgical plan.

[0232] Through this approach, the hip center of rotation can be calculated in real time during surgery, significantly enhancing the surgeon’s ability to achieve optimal alignment and implant positioning without additional tracking equipment. This innovation supports streamlined intraoperative workflows while providing the precision needed for high-quality surgical outcomes. A successful application of this mathematical approach allows surgeons to confidently use the estimated hip center of rotation as an anatomical reference during surgery, improving alignment and outcomes without additional tracking equipment.

[0233] This example highlights the value of single object tracking in real-time during surgery, where critical anatomical landmarks must be continuously monitored and accurately calculated to guide the procedure.

[0234] Two Object Tracking and Measuring (Gap Balancing Example)

[0235] A more complex use case for tracking occurs when two or more objects are tracked simultaneously. While a previous section described how distances can be calculated between two objects through registration, this concept can be extended into a continuous tracking mode, where multiple registrations are performed over time to monitor the relative positions of multiple objects. This is especially valuable in real-time scenarios, where dynamic forces are applied, and accurate measurements must be taken as the objects move in relation to each other.

[0236] An example of this in a TKA procedure is when the surgeon uses a gap balancing methodology to assess the gaps between the medial and lateral condyles of the femur and the tibial plateau. By way of example, Fig. 24 is a flowchart depicting an example method 350 of assessing gaps between two objects, for example femoral condyles (e.g., medial and lateral) anda tibial plateau. In this method 350, the goal is to ensure that the gaps between the femur and tibia are balanced when the knee is in both extension and flexion, which helps maintain proper knee alignment and functionality after the procedure.

[0237] Measuring these gaps presents a challenge because the surgeon does not perform this measurement passively. Instead, they manually apply forces to the knee in both a varus (outward) and valgus (inward) direction. This creates a dynamic scenario where the knee joint, the entire leg, and the surgeon’s hands are moving as the force is applied. The critical challenge is that the surgeon is not looking for a single static gap value but rather the peak gap under maximum load. Because significant force is applied during this process, the positions of both the femur and tibia are constantly shifting, and these movements happen quickly. The gap measurement needs to be recorded at high speed, tracking the movement of both the femur and tibia in real time to capture the moments where the force is greatest, and the gap is widest.

[0238] In some embodiments, a first step 352 in the method 350 of assessing gaps between femoral condyles and a tibial plateau is to acquire topological scan data of the femur and tibia and generate reconstructed 3D point clouds of each, for example using techniques described above. In some embodiments, a second step 354 of the method 350 of assessing gaps between femoral condyles and a tibial plateau is to independently register the femur and tibia in the CAAT system 10 as described above, for example using the acquired and reconstructed scan data in addition to reference data 356 of the femur and tibia, for example from preoperative imaging (e.g., CT imaging) or data collected intraoperatively. In some embodiments, a third step 358 in the method of 350 assessing gaps between femoral condyles and a tibial plateau is to use transformation matrices to convert reference data to the surgical coordinate system or to another virtual space. In some embodiments, a fourth step in the method 350 of assessing gaps between femoral condyles and a tibial plateau is to calculate the distances between key points on the femur and tibia in real time. As the surgeon applies varus and valgus forces, the system tracks the resulting movements and dynamically measures the gaps between the medial and lateral condyles and the tibial plateau. In some embodiments, this includes performing a first sub step 360 of computing the medial gap (e.g., the minimum distance between the femur and tibia on the medial side) and a second sub step 362 of computing the lateral gap (e g., the minimum distance between the femur and tibia on the lateral side). In some embodiments, a fifth step 364 in themethod 350 of assessing gaps between femoral condyles and a tibial plateau is to consolidate the data and present it to the surgeon in a usable format. In some embodiments, the next step 366 is to determine whether there is enough data to proceed with the procedure. If the answer is no, then the workflow reverts to the first step 352 of acquiring and reconstructing topological scans of the femur and tibia, and the workflow continues to cycle through the steps until enough data has been obtained to assess the gaps between femoral condyles and a tibial plateau, at which point the tracking may end (box 368).

[0239] In some embodiments, by performing multiple registrations rapidly at a rate defined as real-time (e.g., 20+ registrations per second), the CAAT system 10 can record the gap values continuously as the knee moves under load. In some embodiments, recorded measurements may be analyzed to determine the peak gap value at the moment when the maximum force is applied in each direction. Because these forces are only applied for a brief period, rapid and continuous tracking is essential to ensure the correct measurements are captured at the precise moment. In some embodiments, the rapidly occurring multiple registrations may be referred to as a “tracking mode.”

[0240] By way of example, Fig. 25 is a block diagram illustrating several steps of a method 700 of calculating joint gaps based on changes in alignment between the femur and tibia under applied stress. In some embodiments, this method may be used to compute the medial gap 360 and lateral gap 362 as part of the method 350 of assessing gaps between femoral condyles and a tibial plateau described above. For example, in some embodiments, after completing bone registration, the CAAT system 10 may be configured to calculate joint gaps based on changes in alignment between the femur and tibia under applied stress. This approach provides real-time gap measurements during surgery without the need for external optical trackers. By way of example, the method 700 is described herein with respect to calculating lateral gap measurement when varus stress is applied to the affected knee. In some embodiments, the method 700 may calculate medial gap measurement in the same manner when valgus stress is applied to the affected knee.

[0241] By way of example, a first step 702 of the method 700 of calculating joint gaps based on changes in alignment between the femur and tibia under applied stress is positioning theaffected knee in extension, as shown by way of example only in Fig. 26. With the knee in full extension, the femur 274 and tibia 276 may be positioned and stabilized the to allow initial angle measurements. In some embodiments, the CAAT system 10 may be configured to include (e.g., by way of user interface 16 which may include a display component) the flexion / extension (F / E) angle display 720, the varus / valgus (V / V) angle display 722, and the internal / external (int / ext) rotation angle display 724 to help the surgeon achieve neutral alignment (e.g., approximately zero degrees in F / E and int / ext rotation).

[0242] In some embodiments, a second step 704 of the method 700 of calculating joint gaps based on changes in alignment between the femur and tibia under applied stress is establishing the zero-stress V / V angle 722 and zero gap condition. In some embodiments, the V / V angle 722 in this no-stress position is recorded as the baseline measurement. For example, at this baseline position, it can be assumed that no gap exists on either the medial or lateral side of the joint, with both contact points aligned in the absence of applied stress. This alignment relies on the topological scan for data collection, eliminating the need for optical trackers.[00243J In some embodiments, a third step 706 of the method 700 of calculating joint gaps based on changes in alignment between the femur and tibia under applied stress is applying stress to the affected joint. In some embodiments, for example to ultimately calculate the lateral gap 726, sub-step 706a is to apply positive varus stress 728 to the affected joint to open the lateral side and create stable contact 730 on the medial side, for example as shown in Fig. 27. Similarly in some embodiments, for example to ultimately calculate the medial gap 732, sub-step 706b is to apply positive valgus stress 734 to the affected j oint to open the medial side and create stable contact 736 on the lateral side, for example as shown in Fig. 28. By way of example, the CAAT system 10 may be configured to continuously display the F / E 720, V / V 722, and int / ext 724 rotation angles based on ongoing topological scans, providing real-time feedback so the surgeon can correct minor misalignments in F / E or int / ext rotation if needed.

[0244] In some embodiments, a fourth step 708 of the method 700 of calculating joint gaps based on changes in alignment between the femur and tibia under applied stress is topological data collection through structured light scanning. For example, in some embodiments, once the knee is stabilized, a series of scans may be collected to produce high-resolution data reflectingthe joint’s response to the applied stress. This scanning relies solely on topological data, removing the need for recalibration or line-of-sight maintenance typical of optical tracking.

[0245] In some embodiments, a fifth step 710 of the method 700 of calculating joint gaps based on changes in alignment between the femur and tibia under applied stress is measuring change in the V / V angle 722. By way of example, the change in the V / V angle (A9) may be calculated by comparing the currently measured angle to the established baseline under no-stress condition. By way of example, for a varus stress application 728, the angle change A9 quantifies the lateral opening or gap 726 of the joint while the medial side maintains contact under varus stress. Similarly, for a valgus stress application 734, the angle change AO quantifies the medial opening or gap 732 of the joint while the lateral side maintains contact under valgus stress.

[0246] In some embodiments, a sixth step 712 of the method 700 of calculating joint gaps based on changes in alignment between the femur and tibia under applied stress is continuous tracking for peak gap measurement. By way of example, achieving an accurate joint laxity assessment requires capturing the peak values for both the medial gap 732 and lateral gap 726. Since consistent force application is challenging over time, only continuous tracking of the femur 274 and tibia 276 in real time can reliably identify the peak gap values. By continuously tracking both bones during force application, the CAAT system 10 ensures that peak gap values are recorded for both sides of the joint, providing a full view of joint laxity based on maximum displacement under stress.

[0247] In some embodiments, a seventh step 714 of the method 700 of calculating joint gaps based on changes in alignment between the femur and tibia under applied stress is determining the known articular contact distance (D). By way of example, this may be accomplished by measuring or estimating the distance between medial and lateral contact points (e.g., between the femoral condyles and tibial plateau), serving as the effective lever arm for calculating the lateral and medial gaps 726, 732. In some embodiments, the articular contact distance (D) may be estimated by the CAAT system 10 based on topological data collected by one or more scans acquired by the 3D scanner 12, which may provide the articular contact distance (D) based on cartilage contact. In some embodiments, the articular contact distance (D) may be estimated from the pre-operative CT-scan tibia and femur data registered to topological data collected byone or more scans acquired by the 3D scanner 12, which may have closest-approaching bones. By way of example, if the registered femur and tibia do not meet in the tracking of the CT-based bone, a presumption may be that cartilage is between them. Once this contact distance is determined, no further distance measurements are required.

[0248] In some embodiments, an eighth step 716 the method of calculating joint gaps based on changes in alignment between the femur and tibia under applied stress is applying a trigonometric calculation to calculate the lateral and medial gaps 726, 732. By way of example, using the known articular contact distance and V / V angle change AO, the lateral gap 726 (GL) may be calculated as:GL = D x tan(AO) where D is the articular contact distance, and AO is the change in the V / V angle from baseline when the affected j oint is under applied varus stress. Similarly, by way of example, the medial gap 732 (GM) may be calculated as:GM = D x tan(AO) where D is the articular contact distance, and AO is the change in the V / V angle from baseline when the affected j oint is under applied valgus stress.

[0249] By way of example, the calculated lateral gap value G represents the increase in space on the lateral side of the joint from the baseline zero-gap condition under applied varus stress. Similarly, the calculated medial gap value G represents the increase in space on the medial side of the joint from the baseline zero-gap condition under applied valgus stress. By way of example, these calculations quantify medial and lateral joint laxity without the need for tracker-based measurements. In some embodiments, this calculation is ultimately an approximation because the closest point varies with angle due to the bone surfaces being not planar. Because the CAAT system 10 is scanning the cartilage surface, it may be advantageous to directly determine the distances between the cartilage surfaces.

[0250] In some embodiments, because the CAAT system 10 determines the surfaces directly, the medial and lateral gaps maybe be determined numerically, for example from a registered CTscan, by calculating closest-points for each bone (or cartilage from registering optical scan data to the new bone positions).

[0251] Example[00252J Suppose an example environment in which (i) the no-stress baseline V / V angle 0baseiine=O°, (ii) under varus stress, the V / V angle changes to 0 stressed 5 , giving a positive angle change (e.g., A0 = ^stressed - ©baseline = 5°), and (iii) the distance D between medial and lateral contact points is measured at 60 mm. Using the calculation disclosed above, the gap can be calculated asG=60mm * tan(5°) « 5.2mmThis 5.2 mm lateral gap reflects the knee’s lateral joint laxity under varus stress, calculated exclusively from topological data without optical tracking. Using a gap balancing technique, a surgeon would then measure the gap medially and then modify the surgical plan by adjusting the implant position and orientation to achieve and equal gap on both sides of the knee.

[0253] The method of calculating joint gaps based on changes in alignment between the femur and tibia under applied stress described herein offers a relatively simple and efficient approach to calculating relative joint gaps by leveraging changes in the varus / valgus (V / V) angle and a known contact distance between the medial and lateral points of the joint surfaces. However, alternative methods for calculating relative gaps may also be used to meet specific needs or anatomical considerations in similar surgical applications. Some examples include cross-sectional area analysis and 3D surface distance mapping.

[0254] Regarding cross-sectional area analysis, in some embodiments, instead of using a single angle change and contact distance, a cross-sectional approach could be taken, where serial cross-sections of the joint are analyzed to calculate gap changes across a region. By capturing multiple measurements in parallel planes, the system can calculate an average or profde of gap changes across the lateral or medial side, providing a more detailed spatial representation of joint laxity.

[0255] In some embodiments, the 3D surface distance mapping method involves point-to- point or point-to-surface distance mapping across the entire 3D surface of the femur and tibia. In this approach, the system would calculate the average distance or minimum distance between the entire surfaces, providing an integrated measurement of the gap. This method can give a comprehensive picture of joint laxity by capturing the entire interaction surface rather than relying on a single angle or lever-arm distance.

[0256] These alternative methods are more computationally intensive but could provide greater detail in cases where understanding spatial variations in joint laxity across a broader area is beneficial.

[0257] While the presented example focuses on measuring relative joint gaps between the femur and tibia during a TKA procedure, the principles behind this method are broadly adaptable. This approach can be applied to measure relative gaps or angular changes between any two anatomical structures where topological scanning and registration to preoperative imaging data (e.g., CT, MRI, or an optical reference scan) are feasible. By capturing 3D surface data and aligning it with preoperative imaging, this approach can measure gaps, angular relationships, or relative displacements in other anatomical contexts. For example, in shoulder arthroplasty or repair, shoulder joint laxity, or relative gaps between the humeral head and glenoid cavity, could be calculated using similar principles. Changes in the shoulder's rotation or flexion angles under applied forces could be used to assess joint stability or soft tissue tension. In spinal procedures, relative measurements could track the gap or height between vertebrae by assessing changes under varying loads, angles of flexion, or introduction of implants. This approach could support intervertebral disk replacement or fusion procedures where maintaining precise intervertebral spacing is critical. As another example, during a hip replacement procedure the relative gap between the acetabulum and femoral head could be measured to assess joint stability and implant alignment. This measurement could help optimize soft tissue tension and balance around the joint.

[0258] These examples show that the method 700 is adaptable to measure relative geometric relationships across various anatomical structures, as long as 3D topological data can be captured and registered to preoperative imaging or optical reference scans if the optical reference scancaptures the relevant anatomy of the separations. This versatility broadens the method’s utility, making it applicable for multiple orthopedic and surgical contexts where joint alignment, laxity, or spacing measurements are essential for procedural success. Thus, the method 700 disclosed herein provides flexible, accurate, and non-invasive measurement capabilities for assessing relative gaps or distances between specific points of anatomical structures, enhancing intraoperative guidance and potentially improving patient outcomes across multiple surgical fields.

[0259] There are several benefits to real-time gap measurement. For example, the ability to track the femur and tibia in real time allows for dynamic gap measurements under varying loads, providing more accurate and clinically relevant data than static measurements. Additionally, real-time tracking ensures that the CAAT system 10 captures the peak gap values, which are critical in determining the balance between the medial and lateral sides of the knee.Furthermore, since the leg and knee joint are moving throughout the measurement process, continuous tracking allows the system to account for these movements and still produce accurate results.

[0260] Tracking Anatomy with a Robot-Assisted Surgical Instrument

[0261] A more complex use case for tracking arises when a surgical instrument is attached as the end effector of a robot. In this scenario, the position and orientation of the instrument must be continuously updated to accommodate any anatomy movement. An example of this in TKA is when an oscillating saw is attached to a robotic system, and the robot assists the surgeon by positioning the saw in the correct plane for a planned osteotomy. By way of example, Fig. 4 illustrates an operating environment 22 in which a surgical instrument 28 is positioned on the end of a robotic arm 30.

[0262] By way of example, the robot’s function is to prevent the saw blade from drifting outside of the predefined cut plane, maintaining specific translational and angular tolerances. This process is straightforward if the knee is rigidly fixed. However, in practice, the knee may move slightly due to various factors such as imperfect manual stabilization or surgeon manipulation. As the knee moves, the robot must continuously adjust to follow the anatomy and ensure the osteotomy remains accurate. Traditionally, this can be achieved using OpticalMarkers (OMs) and Optical Trackers (OTs) (e.g., unless they get bumped), but it can also be accomplished using real-time topological data collected from the 3D scanner 12 without the use of OMs and / or OTs. For example, Fig. 29 is a block diagram illustrating an example method 370 of tracking movable anatomy and a movable surgical instrument configured to affect the movable anatomy, according to some embodiments of the disclosure. In some embodiments, this method 370 is essentially an extension of the multi-object tracking described previously, but with the additional complexity of having to communicate the anatomy’s movement to the robot so that the position of the end effector can be updated in real time.

[0263] In this example scenario, multiple objects are tracked simultaneously. Thus, in some embodiments, a first step 372 in the method 370 is to acquire topological scan data (e.g., using the 3D scanner 12) of the target anatomy (e.g., a femur) and the surgical instrument 28 and generate reconstructed 3D point clouds of each, for example using techniques described above. In some embodiments, a second step 374 of the method 370 of tracking movable anatomy and a movable surgical instrument configured to affect the movable anatomy is to independently register the target anatomy and the surgical instrument 28 on the end of the robot arm 30 in the CAAT system 10 as described above, for example using the acquired and reconstructed scan data in addition to reference data 376 of the target anatomy and surgical instrument. In some embodiments, the reference data 376 of the target anatomy may be acquired from preoperative imaging (e.g., CT imaging) or data collected intraoperatively. In some embodiments, the reference data 376 of the surgical instrument 28 may be acquired from preoperative CAD data, preoperative scans, intraoperative scans, and / or a fiducial marker if the instrument does not scan well. In some embodiments, the origin point (or other point of interest) of the instrument 28 is identified in the data set or calibrated at the time of surgery. In some embodiments, for example, a fiducial may be registered relative to the tool in order to calibrate its position relative to the end effector.

[0264] In some embodiments, a third step 378 in the method 370 is to use transformation matrices to convert the target anatomy position and orientation into robot space. By way of example, the robot's base remains fixed, and a point on the base is defined as the origin of the surgical coordinate system (or “SCS”), which in this example may also be referred to as "robot space." In some embodiments, all other objects (3D scanner 12, instrument (e.g., saw) 28, femur274, etc.) may be transformed into the robot space to ensure alignment. By way of example, by tracking the femur, its position can be continuously transformed into the robot space. In some embodiments, an optional step 380 of the method 370 may be performed to smooth the input data to reduce the effects of noise before the data is sent to the robot, for example using a moving average and / or Kalman fdter. In some embodiments, because the robot base might inadvertently move relative to the patient, the common coordinate system may preferentially be the patient anatomy itself (either femur or tibia, for example). This is advantageous and more direct than the common space being trackers that have been attached rigidly to bone, for example.

[0265] In some embodiments, a fourth step 382 in the method 370 is to send the target anatomy position and orientation to the robot. By way of example, the instrument 28 (e.g., saw blade) can be tracked either directly using the 3D scanner 12 as disclosed herein or by relying on data from the robot's internal control system, which already knows the position of its end effector (and hence the saw blade) in robot space. However, for the CAAT system 10 to navigate with the scanner 12, the corresponding position in scanner space would need to be known. Thus, the robot space will need to have been registered to the scanner space, which may be done by preliminarily scanning and registering the instrument 28 in a few positions. After that the CAAT system 10 can transform the robot-reported position of the end effector to the space of the scanner. In that way the CAAT system 10 determines where the instrument 28 is in scanner space, even if the instrument 28 is not within the field of view of the scanner.

[0266] In some embodiments, a practical result of the method 370 is that the position of the target anatomy is continuously tracked as it moves during surgery, for example using the CAAT system 10 and the 3D scanner 12. In some embodiments, the updated position of the target anatomy is continuously communicated to the robot 30, which as a fifth step 384 in the method 370, may intraoperatively plan the optimal kinematic motion path to maintain relative position to the target. Stated differently, the robot may adjust the instrument 28 (e.g., saw blade) movement in real time to maintain alignment with the target anatomy and the predefined cut plane. By way of example, the robot effectively compensates for any slight anatomical shifts, ensuring the accuracy of the osteotomy.

[0267] This real-time adjustment can be performed within predefined translational and angular tolerances, providing the surgeon with a more precise and reliable way of cutting withina moving surgical field. Depending on the design of the robot and its positional control system, it may be necessary to update the registration data at a rate faster than 20 registrations per second (rps) to achieve a finely tuned control system.

[0268] In some embodiments, a sixth step 386 in the method 370 is to execute the motion path of the instrument with minimal latency between the time the 3D topological data is captured to when an updated transformation matrix is sent to the robot. In such a use case, latency becomes critical. Any delay in updating the robot’s position relative to the anatomy can lead to the saw blade falling out of alignment with the femur, reducing the accuracy of the cut. Minimizing latency between the time the 3D topological data is captured and when the updated transformation matrix 2 is sent to the robot is essential for maintaining real-time performance. For instance, if the system operates at 30 registrations per second (rps) and the latency is 50 milliseconds (ms), then the robot will get data on its new position 50 ms after the bone being tracked was there. Additionally, there may be additional latency introduced by the robot’s control system (e.g., the robot needs to calculate an optimal path, engage motors, accelerate, and finally reach the intended point). If, for example, it takes the robot 20 ms to perform these tasks, the total lag would be 70 ms. Thus, it is essential to reduce both sources of lag in a real-time tracking system to ensure that the robot can respond in real time to anatomical movements.

[0269] In some embodiments, a seventh step 388 in the method 370 is to determine whether tracking of the target anatomy and surgical instrument needs to continue. If the determination is affirmative, then the workflow cycles back to step 372 of acquiring and reconstructing topological scan data of the target anatomy. If the answer is no (e.g., the osteotomy has been completed), then the system may end the tracking (e.g., box 389).

[0270] By increasing the registration rate and reducing computational latency, the system can deliver accurate, real-time adjustments to the robot, ensuring that the saw blade remains in the correct plane even as the target anatomy moves. This approach maintains the accuracy of the osteotomy and helps prevent unintended deviations, improving surgical outcomes.

[0271] Anatomy Tracking and Robot Following with a Mounted Scanner

[0272] In previous examples, the 3D scanner 12 was positioned at a fixed location. However, mounting the scanner on the robotic arm 34 (e.g., as shown by way of example in Fig. 4) allows the 3D scanner’s 12 position and field of view 68 to change dynamically as the robot 34 moves. This configuration offers significant advantages during surgical procedures where the anatomy or surgical site frequently shifts.

[0273] For instance, in a Total Knee Arthroplasty (TKA) procedure, the knee is often moved by the surgeon, transitioning from full extension to 90 degrees of flexion, and sometimes into full flexion. While a fixed 3D scanner 12 can capture the surgical site in one position, maintaining a clear view may become challenging as the anatomy moves. If the knee or other anatomical structures shift out of the 3D scanner 12’ s field of view 68, manual repositioning of the scanner 12 with respect to the affected knee would be required, interrupting the workflow.

[0274] By mounting the 3D scanner 12 on the robot’s arm 34 to move along with the end effector (or by itself on a robot arm 34 independent of an end effector, as shown by way of example in Fig. 4), the system can continuously track bones like the femur or tibia and automatically adjust the scanner’s position to maintain a fixed perspective on the surgical site. As the knee moves through different angles, the robot dynamically updates the scanner's position, keeping the anatomy within the field of view. This automation reduces the need for manual intervention, saving time and effort. In some embodiments, the robot may be provided with one or more degrees of freedom to enable movement of the 3D scanner 12. By way of example, the robot shown in Fig. 4 includes seven (7) degrees of freedom, however robots with fewer degrees of freedom (e.g., 1, 2, 3, etc.) or more degrees of freedom (e.g., 8, 9, 10, etc.) may be used as may be needed or available.

[0275] In addition to maintaining a fixed view, the robot-mounted scanner synchronizes its movements with the anatomy in real-time. This optimizes scanner positioning throughout the procedure, allowing the surgeon to focus on the operation without worrying about viewpoint adjustments. In some embodiments, multiple viewpoints can be pre-programmed or customized for different stages of the surgery, with the surgeon easily switching between them via the user interface (UI), further enhancing procedural efficiency.

[0276] Anatomy Tracking with Robot-Controlled Instrument (Oscillating Saw Example)

[0277] In this use case, both a 3D scanner 12 and an oscillating saw 28 are mounted to a robot arm. By way of example, the 3D scanner 12 tracks the anatomy while the robot positions the saw 28 to make precise cuts along a predefined cut plane, which may be modified intraoperatively based on the surgeon’s real-time observations. In some embodiments, if the 3D scanner 12 position changes as the robot moves, it may be necessary to apply one or more transformations to ensure that the anatomical data, 3D scanner 12, robot base, and saw 28 remain aligned in the same coordinate space. In some embodiments, the common coordinate system may be referenced to the patient, the 3D scanner 12, the robot base, or to another object.

[0278] By way of example only, Figs. 30A-B depicts a block diagram illustrating a method 390 of complex anatomy tracking with robot-controlled instrument in the context of a TKA, where the robot assists with cutting the femur.

[0279] In some embodiments, a first step 392 of the example method 390 of complex anatomy tracking with robot-controlled instrument is to mount the scanner 12 and the saw or saw blade 28 to a robot end effector 28, which by way of example may be an articulating arm having multiple degrees of freedom. In some embodiments, a next step 393 in the method 390 is to move (or initiate the robot to move) the robot arm to a known position in which the scanner 12 faces the robot base. In some embodiments, a next step 394 in the method 390 is to use the scanner 12 to acquire a dataset comprising structured light image data of the robot base (“robot base image dataset”). In some embodiments, a next step 395 in the method 390 is to upload a dataset comprising the robot base CAD geometry to the CAAT system 10 (“robot base CAD dataset”). In some embodiments, a next step 396 in the method 390 is to register the robot base image dataset to the uploaded robot base CAD dataset. In some embodiments, a next step 397 in the method 390 is to compute a transformation matrix T(B / S) (transformation matrix of the robot base dataset to the common coordinate system) at this fixed position to relate the robot base to the common coordinate system. In some embodiments, a next step 398 in the method 390 may be to track or continuously monitor robot position sensors and update the transformation matrix T(B / S) in real time.

[0280] By way of example, the above steps establish a robot base position in the common coordinate system. As previously mentioned, all other objects (3D scanner 12, instrument (e.g.,saw) 28, femur 274, etc.) are transformed into the common coordinate system to ensure alignment (e.g., not necessarily in that order, and not necessarily after establishing the robot base position in the common coordinate system as described). By way of example, the robot's base remains fixed, and in some embodiments a point in the common coordinate system may be referenced to the base. In some embodiments, the scanner 12 may be used for correcting errors resulting from base-frame drift. For example, after robot-patient registration, in the absence of any motion of a tracker attached to the patient, the end-effector is known relative to the position of the base based on its own localizers. However, the end-effector position may become inaccurate relative to the patient if the robot base is not perfectly rigid with respect to the patient, even if the patient does not move in any way. So, even though the robot-reported position is used once the system 10 is calibrated, the scanner 12 has the additional capability of checking the actual relative location of the end-effector with respect to the patient. That way, if the base flexes, creating a millimetric deviation at the end-effector (for example), the CAAT system 10 can capture that and correct the relative end-effector location relative to the patient.

[0281] In some embodiments, the following steps of the method 390 of complex anatomy tracking with robot-controlled instrument may be followed to relate the position of the saw 28 or saw blade on the robot end effector to the common coordinate system. In some embodiments, this may be accomplished by expressing the position of the robot end effector in real time as a transformation matrix T(SB / S) (transformation matrix of the saw blade dataset to the common coordinate system). By way of example, the position of the robot's end effector may be known in real-time through the robot control system. This transformation matrix T(SB / S) is used to relate the end effector’s (and therefore the sawblade’s) position to the common coordinate system.

[0282] In some embodiments, a next step 399 in the method 390 is to use the 3D scanner 12 to acquire a structured light image dataset of the saw blade (“saw data set”) if the saw blade is positioned within the camera’s field of view or a saw blade calibration tool if the saw blade is not within the camera’s field of view. In some embodiments, a next step 400 in the method 390 is to upload a saw blade or saw blade calibration tool CAD geometry dataset to CAAT system 10. In some embodiments, a next step 401 in the method 390 is to register the saw dataset to the uploaded CAD geometry dataset. In some embodiments, a next step 402 in the method 390 is tocompute a transformation matrix T(SB / S) (transformation matrix of the saw dataset to the scanner space) at the fixed position. In some embodiments, a next step 404 in the method 390 is to track or continuously monitor robot position sensors and update the transformation matrix T(SB / S) in real time.

[0283] In some embodiments, the following steps of the method 390 of complex anatomy tracking with robot-controlled instrument may be followed to register and track the femur position in real-time within the common coordinate system. In some embodiments, the CAAT system 10 may accomplish this by producing the transformation matrix T(F / S) (transformation matrix of the femur to the common coordinate system).

[0284] In some embodiments, a next step 406 in the method 390 is to manually move the robot to position where the scanner faces the femur. In some embodiments, a next step 408 in the method 390 is to use the 3D scanner 12 to acquire a structured light image dataset of the femur (“femur dataset”), which will include a subset data representative of femoral cartilage. In some embodiments, a next step 410 in the method 390 is to determine whether a reference dataset (e.g., preoperative CT scan) of the femur is available. If yes, then in some embodiments, a next step 412 in the method 390 is to upload a segmented reference dataset of the femur to the CAAT system 10. In some embodiments, a next step 414 in the method 390 is to register the femur / femoral cartilage scan dataset to the reference (e.g., CT) dataset, for example using one or more methods described herein. If there is no available reference dataset, then in some embodiments, a next step 416 in the method 390 is to create a femur / femur cartilage model dataset from the femur dataset obtained by intraoperative scanning. In some embodiments, a next step 418 in the method 390 is to register the created model dataset to the femur / femoral cartilage scan dataset, for example using one or more methods described herein. In some embodiments, a next step 420 in the method 390 is to compute a transformation matrix T(F / S) to transform this registration into the common coordinate system. In some embodiments, a next step 422 in the method 390 is to continuously repeat the previous steps (e.g., scan femur / femoral cartilage, register the scan dataset to the reference dataset, and recompute the transformation matrix T(F / S) in real time to track or monitor the position of the femur in the common coordinate system.

[0285] In some embodiments, the registrations described above may be performed in any order, since the goal of registering each of the robot, target anatomy (e.g., femur), surgical instrument (e.g., saw blade) and / or scanner into a common coordinate system is the same regardless of which occurs first. For example, in some embodiments, it may be advantageous to register the patient anatomy first, using the patient anatomy as the origin of the common coordinate system to minimize the need to re-register the patient anatomy in the event of movement.

[0286] In some embodiments, the following steps of the method 390 of complex anatomy tracking with robot-controlled instrument may be followed to register the surgical plan (e.g., cut plane) into the common coordinate space and perform the surgical cut. By way of example, the cut plane is initially determined during preoperative planning and adjusted during surgery based on patient-specific anatomy and knee kinematics. In some embodiments, the cut plane may be defined relative to the femur.

[0287] In some embodiments, a next step 424 in the method 390 is to load a preoperative CT dataset and / or femur image data into a surgical planning tool and determine proper cut planes in the common coordinate system. In some embodiments, a next step 426 in the method 390 is to continuously convert robot base, scanner, and saw blade into the common coordinate system by applying the correct respective transformation matrices and / or their inverses as needed. In some embodiments, a next step 428 in the method 390 is to continuously update the robot’s position to validate the position of the robot with respect to the anatomy, for example to ensure the saw blade remains in the cut plane at all times even with movement of the femur. In some embodiments, a next step 429 in the method 390 is to manually power and move the saw blade within the cut plane as needed to execute the cut as planned.

[0288] By way of example, all of these transformation matrices are computed in real-time, and the data is continuously fed into the robot's control system, to maintain the saw’s alignment within the predefined cut plane (for example) while adjusting its position based on real-time anatomical movement. This allows the surgeon to follow the operative plan, for example to manipulate the saw within the plane while the robot ensures that the blade remains within the precise boundaries of the cut plane.

[0289] It is important to note that the same principles of transforming anatomy and objects into a common coordinate space apply to all the previous use cases described but the equations were not explicitly given. Whether it’s tracking a single object like the femur, maintaining a stable field of view, or synchronizing a scanner and robot, the underlying process of sequentially applying transformation matrices remains a key part of the system. These transformations ensure that all components of the system — whether anatomy, surgical instruments, or the robot — are correctly aligned within the SCS, enabling precise, real-time interactions.

[0290] By way of example, Fig. 31 is a block diagram depicting an example novel TKA workflow 430 with pre-operative imaging using the CAAT system 10 of the present disclosure. By way of example, the workflow 430 may be divided into several portions, including a preoperative portion 432, OR setup portion 434, initial registration portion 436, surgical planning portion 438, Bone resection portion 440, and implantation portion 442.

[0291] Regarding the pre-operative portion 432, prior to surgery, detailed imaging 444 of the target anatomy is often acquired using CT (Computed Tomography) or MRI (Magnetic Resonance Imaging). In some embodiments, these scans may be segmented to create a 3D model of the relevant structures, such as the femur and tibia in TKA. These 3D models are essential for pre-operative planning 446 of the procedure and aligning the patient's real-time anatomy with the surgical plan. Important anatomical planes and landmarks are also labeled within the model to assist the surgeon during the operation.

[0292] For this workflow, we assume a patient-specific CT scan has been segmented to provide a detailed 3D model of the femur and tibia, though the imageless mode can also be utilized if necessary.

[0293] By way of example, the OR setup portion 434 may include patient positioning 448, 3D scanner setup 450, robot setup 452, and instrument setup 454. Before surgery begins, the CAAT system 10 requires calibration. In some embodiments, scanner setup 450 comprises one or more 3D scanners 12 being positioned to ensure full visibility of the surgical field. Calibration ensures that the CAAT system 10 aligns correctly with the surgical setup, accounting for distances, camera angles, and the positioning of all equipment.

[0294] Robotic equipment requires additional steps for calibration to ensure that the robot functions correctly. The robot setup 452 process includes aligning the robot's physical movements with the virtual model of the surgery to maintain accuracy and ensure that the robot performs as expected during the procedure.

[0295] In some embodiments, patient positioning 448 occurs after setup and calibration. By way of example, in a TKA procedure the patient is positioned on the operating table, typically in a supine position with the knee exposed for TKA procedures, as shown by way of example in Fig. 4. The surgical site is prepped and draped in a sterile fashion.

[0296] By way of example, the initial registration portion 436 may include surgical exposure 456, scanning 458, bone to cartilage registration 460, hip center calculation 462 via tracking, and ankle center calculation 464 using a pointer. In some embodiments, upon completion of the OR setup 434 portion, surgical exposure 456 may occur, with an incision made to access the knee joint and prepare it for the robotic system to perform the procedure.

[0297] In some embodiments, femoral registration may occur as described above, by using the 3D scanner to generate a 3D point cloud model registered to the SCS. This real-time anatomical model is maintained within the Surgical Coordinate System (SCS), enabling accurate tracking of the patient’s anatomy throughout the procedure. In some embodiments, the CAAT system 10 may perform bone to cartilage registration 460, including misalignment correction as described below.

[0298] Following patient registration, the surgeon may identify key anatomical landmarks necessary for establishing the mechanical axis of the leg, for example including the hip center 462 (e.g. identified by rotating the leg while the CAAT system 10 calculates the center of rotation of the femoral head as described above), and / or ankle center 464 (e.g., identified using the optical tracking system on the medial and lateral malleoli (ankle bones) to calculate the center of the ankle joint or the pointer instrument described below). These landmarks, or others, or measured curvatures, for example, are crucial for ensuring proper implant alignment. The mechanical axis is one common methodology, but other alignment methods, like kinematic alignment, may use different anatomical landmarks depending on the procedure.

[0299] At this stage, a virtual surgical plan 438 is developed to guide the robotic system and surgeon throughout the procedure. If pre-operative planning was done, it serves as an initial basis but must be verified and adjusted in real time as the surgery progresses.

[0300] By way of example, the virtual surgical plan 438 may involve the implant selection 466, initial implant positioning 468, gap balancing via tracking 470, positioning adjustments 472, and cut plane definitions 474. Regarding implant selection 466, in some embodiments the CAAT system 10 may assist the surgeon in selecting the optimal implant size based on the patient’s anatomy. Proper implant sizing ensures both the long-term success of the surgery and post-operative joint function. Regarding implant positioning, in some embodiments the CAAT system 10 may help the surgeon determine the ideal implant position to achieve accurate joint alignment. The goal is to ensure proper leg alignment and balance between the medial and lateral sides of the knee. The alignment is measured in terms of varus (inward tilt) and valgus (outward tilt), with the system assisting the surgeon in making appropriate adjustments. In some embodiments, gap balancing 470 ensures equal tension in the soft tissues (primarily ligaments) around the knee joint throughout its range of motion. By way of example the CAAT system 10 dynamically tracks the relative positions of the bones and calculates the space between them to guide the surgeon in balancing the medial and lateral gaps during the procedure, as described above. At this time, any positioning adjustments 472 should be made. Regarding defining the cut plane 474, in some embodiments based on the selected implant size and position, the system defines the necessary osteotomies (bone cuts). These cuts must be precise to ensure proper implant placement.

[0301] In some embodiments, during the bone resection 440 portion, the CAAT system 10 assists the surgeon in making precise cuts according to the surgical plan. The system continuously tracks the patient's anatomy and provides feedback, ensuring the cuts are executed accurately. After each bone cut, the system verifies the accuracy of the cuts, implant alignment, and gap balancing. If deviations are detected, the system alerts the surgeon, allowing for realtime adjustments to be made. By way of example, this cut and verify process occurs for each osteotomy cut, including anterior 476, posterior 478, distal 480, posterior chamfer 482, anterior chamfer 484, and / or tibia 486.

[0302] In some embodiments, after bone cuts are verified, the workflow 430 proceeds with the implantation 442 portion. In some embodiments, trial implants 488 are placed to assess the fit, alignment, and joint movement. If necessary, adjustments are made before the final implant 490 is inserted. In some embodiments, the surgeon then performs a final assessment (e.g., including range of motion tests 492) to ensure proper alignment and stability of the new joint. Once confirmed, the surgical site is closed 494, and the patient is prepared for recovery.

[0303] By way of example, Fig. 32 is a block diagram depicting an example novel TKA workflow 630 using the CAAT system 10 of the present disclosure without pre-operative imaging. By way of example, the workflow 630 may be divided into several portions, including a pre-operative portion 632, OR setup portion 634, initial registration portion 636, surgical planning portion 638, Bone resection portion 640, and implantation portion 642.

[0304] Regarding the pre-operative portion 632, prior to surgery, detailed imaging 444 of the target anatomy is often acquired using CT (Computed Tomography) or MRI (Magnetic Resonance Imaging). In some embodiments, these scans may be segmented to create a 3D model of the relevant structures, such as the femur and tibia in TKA. These 3D models are essential for pre-operative planning 446 of the procedure and aligning the patient's real-time anatomy with the surgical plan. Important anatomical planes and landmarks are also labeled within the model to assist the surgeon during the operation.

[0305] Regarding the pre-operative portion 632, prior to surgery, a standard 3D model 646 of the bones may be used instead of patient-specific scans described in the previous workflow 430. Anatomical landmarks, such as the medial and lateral epicondyles of the femur, are identified during surgery using the 3D scanner 12 as described above. In some embodiments, the distance between these landmarks is used to scale the standard model to match the patient’s unique anatomy, allowing the surgeon to visualize implant placement with appropriate proportions, even in the absence of patient-specific imaging. These 3D models 646 may be used for pre-operative planning 644 of the procedure and aligning the patient's real-time anatomy with the surgical plan. Important anatomical planes and landmarks are also labeled within the model to assist the surgeon during the operation.

[0306] By way of example, the OR setup portion 634 may include patient positioning 648, 3D scanner setup 650, robot setup 652, and instrument setup 654. In some embodiments, scanner setup 650 comprises one or more 3D scanners 12 being positioned to ensure full visibility of the surgical field and registration of the one or more 3D scanners 12 in a common coordinate system. In some embodiments, robot setup 652 may include registration of the robot base in the common coordinate system. In some embodiments, instrument setup may include registration of a surgical instrument within the common coordinate system, including but not limited to a saw blade (for example) attached as a robot end effector. By way of example, registration ensures that the various components of the CAAT system 10 align correctly to one another within the surgical setup, accounting for distances, camera angles, and the positioning of all equipment.

[0307] In some embodiments, patient positioning 648 occurs after setup and calibration. By way of example, in a TKA procedure the patient is positioned on the operating table, typically in a supine position with the knee exposed for TKA procedures, as shown by way of example in Fig. 4. The surgical site is prepped and draped in a sterile fashion.

[0308] By way of example, the initial registration portion 636 may include surgical exposure 656, scanning and segmentation 658, articular surface matching (ASM) registration 660, and initial size and position setup 662. In some embodiments, upon completion of the OR setup 634 portion, surgical exposure 656 may occur, with an incision made to access the knee joint and prepare it for the robotic system to perform the procedure.

[0309] In some embodiments, femoral registration may occur as described above, by using the 3D scanner 12 to generate a 3D point cloud model registered to the common coordinate system. This real-time anatomical model is maintained within the common coordinate system, enabling accurate tracking of the patient’s anatomy throughout the procedure. By way of example, during ASM registration 660, the CAAT system 10 analyzes intraoperatively acquired topological data of the patient’s articular surfaces, and automatically compared them to digital models of available implants, assessing which implant design and size will best match the patient’s unique anatomy (e.g., initial size and position set 662).

[0310] At this stage, a virtual surgical plan 638 is developed to guide the robotic system and surgeon throughout the procedure. If pre-operative planning was done, it serves as an initial basis but must be verified and adjusted in real time as the surgery progresses.

[0311] By way of example, the virtual surgical plan 638 may involve gap balancing via tracking 670, positioning adjustments 672, and cut plane definitions 674. In some embodiments, gap balancing 670 ensures equal tension in the soft tissues (primarily ligaments) around the knee joint throughout its range of motion. By way of example the CAAT system 10 dynamically tracks the relative positions of the bones and calculates the space between them to guide the surgeon in balancing the medial and lateral gaps during the procedure, as described above. At this time, any positioning adjustments 672 should be made. Regarding defining the cut plane 674, in some embodiments based on the selected implant size and position, the system defines the necessary osteotomies (bone cuts). These cuts must be precise to ensure proper implant placement.

[0312] In some embodiments, during the bone resection 640 portion, the CAAT system 10 assists the surgeon in making precise cuts according to the surgical plan. The system continuously tracks the patient's anatomy and provides feedback, ensuring the cuts are executed accurately. After each bone cut, the system verifies the accuracy of the cuts, implant alignment, and gap balancing. If deviations are detected, the system alerts the surgeon, allowing for realtime adjustments to be made. By way of example, this cut and verify process occurs for each osteotomy cut, including anterior 676, posterior 678, distal 680, posterior chamfer 682, anterior chamfer 684, and / or tibia 686.

[0313] In some embodiments, after bone cuts are verified, the workflow 630 proceeds with the implantation 642 portion. In some embodiments, trial implants 688 are placed to assess the fit, alignment, and joint movement. If necessary, adjustments are made before the final implant 690 is inserted. In some embodiments, the surgeon then performs a final assessment (e.g., including range of motion tests 692) to ensure proper alignment and stability of the new joint. Once confirmed, the surgical site is closed 694, and the patient is prepared for recovery.

[0314] Anatomical Point Location Without Optical Trackers

[0315] By way of example, in cases where neither direct measurement (e.g., as disclosed herein in relation to gap measurement) nor inferred calculations (e.g., as disclosed herein in relation to calculating the hip center of rotation) are possible - such as when the anatomical landmark is entirely outside the scan field and no calculation model is feasible - another approach may be necessary. By way of example, this disclosure introduces a method to gather anatomical data beyond the scanner’s field of view without relying on scanned anatomic reference data or visible calculations. In some embodiments, this method may be used with a specialized pointer instrument with a unique handle geometry and / or finial feature, designed to stay visible to the scanner even as the pointer contacts anatomical landmarks outside the scan field. The system registers the finial’s position and, using the pointer’s known geometry, infers the precise location of the anatomical landmark.

[0316] For example, in TKA, identifying the center of the ankle is crucial for aligning the lower limb during surgery. In some embodiments, the surgeon can position a distal tip of pointer at the medial and lateral malleoli of the patient’s ankle 6, with the proximal end having a unique geometry and / or finial remaining within the scanner’s view (e.g., near the patient’s knee 4) (e.g., Fig. 33). By registering the finial to the common coordinate space (e.g., which may be the scanner space) and applying an anatomical offset based on the pointer’s geometry, the CAAT system 10 may calculate the ankle center as a reference for alignment. This process eliminates the need for complex optical tracking systems and does not rely on any calculations from visible anatomy, making it suitable for locating points completely beyond the scan field.

[0317] The differences among the various methods may be explained as follows. By way of example, the method 700 described above measures anatomical data directly within the scanner 12 field of view 80, such as joint laxity or gap measurements, by capturing high-resolution 3D surface data of the relevant structures. It requires that the anatomy of interest be within the scan field for real-time measurement, making it effective for intraoperative assessments but limited for points outside this range. By way of example, the method 320 disclosed above infers anatomical data beyond the scanner’s field of view through indirect calculations based on scanned reference points. This method applies when the anatomical landmark can be estimated mathematically from visible structures, such as calculating the hip center of rotation using a sphere-fitting algorithm on femoral head data visible within the scan field. By way of example,the method 800 disclosed herein gathers anatomical data for points beyond the scan field without requiring direct measurement or mathematical inference. In some embodiments, this method may use a pointer instrument with a visible finial to register the pointer’s position within the common coordinate system. The known geometry of the pointer enables determination of a precise location of anatomical points that lie entirely outside the scanner’s field of view, such as the ankle center, without relying on visible reference data or optical trackers.

[0318] In this way, the method 800 disclosed herein extends the capabilities of image-guided navigation by providing a solution for point localization in scenarios where neither direct measurement nor inferred calculations are viable. This innovation facilitates more versatile, accurate landmark identification, broadening the scope of intraoperative navigation options in complex orthopedic procedures.

[0319] Fig. 33 depicts an example of a pointer instrument 750 configured for use with the CAAT system 10 to identify specific anatomical points that are beyond the scanner’s field of view without performing calculations on visible anatomy or using optical trackers, according to some embodiments. In some embodiments, the pointer instrument 750 comprises a proximal end 752, a distal end 754, and an elongated shaft 756 extending between the proximal and distal ends 752, 754. In some embodiments, the proximal end 752 includes a handle 758. In some embodiments, the pointer instrument 750 includes a uniquely shaped finial 760 extending proximally from the handle 758. In some embodiments, the handle 758 may have a unique scannable surface geometry. In some embodiments, the pointer instrument 750 may be positioned such that the proximal end 752 including the handle 758 (e.g., with scannable surface geometry) and / or the finial 760 remains visible to the scanner 12 even as the distal end 754 of the pointer instrument 750 reaches points beyond the limits of the scanner 12 field of view. In some embodiments, the finial 760 may have a geometry that enables the CAAT system 10 to register its orientation and position accurately within the common coordinate space, ensuring that the inferred location of anatomical points is precise. In some embodiments, the finial 760 may be a single shape with a complex geometry. In some embodiments, the finial 760 may comprise a plurality of shaped ends clustered together. In some embodiments, as shown by way of example only in Fig. 33, the finial 760 may comprise one or more spheres.

[0320] In some embodiments, the distal end 754 may include a distal tip 762 configured for placement on, in, or near target anatomy. In some embodiments, the distal tip 762 may be pointed to facilitate penetration of the distal tip 762 into target anatomy. In some embodiments, the distal tip 762 may be blunt to ensure that the distal tip 762 does not penetrate the target anatomy. In some embodiments, the distal end 754 may have a bend or angular offset 764 from a longitudinal axis of the elongated shaft 756 to improve positioning of the distal tip 762 against target anatomy (e.g., the ankle).

[0321] In some embodiments, the elongated shaft 756 may be rigid. In some embodiments, the elongated shaft 756 may be straight. In some embodiments, the elongated shaft 756 may be curved. For example, there may be scenarios when the location of a hidden posterior anatomy may be desired while scanning with an anterior view. In that case a pointer tool 750 having a curved shaft 756 configured to extend around the anatomy culminating in a distal tip 754 positioned at the target (hidden) anatomy, may also present for the anterior view a finial that enables registration of the hidden anatomy. In some embodiments, it may be possible to localize anatomic features hidden from view due to a limited exposure, for example, whereby the pointer distal tip 762 is within the surgical site but outside the field of view of the 3D scanner 12, while the proximal end is visible for localization. In this case, the pointer instrument 750 itself would be made of materials suitable for contact with the patient tissue, and sterilizable.

[0322] In some embodiments, the geometry of the handle 758 and / or finial 760 may indicate the spatial orientation of the distal tip 762.

[0323] By way of example, Fig. 34 is a block diagram depicting several steps in a method 800 for identifying specific anatomical points that are beyond the scanner’s field of view without performing calculations on visible anatomy or using optical trackers, according to some embodiments. In some embodiments, the method 800 may comprise the following steps.

[0324] In some embodiments, a first step 802 in the method 800 is preparing the 3D scanner system for topological registration. By way of example, before the procedure begins, the 3D scanner system may be calibrated to capture and map visible anatomical surfaces within its field of view with high resolution, creating a precise 3D reference map. In some embodiments, this setup establishes a common coordinate system, which aligns with preoperative imaging orintraoperative reference models, providing a framework for accurate real-time tracking. In some embodiments, in the context of TKA, common coordinate system may be referenced to the 3D scanner 12, allowing the scanner 12 to serve as a stable reference frame for tracking the pointer and determining the ankle center. In some embodiments, as previously mentioned, the common coordinate system may be referenced to any object within the scanner’s field of view, including but not limited to the scanner 12, robot base, or a portion of the patient’s anatomy. In some embodiments, during this setup process the CAAT system 10 may be calibrated with precise specifications of the pointer instrument 750, including but not limited to length, orientation of the pointer tip 762 relative to a unique shape of the handle 758 or finial 760, and / or distance between the finial 760 and the distal tip 762 such that the CAAT system 10 can determine the precise location and orientation of the distal tip 762 when the finial 760 (or uniquely shaped handle 758) is positioned within the camera 12 field of view.

[0325] In some embodiments it may be necessary to calibrate the location of the distal tip 762 with respect to the finial 762, and not rely on precises specification. For example, the elongated shaft 756 may occasionally bend or get out of alignment over time and especially between cases. In some embodiments, the calibration procedure may entail 3D scanning along the length of the shaft 756 in a way that stitching of one more scan datasets may occur or be required, because it may be that the entirety of the pointer instrument 750 is not visible at once to the scanner 12. This could be facilitated by one or more regular markings on the pointer that are discernable at high resolution by the camera 12 and facilitate the dataset stitching. Practically, for the pointer instrument 750 to reach the ankle, only one dataset stitching may suffice. It is possible also that in addition to, or instead of, the markings, that unique 3D fiducial shapes be present on the pointer 750 that are visible to the scans being stitched, for purpose of facilitating the stitching. For this calibration procedure, the dataset stitching may occur using a subsection of the overall scanning volume that has been verified to be optimally accurate and precise, but that requires more stitching than would be required if the entire scanning volume were used. Otherwise, if the accuracy can maintain it, a single stitch could be used. To check the calibration, a known rotation of the pointer 750 could be applied and the calibration procedure performed again, to make sure that the calibration procedure results in a relative position of the tip to the finial is consistent with the rotation, and not biased by systematic inaccuracies in the scanner volume.

[0326] In some embodiments, a second step 804 in the method 800 is positioning the pointer instrument 750 such that the distal tip 762 is placed on the medial and lateral malleoli. By way of example, with topological registration active, the surgeon places the pointer’s tip 762 sequentially on the medial and lateral malleoli of the ankle - two key bony landmarks necessary for calculating the ankle center. Throughout this process, the finial 760 remains within the 3D scanner 12 field of view, enabling continuous registration of the pointer 750 position even though the actual landmarks are outside the scan field.

[0327] In some embodiments, a third step 806 in the method 800 is real-time scanning and finial registration. By way of example, the 3D scanner 12 registers the unique shape and position of the finial 760 in real-time, continuously capturing its exact location within the common coordinate system. By tracking the finial 760, the CAAT system 10 accurately infers the pointer’s 750 full spatial orientation and position relative to each malleolus, using the pointer’s known geometry to extrapolate the location of the tip 762.

[0328] In some embodiments, a fourth step 808 in the method 800 is calculating the ankle center. By way of example, after marking both the medial and lateral malleoli, the CAAT system 10 uses these two points to calculate the approximate midpoint. Since the ankle center does not lie precisely at this midpoint due to anatomical asymmetry, the CAAT system 10 may apply a slight medial and superior offset based on anatomical data, refining the calculation to determine the true ankle center location. This inferred point, accurately calculated within the scanner’s common coordinate system, provides the surgeon with a reliable reference for lower limb alignment.

[0329] In some embodiments, a fifth step 810 in the method 800 is using the calculated ankle center as a reference for navigation. By way of example, once calculated, the ankle center serves as a critical reference point for intraoperative navigation and alignment. The surgeon can use this landmark to ensure precise alignment in TKA or other orthopedic procedures. This approach eliminates the need for additional optical trackers and provides a reliable, straightforward method for integrating anatomical data beyond the scan field into the surgical workflow.

[0330] By way of example, the method 800 for identifying specific anatomical points that are beyond the scanner’s field of view without performing calculations on visible anatomy or using optical trackers disclosed herein applies broadly in TKA to locate necessary alignment points, such as the ankle center, using the medial and lateral malleoli as references (for example). However, this approach is adaptable to any procedure requiring anatomical point localization beyond the scan field, providing versatile use in various orthopedic and alignment-driven surgeries. By eliminating optical trackers and additional equipment, this method reduces setup complexity and enhances efficiency in the surgical workflow.

[0331] Unlike traditional tracking systems, the method 800 for identifying specific anatomical points that are beyond the scanner’s field of view without performing calculations on visible anatomy or using optical trackers disclosed herein achieves anatomical point localization without optical markers, allowing consistent reference even outside the scan field. No intraoperative calibration or setup is required for the pointer (e.g., the calibration described above may be done pre-operatively or optionally as regular check of pointer accuracy), reducing potential disruptions due to tracker shifts or line-of-sight interruptions. The fixed geometric reference from the finial 760 enables accurate inference of points like the ankle center, integrating any anatomical offsets automatically.

[0332] In some embodiments, for example, two 3D scanners may be used simultaneously, with one scanner essentially being the tip of the pointer. For example, a finial may be attached to a rod, and at the end of the rod could be a 3D scanner. In some embodiments, this scanner may scan the face of a cranial surgery patient in a prone position (face-down), where the face is out of view of personnel. By way of example, the finial could extend up and be scanned by a second scanner, while scanning the back of the head of the patient, thereby using the face to register the patient, and transferring that registration to its field of view (the back of the head). This is advantageous to current registration techniques which require a surgeon to gain access to the patient’s face anatomy for registration by crouching or crawling along the floor to gain an upward perspective on the patient’s face, and trace points for registration of that patient in the OR, while keeping their (relatively short) pointer in view of a stereoscopic camera system. In some embodiments, if there is no second scanner, the finial may be compatible with a hybridnavigation system where the finial is comprised of fiducial spheres or the like (optical markers, OMs).

[0333] Anatomical Verification Without Optical Trackers[00334J By way of example, the methods for real-time verification of anatomical structures and bone resections disclosed herein utilize continuous or on-demand topological data scanning to replace conventional OT-based systems for checkpoints and resection validation. By eliminating the need for physical OTs, this invention provides a less invasive, streamlined, accurate solution for verifying anatomical consistency and bone resection accuracy. The approach significantly enhances workflow efficiency and precision in procedures such as TKA and other complex orthopedic surgeries where alignment and anatomical verification are critical.

[0335] In traditional surgical navigation systems, checkpoints can serve as verification points to ensure that OTs attached to anatomical structures, such as the femur, remain in a fixed position throughout the procedure. These checkpoints require a screw to be placed in the bone, creating a reference location that can be verified periodically using a pointer equipped with an OT. Any shift detected in the location of the checkpoint relative to the OT indicates potential movement, which requires the surgeon to pause and recalibrate the system before proceeding.

[0336] In some embodiments, this disclosure describes a method 830 for real-time verification of the positioning and orientation of anatomical structures that eliminates the need for both OTs and checkpoints by using real-time topological data to directly register and track an anatomical structure (e.g., the femur in a TKA procedure). Through continuous topological scanning, the system can monitor the femur’s position and orientation without requiring physical markers or trackers. This capability builds on the detailed registration and tracking methods disclosed above, which describe how topological data can be used to establish an accurate, continuous tracking framework.

[0337] By way of example, Fig. 35 is a block diagram illustrating several steps of a method for real-time verification of the positioning and orientation of an anatomical structures, according to some embodiments. In some embodiments, a first step 832 in the method 830 for real-time verification of the positioning and orientation of an anatomical structure is to determine an initialbaseline registration of the anatomical structure into common coordinate space (e g., as described above). By way of example, at the outset of the surgical procedure, a plurality of high-resolution topological scans of the femur are captured for example using the 3D scanner 12, capturing surface contours and anatomical landmarks. These baseline scans provide a comprehensive dataset that defines the femur’s position within the surgical space, leveraging the topological registration principles described above.

[0338] In some embodiments, a second step 834 in the method 830 for real-time verification of the positioning and orientation of an anatomical structure is continuous topological tracking of the anatomical structure. By way of example, as the surgical procedure advances, the CAAT system 10 may continuously collect topological data, comparing live scans of the femur to the baseline registration. In some embodiments, any movement or shift is detected by the CAAT system 10 in real time, allowing the CAAT system 10 to track the femur’s exact position without physical trackers or checkpoints, using the tracking methodologies described above.

[0339] In some embodiments, the CAAT system 10 may employ a process referred to as “transfer registration” as a third step 836 of the method 830 for real-time verification of the positioning and orientation of an anatomical structure. By way of example, in transfer registration, each newly acquired optical scan is registered to a prior reference scan representing the immediately preceding anatomical state. In some embodiments, each registration thereby links the current intraoperative anatomical state to the previous anatomical state. Through a sequence of intermediate anatomical states, the current anatomical state is linked to the original anatomic reference model. Because each transformation is based on successive stages of anatomy exhibiting maximum geometric commonality, the resulting registrations are statistically robust to cumulative alignment error.

[0340] In some embodiments, successful registrations rely on statistical sufficiency of common anatomical features between consecutive stages. It is important in transfer registration to maintain common anatomy sufficient for confident registration in all six degrees of freedom. For this reason, it is likely that scans will take place at standard timepoints in a procedure. Metrics on the number of inliers and their distance, and statistical outputs such as covariant matrices could enable the system to quantify the robustness of each stage’s registration.

[0341] In surgical contexts such as TKA or spine surgery, transfer registration may be used to update the reference model as anatomical modifications occur. For example, in TKA, the updated reference may exclude bone and / or cartilage that has been resected and include newly exposed cut surfaces. In spine procedures, for example, the reference may similarly exclude removed bony structures while incorporating newly revealed rigid anatomy. Rigidly affixed or relatively immobile tissue may also contribute to stable registration across stages when such tissue remains unaffected by the surgical manipulation. Although described primarily in the context of a TKA, transfer registration is suitable for many procedures where anatomy is affected, for example resection of bone, and is not limited to those discussed here.

[0342] In some embodiments, between major stages of anatomical change, the system may acquire new optical scans to update the current reference frame. For example, during stages in which the anatomy remains relatively unchanged (e.g., during drilling or probing operations), the system may rely on continuous motion tracking relative to the most recent reference, maintaining spatial accuracy in real time. In this manner, the system combines stage-based re-registration with intra-stage tracking to preserve registration accuracy throughout the procedure.

[0343] This approach allows intraoperative registration updates without cumulative drift exceeding acceptable limits, provided that each individual registration meets accuracy requirements and sufficient anatomical overlap exists between consecutive states.

[0344] Without the need for periodic OT verifications or recalibrations, the surgeon can proceed through the procedure uninterrupted (e.g., a third step 836 of the method 830). This continuous tracking reduces workflow disruptions, enhances accuracy, and removes reliance on physical markers that are subject to movement or accidental shifts. By replacing traditional OT- dependent checkpoints with real-time topological tracking, the method 830 streamlines the verification process, ensuring consistent and precise anatomical registration throughout the procedure.

[0345] Verification of Osteotomies and Bone Resections Using Topological Data

[0346] In TKA and similar procedures, accurate bone resection is critical for ensuring that implants align correctly with the surgical plan. Traditionally, OT-based tools with planarsurfaces are used to verify each cut, comparing the resected surfaces against the preoperative plan. This approach requires precise OT calibration and positioning, which adds procedural complexity and can lead to cumulative error.

[0347] By way of example, Fig. 36 is a block diagram illustrating several steps of a method 850 for real-time verification of osteotomies and bone resections that eliminates the need for OT- based instruments by leveraging topological data in tandem with CT-based registration to verify location and orientation of osteotomies and bone resections relative to an original surgical plan. Using high-resolution scans before and after each resection, the method 850 provides the surgeon with a streamlined, tracker-free means of validating the accuracy of each cut.

[0348] In some embodiments, a first step 852 of the method 850 for real-time verification of osteotomies and bone resections is initial registration of CT and scan data. By way of example, before any bone resection, the anatomical site is registered to a common coordinate system by aligning preoperative CT data with an initial topological scan, for example according to the registration procedure described above. This foundational registration process aligns the CT data with the surgical field and sets up the common coordinate system. This enables the surgical navigation system to precisely understand the location and orientation of the femur (for example) relative to the planned resection planes.

[0349] In some embodiments, a second step 854 of the method 850 for real-time verification of osteotomies and bone resections is assisted bone resection. For example, once the femur is registered, the resection may be performed with robotic assistance or guided using real-time topological scanning to ensure precision during the cut. Methods for guiding cuts using topological data and robotic assistance are detailed above, providing reference for systems that integrate cutting and tracking functions.

[0350] In some embodiments, a third step 856 of the method 850 for real-time verification of osteotomies and bone resections is post-resection scan and idealized model comparison. For example, after completing the resection, one or more new topological scans of the surgical site may be taken. To facilitate accurate post-resection verification, the initial CT-based femur model may be modified to reflect the planned resection by virtually removing the cut portion of the bone from the data set. This creates an idealized, post-resection model of the femur,preserving only the surfaces that remain after the resection. During verification and / or tracking, the system registers this modified CT dataset with the new post-resection scan.

[0351] In some embodiments, a fourth step 858 of the method 850 for real-time verification of osteotomies and bone resections is comparison of the resected surface with surgical plan. For example, with the modified CT model aligned to the current scan, the system can identify the planar surface of the bone cut and compare its position and orientation against the planned resection plane. This comparison allows the system to calculate any deviations, providing immediate feedback on the location and orientation of the cut relative to the original surgical plan.

[0352] In some embodiments, a fifth step 860 of the method 850 for real-time verification of osteotomies and bone resections is surgeon decision-making. For example, if the difference between the actual cut and the planned resection exceeds predefined tolerances, this system may alert the surgeon, who can then assess whether the cut is close enough to proceed or if adjustments to the surgical plan are needed to accommodate the discrepancy.

[0353] By registering and verifying the cut using both CT-based and real-time topological data, this disclosure provides precise and reliable feedback on resection accuracy without relying on OT-based tools. This method enables seamless, data-driven validation of each bone resection, ensuring high alignment accuracy and reducing the need for additional instrumentation or calibration.

[0354] Hybrid system

[0355] By way of example, Fig. 37 is a block diagram representing a hybrid navigation system 910 according to the present disclosure. By way of example, the hybrid navigation system 910 as described herein is essentially a combination of the CAAT system 10 described above with elements of a stereoscopic camera system added in. In some embodiments, the hybrid navigation system 910 may feature a first data acquisition apparatus 911 (e.g., which in the example described herein throughout is a structured light 3D scanner 12 as described above), hub 14, user interface 16, distributed computation 18, and a software program 20 embodied in a non-transitory computer readable storage medium (e.g., the hub 14) and comprising a set ofinstructions configured to enable the computer to perform the various tasks described herein. Tn some embodiments, one or more peripheral computing devices 21 may be associated with the hybrid navigation system 910, for example to integrate use and control of certain medical and surgical equipment that may be used in robotics-driven procedures. Such peripheral computing device(s) 21 may be configured to be controlled by the hub 14 and / or user interface 16. By way of example, these components are essentially identical to the same components described above in relation to the CAAT system 10. Notably, the hybrid navigation system 910 further includes a second data acquisition apparatus 913 which is not a structured light 3D scanner 12, but which by way of example may include one or more stereoscopic cameras 912 and a one or more optical trackers 914 (as described as the primary example herein throughout), an electromagnetic navigation apparatus 915, ultrasound imaging apparatus 917, and / or an intraoperative imaging apparatus (e g., CT, MRI, etc ). In some embodiments, although represented as a single computing system, the hybrid navigation system 910 may include multiple independent computing systems working together, for example at least one each for the structured light 3D scanner 12 and the second data acquisition apparatus 913 that facilitate normal operation of those systems as well as the additional functionality of the hybrid navigation system 910 described herein.

[0356] A fundamental requirement for surgical navigation systems is the ability to consistently locate and track objects (e.g., anatomical structures, surgical instruments, and / or equipment) within a unified spatial framework, referred to herein as the “surgical coordinate system (SCS).” By way of example, the SCS is typically defined as a three-dimensional Cartesian coordinate system. Its origin can be placed at a convenient, easily identifiable location, such as the base of a surgical robot, the center of a stereoscopic camera’s field of view, a fixed anatomical landmark, and / or an OT attached to anatomy. In some embodiments, for Cartesian systems, the SCS is defined by three orthogonal axes (X, Y, Z) that establish the spatial orientation and relationships of objects. Although Cartesian coordinates are common, in some embodiments other coordinate systems, such as polar or spherical systems, may be used if better suited to a specific procedure, surgical environment or interface with other systems. By way of example, in alternative systems, such as polar or spherical coordinates, positions are described using angular and radial values relative to the origin.

[0357] In some embodiments, the SCS may remain fixed or rigid with respect to the patient. For example, the SCS may be a tracker rigidly affixed to the patient (who is not necessarily fixed in space) so that even if imaging equipment (and / or surgical robot) moves, the SCS remains static relative to the patient. In some embodiments, the SCS may remain fixed in space during a procedure (e.g., relative to the robot base). In some embodiments, dynamic configurations of the SCS are also possible, where the origin or orientation may change to adapt to movements of the patient or equipment. When dynamic configurations are used, real-time updates are required to ensure accurate tracking.

[0358] In some embodiments, the SCS serves as the backbone of the hybrid navigation system, ensuring that all data — whether from anatomical scans, instrument tracking, or robotic movements — can be integrated into a unified spatial reference. In some embodiments, the SCS includes several core functionalities, including but not limited to consistency across systems (e.g., so that positional and orientation data from multiple navigation modalities are aligned), unified localization, (e.g., providing a single, reliable framework for tracking anatomy, instruments, and equipment relative to one another), adaptability (e.g., supporting the use of Cartesian or alternative coordinate systems, accommodating the specific demands of different surgical procedures), and precision alignment (e.g., facilitating transformations from local coordinate systems into the SCS, enabling accurate spatial representation and coordination).

[0359] By way of example, the SCS is essential for integrating the structured light scanning and stereoscopic navigation components of the hybrid navigation system, ensuring that all elements are synchronized in a single, consistent reference framework.

[0360] In some embodiments, to unify data from different navigation systems within the surgical coordinate system (SCS), the hybrid navigation system 910 may use transformation matrices (TMs or “transforms”). By way of example, TMs are mathematical tools that translate positions and orientations of objects from their local coordinate systems to the SCS for spatial alignment. In some embodiments, TMs play a critical role in the hybrid navigation system by localization, (e.g., translating positions and orientations of objects, such as anatomical landmarks or instruments, from their local coordinate systems into the SCS) and inter-system integration (e g., allowing data from different navigation technologies, e.g., structured light scanners 12 andstereoscopic systems 912, to be combined and interpreted consistently). In some embodiments, a transformation matrix and / or its inverse may be needed when daisy-chaining TMs so that the data is related to the correct coordinate system.

[0361] In some embodiments, transformation matrices may be structured in a 4x4 matrix format, as shown by way of example in Fig. 1. By way of example, transformation matrices encode translation components (e.g., specifying the positional offset between coordinate systems), rotation components (e.g., representing angular differences in orientation), and scaling components (e.g., accounting for any scaling effects (typically set to unity for rigid systems)). In some embodiments, the fourth row of the matrix is always [0, 0, 0, 1], which supports homogeneous coordinates and simplifies mathematical operations. In some embodiments, the transformation matrix may be a standard 4x4 homogeneous matrix representing a rigid transform (no shear, no warp, scaling effectively 1).

[0362] Chaining Transformations

[0363] In surgical navigation, objects may need to be mapped through multiple intermediate systems before reaching the SCS. In some embodiments, this is achieved by multiplying transformation matrices and / or inverse transformation matrices, effectively "chaining" transformations together to create a result.

[0364] Example: Locating the Lateral Epicondyle of the Femur

[0365] By way of example, Fig. 38 is a depiction of an example of a surgical pointer instrument 920 configured for use with the hybrid system 910 described herein. In some embodiments, the pointer instrument 920 may comprise a handle 922, elongated shaft 924, and a tip 926. In some embodiments, the pointer instrument 920 may include an OT 914 attached to the handle 922.

[0366] By way of example, Fig. 39 is a block diagram depicting a method 930 of locating an anatomical landmark in the SCS, for example by chaining transformation matrices to map an anatomical location at the tip of the pointer instrument 920 into the SCS. By way example, only, the method 930 is described using a lateral epicondyle of the femur as the anatomical landmark, however the method 930 may be used to map any surgical landmark into the SCS.

[0367] Fig. 40 illustrates an example of a surgical environment 22 which may be a sterile field within an operating room at a hospital or surgical center, in which a patient 24 is positioned on an operating table 26 for a surgical procedure such as a TKA. By way of example, a surgical instrument 28 (e.g., drill, oscillating saw, etc.) may be provided at the end of a first robotic arm 30 having a base 31, which itself is attached to a first cart 32. In some embodiments, the surgical environment 22 is set up with two navigation systems. In this example, this includes a stereoscopic camera system (SSC) 912 mounted traditionally, several meters from the surgical site, and a structured light 3D scanner (SLS) 12 forming part of the hybrid system 910 mounted above the surgical site with visibility of the surgical exposure, for example positioned at the distal end of a second robotic arm 34 which is provided on a second cart 36. In some embodiments, an OT 914 is mounted to the robot base 31 which is fixed (e g., to first cart 32) and considered the SCS in this example. By way of example, in this configuration, the OT 914 is within the field of view 916 of the stereoscopic camera system 912.

[0368] It should be noted that this surgical environment 22 is provided by way of example only to illustrate one possible setup and positioning of the structured light 3D scanner 12 and stereoscopic camera system 912 in relation to a patient during a surgical procedure. In some embodiments, the second robotic arm 34 to which the structured light 3D scanner 12 is attached may be attached to a wall, ceiling, immobile table, articulating arm, or other structure. In some embodiments, the stereoscopic camera system 912 may be attached to a wall, ceiling, immobile table, or other structure.

[0369] In the current example embodiment, the SCS origin is defined at the base 31 of the surgical robot 30. As such, the SCS may also be referred to as a “robot coordinate system” or “robot space.” In this embodiment, the stereoscopic camera system 912 must detect the OTs 914 for both the pointer instrument 920 and the robot base 31 in the same scan dataset to establish a consistent coordinate mapping.

[0370] In some embodiments, a first step 932 in the method of locating an anatomical landmark in the SCS is calibrating the pointer tip 926 in OTp space. By way of example the pointer 920 may comprise two coordinate systems, a tip coordinate system (or “tip space”), withits origin at the tip 926, and an OTp coordinate system (or “OTp space”), associated with the optical tracker 914 attached to the pointer 920. A transformation matrix,TM[OTP / Tip] may be created by the hybrid system 910 during calibration (e.g., as a translation TM describing the tip offset from the origin). This transformation matrix establishes the geometric relationship between the tip coordinate system and the OTp coordinate system, based on the pointer's design and manufacturing tolerances. Once calibrated, the transformation matrix TM[OTp / Tip] may be used to map the position (e.g., offset) of the pointer tip 926 from tip space to OTp space. In some embodiments, the pointer tip 926 may be represented by a known (calibrated) point in the OTp coordinate system based on the known mechanical specifications of the instrument, obviating the need for the described transformation matrix.

[0371] In some embodiments, a next step 934 in the method of locating an anatomical landmark in the SCS is using the stereoscopic camera 912 to detect OTp within an acquired scan dataset to capture tip point 926 locations associated with pre-defined anatomic features, while positioning the pointer tip 926 on the anatomical landmark to be mapped, for example the lateral epicondyle of the femur. In some embodiments, the pointer tip 926 is placed directly on the anatomical landmark (e.g., lateral epicondyle), marking the tip origin at this spot. In some embodiments, the hybrid system 910 detects the OT 914 attached to the pointer 920 within the field of view 916 of the stereoscopic camera system 912 (e.g., “OTp”), enabling the hybrid system 910 to map the position of the pointer OT 914 into camera space. A transformation matrix,TM[Camera / OTp] maps the OTp coordinate system into camera space. By way of example, this works to map a single point of the anatomic feature (or a point-like anatomic feature if the feature can be adequately represented by a single point), for example a single point on a condyle, assuming that the transformation matrix between the selected point and the corresponding anatomy (preoperative) has been established. As a preliminary step, mapping an entire anatomic feature (e.g., an entire condyle) itself requires more points to establish the transformation matrix. Forexample, there must be multiple points to describe the position and orientation of the anatomic feature (e.g., a minimum of three points, with a built-in asymmetry).

[0372] In some embodiments, a next step 936 in the method of locating an anatomical landmark in the SCS is mapping the camera space to the SCS (e g., robot space), for example by detecting the OT 914 attached to the robot base 31 (e.g., “OTR”) within a scan dataset of the stereoscopic camera 912 and using the known calibrated transformation matrix between the OTR and the robot base 31 to map camera space to the surgical coordinate system (e.g., robot space). By way of example, the surgical robot's base 31 is equipped with an OT 914 that defines the robot coordinate system, which in this example is the SCS. When the hybrid system 910 detects the robot base OT 914 in the stereoscopic camera 912 scan, the hybrid system 910 which registers the surgical coordinate system into camera space and produces a transformation matrix,TM[Camera / Robot] which maps the surgical coordinate system into camera space. To map camera space into the surgical coordinate system, the inverse of TM[Camera / Robot] is used:TM[Robot / Camera] =TM[Camera / Robot]1

[0373] In some embodiments, a next step 938 of the method of locating an anatomical landmark in the SCS is determining the position of the target anatomical landmark (e.g., lateral epicondyle) in robot space. By way of example, the hybrid system 910 may determine this by chaining the transformations:Epi condyle [Robot] = Tip[Robot] = TM[Camera / Robot]1• TM[Camera / OTP] • Tip[OTP]This equation ensures that the position of the pointer tip 926, initially defined in the OTp coordinate system, is accurately expressed in the unified surgical coordinate system (e.g., robot space).

[0374] The hybrid navigation system 910 enables structured light scanning and stereoscopic navigation to work together by aligning data from both systems within the unified surgicalcoordinate system. This alignment can be achieved using one of two methods, depending on the specific setup and clinical requirements. A first example method 940 of aligning data involves attaching an OT 914 to the structured light scanner 12. A second example method 950 of aligning data involves attaching an OT 914 to the patient 24. Both methods are described by way of example below.

[0375] First Approach: Optical Tracker Attached to the Scanner

[0376] Fig. 41 illustrates an example of a surgical environment 22 which may be a sterile field within an operating room at a hospital or surgical center, in which a patient 24 is positioned on an operating table 26 for a surgical procedure such as a TKA. By way of example, a surgical instrument 28 (e.g., drill, oscillating saw, etc.) may be provided at the end of a first robotic arm 30 having a base 31, which itself is attached to a first cart 32. In some embodiments, the surgical environment 22 is set up with two navigation systems. In this example, this includes a stereoscopic camera system 912 mounted traditionally, several meters from the surgical site, and a structured light 3D scanner 12 forming part of the hybrid system 910 mounted above the surgical site with visibility of the surgical exposure, for example positioned at the distal end of a second robotic arm 34 which is provided on a second cart 36. By way of example, in this approach, an OT 914 is physically attached to the structured light scanner 12, serving as a reference point for alignment between the structured light scanner 12 and the stereoscopic camera system 912. In some embodiments, this attachment may work in conjunction with a sterile drape (not shown). By way of example, it can be assumed that during calibration, the coordinate system of the structured light scanner 12 and its OT 914 are the same, and therefore a transformation matrix (e.g., TM[SLS / OT]) is unnecessary. If, for example, the coordinate system of the structured light scanner 12 and its OT 914 were not the same, then an additional link in the transformation chain would be required. In some embodiments, another OT 914 is mounted to the robot base 31 which is fixed (e.g., to first cart 32) and considered the SCS in this example. By way of example, in this configuration, both OTs 914 are within the stereoscopic camera system 912 field of view 916, and therefore the stereoscopic camera system 912 can monitor both OTs 914 simultaneously.

[0377] It should be noted that this surgical environment 22 is provided by way of example only to illustrate one possible setup and positioning of the structured light scanner 12 and stereoscopic camera system 912 in relation to a patient during a surgical procedure. In some embodiments, the second robotic arm 34 to which the structured light scanner 12 is attached may be attached to a wall, ceiling, immobile table, or other structure. In some embodiments, the stereoscopic camera system 912 may be attached to a wall, ceiling, immobile table, or other structure.

[0378] By way of example, Fig. 42 illustrates a first example method 940 of aligning data from a SLS 12 and a SSC system 912 within the unified SCS (and continuing in the context of a TKA procedure). In some embodiments, a first step 942 of the method 940 of aligning data from a SLS 12 and a SSC system 912 within the unified SCS is to capture anatomical data of the femur, for example using the methods described in the above discussion of the CAAT system 10. In this example embodiment, the hybrid system 910 uses the structured light scanner 12 to acquire topological scan data of the surgical site including the femur, from which anatomical data is captured from reconstructed 3D point clouds of the surgical site and to which preoperative scan data (e.g., CT scan data) is registered in the SLS coordinate system (or “space”) which produces a transformation matrix. In this case, the femur may be registered and to CT data obtained pre-operatively, which has a predefined coordinate system, and tracked. Using the same nomenclature as before, the TM may be defined asTM[SLS / Femur] and provides the TM to transform the femur coordinate system into the SLS coordinate system.

[0379] In some embodiments, a next step 944 of the method 940 of aligning data from a SLS 12 and a SSC system 912 within the unified SCS, is to map the position of the structured light 3D scanner 12 to the stereoscopic camera coordinate space. In some embodiments, the hybrid system 910 may accomplish this by using the stereoscopic camera 912 to track the OT 914 mounted on the structured light 3D scanner 12, calculating a transformation matrix,TM[SSC / SLS] that maps the position of the structured light 3D scanner 12 to stereoscopic camera space.

[0380] In some embodiments, a next step 946 of the method 940 of aligning data from a SLS 12 and a SSC system 912 within the unified SCS, is to map the robot coordinate system (e.g., the SCS) to the stereoscopic camera 912 coordinate system. By way of example, the hybrid system 910 may accomplish this by using the stereoscopic camera system 912 to track the robot base OT 914, generating a transformation matrixTM[SSC / Robot] that maps the robot coordinate system to the SSC coordinate system. Notably, the inverse of this transformation matrix can transform coordinates in the opposite direction (e.g., from the SSC coordinate system to the robot coordinate system or SCS).

[0381] In some embodiments, a next step 948 of the method 940 of aligning data from a SLS 12 and a SSC system 912 within the unified SCS is to map the anatomical data into the robot space / SCS. By way of example, the hybrid system 910 may accomplish this by employing a final transformation chainFemur[Robot] = TM[SSC / Robot]-1• TM[SSC / SLS] • TM[SLS / Femur] • Femur[Femur] that combines all matrices to map the anatomy into the surgical coordinate system.

[0382] By way of example, some advantages of this first example method 940 of aligning data from a SLS 12 and a SSC system 912 within the unified SCS are that the method 940 simplifies alignment by using a fixed OT 914 mounted on the structured light 3D scanner 12 and integrates well with existing stereoscopic workflows without requiring significant modifications. Furthermore, the structured light 3D scanner 12 does not need to be fixed, and could even be handheld.

[0383] Second Approach: Optical Tracker in Common View

[0384] By way of example, the second example method 950 of aligning data from a SLS 12 and a SSC system 912 within the unified SCS relies upon using an OT 914 positioned near the surgical exposure in common view of both the structured light 3D scanner 12 and the stereoscopic camera system 912. In some embodiments, the common view OT 914 may beattached to the patient 24, surgical drapes (not shown), surgical table 26, etc. By way of example, this common view OT 914 provides a shared reference point for both the structured light 3D scanner 12 and the stereoscopic camera system 912. As used herein, the common view OT 914 may also be referred to as “the reference OT 914.”

[0385] Fig. 43 illustrates an example of a surgical environment 22 which may be a sterile field within an operating room at a hospital or surgical center, in which a patient 24 is positioned on an operating table 26 for a surgical procedure such as a TKA. By way of example, a surgical instrument 28 (e.g., drill, oscillating saw, etc.) may be provided at the end of a first robotic arm 30 having a base 31, which itself is attached to a first cart 32. In some embodiments, the surgical environment 22 is set up with two navigation systems. In this example, this includes a stereoscopic camera system 912 mounted traditionally, several meters from the surgical site, and a structured light 3D scanner 12 forming part of the hybrid system 910 mounted above the surgical site with visibility of the surgical exposure, for example positioned at the distal end of a second robotic arm 34 which is provided on a second cart 36. In some embodiments, a reference OT 914 is mounted to the patient 24 (for example). In some embodiments, another OT 914 is mounted to the robot base 31 which is fixed (e.g., to first cart 32) and considered the SCS in this example. By way of example, in this configuration, both OTs 914 are within the field of view 916 of the stereoscopic camera system 912, and therefore the stereoscopic camera system 912 can monitor both OTs 914 simultaneously.

[0386] It should be noted that this surgical environment 22 is provided by way of example only to illustrate one possible setup and positioning of the structured light 3D scanner 12 and stereoscopic camera system 912 in relation to a patient during a surgical procedure. In some embodiments, the second robotic arm 34 to which the structured light 3D scanner 12 is attached may be attached to a wall, ceiling, immobile table, or other structure. In some embodiments, the stereoscopic camera system 912 may be attached to a wall, ceiling, immobile table, or other structure.

[0387] In some embodiments, the reference OT 914 may be securely attached near the surgical exposure using a clamp, adhesive marker, or other fixation method. In some embodiments, the reference OT 914 may be rigidly attached to a patient’s bone. In someembodiments, the reference OT 914 is not rigidly attached to the patient. In some embodiments, if the SLS 12 and SSC 912 are synchronized, the reference OT 914 can be moving. If the SLS 12 and SSC 912 are not synchronized, a trick is to place the pointer tip 926 in several indentations. Even though the OT spheres are captured at different times for the SLS 12 and SSC 912, they correspond to the same position for the pointer tip 926. If this is repeated for three different positions of the tip 926, the transform between the two coordinate spaces may be generated. It should be noted that the reference OT 914 must be in the field of view of both the structured light 3D scanner 12 and the stereoscopic camera system 912. By way of example, the robot base OT 914 which may determine the unified surgical coordinate system must be within the field of view of at least one of the structured light 3D scanner 12 and the stereoscopic camera system 912, or alternatively, the structured light 3D scanner 12 has stitched the scene so that even though the robot base OT 914 is not within its field of view, its position is known (e.g., while everything is stationary).

[0388] By way of example, Fig. 44 illustrates a second example method 950 of aligning data from a SLS 12 and a SSC system 912 within the unified SCS (and continuing in the context of a TKA procedure). In some embodiments, a first step 952 of the method 950 of aligning data from a SLS 12 and a SSC system 912 within the unified SCS is to capture anatomical data of the femur, for example using the methods described in the above discussion of the CAAT system 10. In this example embodiment, the hybrid system 910 uses the structured light scanner 12 to acquire topological scan data of the surgical site including the femur, from which anatomical data is captured from reconstructed 3D point clouds of the surgical site and to which preoperative scan data (e.g., CT scan data) is registered in the SLS coordinate system (or “space”) via a transformation matrix. In this case, the femur may be registered and tracked using CT data obtained pre-operatively, which has a predefined coordinate system. Using the same nomenclature as before, the transformation matrix may be defined as:TM[SLS / Femur] and provides the transformation matrix to transform the femur coordinate system into the SLS coordinate system.

[0389] In some embodiments, a next step 954 of the method 950 of aligning data from a SLS 12 and a SSC system 912 within the unified SCS is to map the position of the reference OT 914 to the SLS space. By way of example, the hybrid system 910 may accomplish this by using the structured light 3D scanner 12 to scan the reference OT 914, producing a transformation matrix:TM[SLS / RefOT] that relates the position of the reference OT to the SLS. In some embodiments, the same scan may be used by the hybrid system 910 to produce the transformation matrix TM[SLS / Femur] and the transformation matrix TM[SLS / RefOT],

[0390] In some embodiments, a next step 956 of the method 950 of aligning data from a SLS 12 and a SSC system 912 within the unified SCS is to map the position of the reference OT 914 to the SSC space. By way of example, the hybrid system 910 may accomplish this by using the stereoscopic camera system 912 to scan the reference OT 914, producing a transformation matrix:TM[SSC / RefOT] to map the position of the reference OT 914 to the SSC space.

[0391] In some embodiments, a next step 958 of the method 950 of aligning data from a SLS 12 and a SSC system 912 within the unified SCS is to map the robot space to the SSC space. By way of example, the hybrid system 910 may accomplish this by using the stereoscopic camera system 912 to scan the robot base OT 914, producing a transformation matrix:TM[SSC / Robot] that maps robot space to the SCS space. In some embodiments, the same scan may be used by the hybrid system 910 to produce the transformation matrix TM[SSC / RefOT] and the transformation matrix TM[SSC / Robot],

[0392] In some embodiments, a next step 958 of the method 950 of aligning data from a SLS 12 and a SSC system 912 within the unified SCS is to map the anatomy (e.g., femur anatomy inthis case) to the SSC space. By way of example, the hybrid system 910 may accomplish this by combining all transformations, using the inverse of the transformations where needed:Femur[Robot] = TM^SC / Robot]1• TM[SSC / RefOT] • TMfSLS / RefOT]'1• TM[SLS / Femur] • Femur [Femur]

[0393] By way of example, some advantages of the second method 950 are that the second method 950 provides a reference point close to the patient’s anatomy and reduces dependency on visibility between the structured light 3D scanner 12 and the stereoscopic camera system 912, as both systems rely on tracking the reference OT. Moreover, the second method 950 potentially reduces errors caused by chaining transformation matrices, for example if the total length from the structured light 3D scanner 12 to the reference OT 914 to the stereoscopic camera system 912 is shorter than the total length of the first method 940. This length would be the length from the femur to the structured light 3D scanner 12 to the stereoscopic camera system 912. By way of example, the shorter length path may have less occurrence of error.

[0394] Real-Time Synchronization and Timestamping

[0395] In some embodiments, a critical feature of the hybrid navigation system 910 disclosed herein is its ability to perform real-time tracking of anatomical landmarks, surgical instruments, and equipment. This is to update continuously positions and orientations throughout the procedure, maintaining accuracy and responsiveness. To achieve this, the structured light 3D scanner 12 and the stereoscopic camera system 912 must have a method to synchronize their scan data. In some embodiments, this may be accomplished by time-stamping every transformation matrix generated by the hybrid navigation system 910 when they are produced, for example with the acquisition time of the data used in generating the transformation matrix.

[0396] By way of example, time stamping scan data from the structured light 3D scanner 12 and the stereoscopic camera system 912 allows the hybrid navigation system 910 to dynamically adjust to movements of the patient 24, reference OT 914, instruments (e.g., pointer 920 or instrument 28), or equipment during surgery. In some embodiments, the following parameters may be employed ensure the effectiveness of timestamping the transformation matrices.

[0397] In some embodiments, the hybrid navigation system 910 may process updates at a minimum frequency of 20 registrations per second (RPS) to qualify as real-time. In some embodiments, the hybrid navigation system 910 may process updates at higher frequencies, such as 30-60 RPS or more, to further improve responsiveness and reduce positional error.

[0398] In some embodiments, both the structured light 3D scanner 12 and the stereoscopic camera system 912 may be operated in real-time to synchronize their data streams. By way of example, the hybrid navigation system 910 each generates transformation matrices for each of the structured light 3D scanner 12 and the stereoscopic camera system 912 at their own respective update frequency, associating each generated transformation matrix with a timestamp. The timestamped transformation matrices are then synchronized based on the timestamps and integrated into the surgical coordinate system.

[0399] In scenarios where multiple computer systems collect sensor data independently (e.g., using multiple standalone computing devices or using distinct computing device on board the structured light 3D scanner 12 and the stereoscopic camera system 912), ensuring temporal alignment requires synchronized timestamps. By way of example only, for applications where the multiple computer systems are not connected to the internet but have direct communication capabilities, common methods synchronizing timestamps may include (but are not limited to) a local master clock with broadcast timestamps, hardware-driven synchronization, pulse-based synchronization, direct time exchange, and / or clock drift compensation. Regarding a local master clock with broadcast timestamps, in some embodiments, one system can be designated as the master clock, broadcasting timestamps to the secondary system over a direct communication link (e.g., serial, USB, or Ethernet). This provides a unified time reference while eliminating the need for external synchronization services. Regarding hardware-driven synchronization, in some embodiments, a shared hardware time source, such as a Real-Time Clock (RTC) module or GPS clock, can act as the authoritative clock for both systems. The hardware clock can be accessed by each system through a shared communication protocol (e.g., I2C, SPI, or GPIO) for consistent timestamps. Regarding pulse-based synchronization, in some embodiments, a hardware synchronization pulse or signal, transmitted from one system to the other (e.g., via a GPIO pin), can establish a common temporal reference point. Both systems record the pulse timing relative to their local clocks and align subsequent timestamps accordingly. Regarding direct timeexchange, in some embodiments, the systems can exchange timestamp information periodically over their communication link, allowing one system to align its clock to the other's. This method is straightforward and effective for applications with limited precision requirements. Regarding clock drift compensation, in some embodiments, for systems with stable local clocks, periodic time exchanges between the systems can also be used to calculate and compensate for clock drift, maintaining relative synchronization over time. In some embodiments, the systems may also synchronize timing by using visible markers or SLS 12 patterns that the SSC 912 can see.

[0400] While these methods are highlighted for their simplicity and practicality, numerous other techniques exist for synchronizing system clocks, including more sophisticated protocols such as IEEE 1588 Precision Time Protocol (PTP) or specialized synchronization middleware. The choice of method depends on the application's specific precision, latency, and hardware requirements.

[0401] In some embodiments, to align data temporally between the structured light 3D scanner 12 and stereoscopic camera system 912, all TMs are time-stamped using a common clock, for example with the acquisition time of the data used in generating the transformation matrix. This ensures positional and orientation data from both optical systems is synchronized for accurate real-time integration. By way of example, timestamps allow the hybrid navigation system 910 to align data from the structured light 3D scanner 12 and stereoscopic camera system 912, even if they operate independently or at different update rates. Moreover, timestamps establish a shared temporal reference, enabling integration of anatomical and instrument data into the SCS.

[0402] By way of example, in hybrid systems, it may be common for components to operate at different update frequencies. In some embodiments, for example, the structured light 3D scanner 12 may update at a rate of 20 registrations per second (RPS). In some embodiments, the stereoscopic camera system 912 may update at a rate of 30 RPS. These update rates are provided by example only and may be higher or lower for each system. In some embodiments, the hybrid navigation system 910 of the present disclosure may employ interpolation and extrapolation to synchronize data streams from the multiple components. In some embodiments, the hybrid navigation system 910 may use interpolation to estimate intermediate transformation matrices forthe system with the lower update frequency. For example, when aligning structured light data (e g., obtained at a rate of 20 RPS) with stereoscopic data (e.g., obtained at a rate of 30 RPS), intermediate TMs may be interpolated by the hybrid navigation system 910 to match the timestamps of transformation matrices generated using the stereoscopic camera system 912. In some embodiments, the hybrid navigation system 910 may use extrapolation to predict future transformation matrices when data is unavailable from the system with the lower frequency. By way of example, extrapolation may be useful for short-term alignment but less accurate than interpolation.

[0403] Fig. 45 illustrates an example method 960 for aligning SLS-derived data with SSC- derived data for synchronized real-time tracking using a hybrid navigation system 910. In some embodiments, a first step 962 in the method 960 is to output time-stamped transformation matrices derived from the structured light 3D scanner 12 data at a first rate. In some embodiments, for example, the structured light 3D scanner 12 may output time-stamped transformation matrices at 50 ms intervals which correlates to 20 RPS. In some embodiments, a second step 964 in the method 960 is to output time-stamped transformation matrices derived from the stereoscopic camera system 912 data at a second rate. In some embodiments, for example, the stereoscopic system 912 may output time-stamped transformation matrices at 33.3 ms intervals which correlates to 30 RPS. In some embodiments, a third step 966 in the method 960 is to use interpolation to generate intermediate transformation matrices to align the time- stamped SLS transformation matrices with the time-stamped SSC transformation matrices. In some embodiments, a fourth step 968 in the method 960 is to integrate aligned data into the surgical coordinate system.

[0404] In some embodiments, after outputting time-stamped transformation matrices derived from the structured light 3D scanner 12 data at a first rate and outputting time-stamped transformation matrices derived from the stereoscopic camera system 912 data at a second rate, the data can be aligned by selecting the time-stamped transformation matrix derived from the stereoscopic camera system 912 that most closely matches the time-stamped transformation matrix derived from the structured light 3D scanner 12 data, without interpolation.

[0405] By way of example, errors introduced during interpolation or extrapolation may be minimized through high update frequencies (e.g., small time intervals between updates reduce discrepancies caused by system delays) and / or advanced filtering (e.g., techniques like Kalman filters smooth positional data, suppress noise, and refine accuracy).

[0406] Ensuring Compatibility Between Navigation Systems

[0407] The hybrid navigation system 910 disclosed herein integrates structured light scanning and stereoscopic navigation technologies during simultaneous operation. By designing the hybrid navigation system 910 to harmonize these two complementary modalities, it can perform accurately and reliably. In some embodiments, this compatibility may be achieved using distinct operational characteristics for each system, time-based coordination, and / or robust environmental adaptability.

[0408] In some embodiments, one compatibility issue that has arisen when using the hybrid navigation system of the present disclosure is the light source for both systems. By way of example, discussion of this topic will continue in the context of an example TKA surgery using a hybrid navigation system 910 disclosed herein comprising a stereoscopic camera system 912 and a structured light 3D scanner 12.

[0409] For example, if optical navigations systems are used, like a structured light 3D scanner 12 or stereoscopic camera system 912, they would ideally work with light from different parts of the spectrum. For example, there would be no compatibility issue if the structured light 3D scanner 12 operates in the infrared (IR) range to capture high-resolution anatomical topology, and the stereoscopic navigation system 912 uses visible or near-IR light to track optical markers on instruments and equipment, or vice versa, where the structured light 3D scanner 12 operates in the near-IR range to capture high-resolution anatomical topology, and the stereoscopic navigation system 912 uses IR light to track optical markers on instruments and equipment. In some embodiments, both the structured light 3D scanner 12 and the stereoscopic navigation system 912 may operate within the same band of light (e.g., IR or near-IR), but at different portions (e.g., wavelengths) of the band. This separation ensures that each system functions independently without confusion or overlap.

[0410] By way of example, if the spectra of two systems are similar, the optical systems can operate in a time-multiplexed manner, where only one system emits light at a time. This operation may require alternating activity, in which the structured light scanner 12 and stereoscopic system 912 take turns collecting data, switching at high frequencies to maintain real-time operation, and synchronization, in which both systems are synchronized using a common clock, ensuring smooth data integration within the surgical coordinate system (SCS).

[0411] Communication Between Navigation Systems

[0412] In some embodiments, for the hybrid navigation system 910 to function effectively, the structured light 3D scanner 12 and stereoscopic navigation system 912 must communicate well. This communication ensures proper integration of data, synchronization of transformations, and smooth operation in real time. The design of this communication framework is critical to achieving the system’s full potential.

[0413] In some embodiments, the hybrid navigation system 910 of the present disclosure enables structured light scanning and stereoscopic navigation to share and process data for various purposes. For example, communication between the structured light 3D scanner 12 and stereoscopic camera system 912 is important for synchronization of transformation matrices (TMs). The two optical systems must share transformation matrices to align anatomical data and instrument tracking within the surgical coordinate system. Communication ensures that time- stamped TMs from each system are synchronized and integrated accurately.

[0414] By way of example, communication between the structured light 3D scanner 12 and stereoscopic camera system 912 is also important for data sharing for enhanced functionality. The structured light 3D scanner 12 provides detailed anatomical mapping, which can be used to register preoperative imaging to the SCS, and the stereoscopic camera system 912 determines instrument position. The hybrid navigation system 910 refines the instrument tracking (due to registrations using the 3D scanning data). In some embodiments, the stereoscopic camera system 912 shares instrument positions, which can be visualized and adjusted in real time using the anatomical model generated by the structured light 3D scanner 12.

[0415] Moreover, communication between the structured light 3D scanner 12 and stereoscopic camera system 912 (e.g., with each other or with a central data processing center such as the hub 14 of the hybid navigation system 912) is important for error checking and validation. Cross-validation between the two systems enhances accuracy. That is, if the hybrid navigation system 910 detects a deviation in a known landmark within scan data derived from one of the optical systems, it can employ the other optical system to verify and correct the data. For example, if the hybrid navigation system 910 determines that a relative shift between an OT 914 rigidly attached to anatomy and the anatomy itself has occurred based on scan data acquired by the structured light 3D scanner 12, it can correct the data derived from the stereoscopic camera system 912 with a correction transformation matrix.

[0416] Finally, communication between the structured light 3D scanner 12 and stereoscopic camera system 912 is important for workflow coordination. For example, the hybrid navigation system 910 must coordinate the workflows of each optical system to maintain real-time operation, such as directing alternating active states in time-multiplexed configurations or avoiding data collisions.

[0417] In some embodiments, to facilitate communication, the hybrid navigation system 910 of the present disclosure employs a robust architecture that supports efficient, reliable data exchange. By way of example, types of data shared may include (but are not limited to) transformation matrices (e.g., to align data from both systems within the SCS), positional and orientation data (e.g., real-time updates of instrument and anatomical landmarks), and / or status updates (e.g., operational states of each system, such as active / inactive or calibration status).

[0418] In some embodiments, the communication protocols utilized by the hybrid navigation system 910 may include network-based communication and / or direct connections. In some embodiments, for example, both optical systems can communicate over a high-speed, low- latency network, such as Ethernet or Wi-Fi, using standard communication protocols like TCP / IP. In some embodiments, for example in tightly coupled setups, the systems may communicate directly through USB, serial interfaces, or dedicated hardware links for faster data exchange.

[0419] In some embodiments, the hybrid navigation system 910 may include centralized data handling. For example, a centralized control unit or computational hub (e.g., hub 14) may receive data from both optical systems, processes the data, and then integrate the data into the SCS. This approach may ensure consistent data handling and transformation alignment and reduced computational burden on individual systems.

[0420] In some embodiments, the hybrid navigation system 910 may include features that increase redundancy and reliability. For example, backup communication channels may be provided to ensure continuous operation in case of a failure in the primary link. In some embodiments, error-checking algorithms may be used to verify data integrity during transmission.

[0421] Fig. 46 illustrates an example workflow 970 for communication between the structured light 3D scanner 12 and stereoscopic camera system 912.

[0422] In some embodiments, a first step 972 in the example communication workflow 970 is initialization of the communication protocol. For example, in some embodiments, the hybrid navigation system 910 may establish a communication link between the structured light 3D scanner 12 and stereoscopic camera system 912. In some embodiments, the hybrid navigation system 910 may ensure that the clock of the structured light 3D scanner 12 and the clock of stereoscopic camera system 912 are synchronized. In some embodiments, calibration data may be shared to align initial positions within the SCS.

[0423] In some embodiments, a second step 974 in the example communication workflow 970 is real-time operation. For example, in some embodiments, the structured light 3D scanner 12 may collect and transmit anatomical data, including transformation matrices and landmark positions, to a central processor such as the hub 14 (for example), or to a computing unit on the scanner 12 or the stereoscopic camera system. In some embodiments, the stereoscopic camera system 912 may collect and send instrument tracking data and anatomic data to the SCS. In some embodiments, both systems can integrate the data in the SCS for real-time visualization and adjustments.

[0424] In some embodiments, a third step 976 in the example communication workflow 970 is dynamic coordination. By way of example, if the hybrid navigation system 910 detects a significant change in one optical system’s scan data, (e.g., a deviation in a known landmark on a patient, instrument, and / or OT within scan data derived from one of the optical systems), it can employ the other optical system to verify and correct the data, as previously mentioned, or can direct recalibration of the affected optical system. Time-stamped data ensures that updates are aligned temporally between the systems.

[0425] In some embodiments, a key advantage of the hybrid or multimodal navigation system 910 is its ability to track anatomy and other objects in real time without relying on rigid fixation of OTs. Both methods described above (e.g., OT attached to the scanner, and OT attached to the patient) enable flexibility by leveraging time-stamped scan data. This ensures that even if a scanner or reference OT moves, the hybrid navigation system 910 can maintain accuracy by comparing scan data captured simultaneously.

[0426] By way of example, each scan may be treated as a snapshot of the hybrid navigation system 910 state at a specific moment in time. Even when subsequent scans occur at different positions, the calculations remain valid, as the object's location and orientation can be reliably mapped into the surgical coordinate system (SCS) using time-stamped transformation matrices.

[0427] Thus, this approach eliminates the need for rigidly attaching reference OTs to the patient using invasive bone pins, simplifying the workflow and reducing patient discomfort without compromising precision.

[0428] In this disclosure, it is assumed that both the structured light 3D scanner 12 and stereoscopic camera system 912 can detect and register an optical tracker. However, the structured light 3D scanner 12 has the unique ability to identify specific geometries beyond the reflective optical markers (OMs) typically arranged in distinct patterns to define an OT.

[0429] By way of example, in conventional TKA navigation, pin-mounted tracker arrays are often placed at a distance from the distal joint surfaces, creating a long mechanical and geometrical lever arm between the proxy tracker and the anatomical region of interest. This configuration is susceptible to “lever-arm effects,” where small angular errors or mechanicalinstabilities at the tracker level translate into larger positional errors at the cutting or implant site. In some embodiments, the hybrid navigation system 910 mitigates this lever-arm effect by scanning the actual knee joint surfaces, including both bone and cartilage, with a surface scanner (e.g., structured light 3D scanner 12) that is registered to the navigation frame under either a tracker-primary or scanner-primary hardware concept. By effectively relocating the sensing locus to the joint surfaces, the hybrid navigation system 910 shortens the lever arm between sensing and action and reduces positional error arising from small angular uncertainties in distant proxies. Repeated scans during the procedure allow the system to detect drift of OTs 914 relative to the femur or tibia, reveal subtle motion or flex in the assemblies holding the trackers, and re- estimate the relationship between the joint anatomy and any external proxies. As a result, the distal femoral and tibial surfaces may be registered more precisely, increasing confidence in guidance for cuts, gap assessment, and implant positioning relative to tracker-only configurations.

[0430] In some embodiments, this expanded capability allows for the introduction of a "hybrid OT” 980, as shown by way of example only in Figs. 47-48. By way of example, a hybrid OT 980 includes the traditional reflective OMs 982 in a unique pattern or constellation for SSC 912 detection while also incorporating additional geometry 984 that the SLS 12 can identify. In some embodiments, the frame holding the OMs may feature this extra geometry (as shown for example in Fig. 48), enabling the hybrid OT 980 to serve both scanning technologies effectively. In any scenario previously described, a standard OT can be replaced with a hybrid OT 980, which adapts to the capabilities of the respective scanning technologies.

[0431] In some embodiments, hybrid OTs 980 may be deployed in configurations in which they function as “tracker-primary” devices within the surgical coordinate system. In such configurations, the optical marker pattern of the hybrid OT 980 is treated as the primary navigation reference, and the additional geometry visible to the SLS 12 is used to confirm that the hybrid OT 980 remains rigidly coupled to the anatomy or device it is intended to represent. Repeated scans of the relevant anatomy or instrument using the SLS 12 may reveal flex, slip, or shift of the underlying structures, including subtle robot base shifts, TKA tracker pins flexing or drifting in bone, or cranial pins in a Mayfield-type clamp allowing small amounts of head wander. When discrepancies are detected between the pose implied by SSC 912 tracking of thehybrid OT 980 and the pose implied by SLS 12 scans of the anatomy or tool, the system may update one or more transforms, or prompt re-registration, to maintain navigation accuracy.

[0432] In other embodiments, hybrid OTs 980 may be deployed in configurations that are “scanner-primary,” in which the geometry recognized by the SLS 12 is treated as the dominant sensing element and the reflective OMs 982 are optional or serve primarily for auxiliary tracking. In such configurations, fiducials and engineered shapes on the hybrid OT 980 are designed principally for robust recognition by the SLS 12, while any associated OMs 982 may still be detected by SSC 912 to provide a global reference frame, assist in initial calibration, or extend the working volume. Scanner-primary examples include hybrid OTs 980 mounted on CT gantry structures, robot bases, ENT scopes, ultrasound probes, or tool-integrated scanners, where the local view of the SLS 12 at or near the point of interaction effectively defines the relevant portion of the surgical coordinate system. In this manner, the tracker-primary and scannerprimary configurations provide a unifying framework for describing how hybrid OTs 980 participate in defining and maintaining accurate navigation across the various embodiments described herein.

[0433] Furthermore, hybrid OTs 980 open new possibilities. For example, a traditional OT could be mounted on one end of a long instrument, with a distinct geometric feature added to the opposite end. These two ends could be mathematically connected through calibration or design to produce a transformation matrix (TM) linking them. This setup could bridge non-overlapping fields of view between two navigation systems, maintaining a shared reference OT that both systems can recognize and track seamlessly. In some embodiments, hybrid OTs 980 may be used in registering a prone brain surgery patient, where registration is performed on the face, for example.

[0434] In some embodiments, the hybrid navigation system 910 may be configured with multiple structured light 3D scanners 12 in the same environment, such that each scanner 12 is calibrated into a shared reference frame using scanner-optimized fiducials, global trackers, or robot kinematics. Overlapping fields of view from these scanners provide extended spatial coverage, redundancy, and cross-checks of anatomy and instrument pose. By way of example, beyond the primary surgical use cases, the same hybrid concepts extend to non-medical domains,including industrial robot calibration, where scanner-based measurements of workpieces or tools refine encoder-based kinematics, and navigation of non-medical flexible endoscopes in complex mechanical assemblies. In all of these scenarios, distant proxies such as bases and linkages are complemented by local scanning near the point of interaction, reinforcing the core strategies of lever-arm reduction, proxy verification, and object-centric sensing described in the main text.

[0435] In some embodiments, hybrid navigation with ultrasound can be affected by tracking a sonographic probe and reconciling ultrasound volumes with surface scans. By way of example, a tracked ultrasound probe can be localized in the navigation frame using optical tracking markers attached to the probe body, robot encoders when the probe is mounted on a robot arm, and / or direct recognition of scanner-visible geometry or fiducials that are integrated into the probe housing. In some embodiments, during ultrasound acquisition, an optical surface scanner may capture the deformation of the visible external anatomy, such as the skin, while the probe is in contact with the patient so that the position of the interior soft tissue can be inferred via numerical modeling. This allows the scanned tissues in the composite ultrasound scan to be interpretable in their deformed, as well as to be understood in their undeformed state. This picture would become clearer as ultrasound images are taken from different angles. For non- rigid anatomy such as brain, liver, or kidney, the system may use rigid or quasi-rigid alignment when deformations are small, or non-rigid registration between pre-procedural CT / MR images and intraoperative ultrasound volumes when larger, spatially varying deformation must be modeled. Surface geometry and internal landmarks extracted from ultrasound can be combined to maintain alignment. Example use cases include brain surgery, where intraoperative ultrasound compensates for brain shift following dural opening, and abdominal interventions, where repeated ultrasound and surface scans track organ motion or deformation relative to the skeletal frame.

[0436] In some embodiments, hybrid navigation systems 910 that use CT images as a data source can employ the CT scanner hardware itself as a navigation coordinate frame. In this configuration, a 3D optical scanner 12 could localize CT machine-mounted fiducial structures or the hardware of the machine itself. Any other navigation component (e.g. stereo-based) would then co-observing optical trackers on the CT scanner and possibly rigidly attached to the optical 3D scanner, or (in the case of the other navigation component being another optical 3D scanner)by directly recognizing scanner-optimized fiducial features or characteristic CT machine geometry. Once the CT scanner pose is known in the navigation frame, the acquired CT image volume is bound to the optical 3D scanner or any other component in the navigation system via a fixed transform. Subsequent intraoperative physical surface scans or tool localizations can then be registered into a common space, such as the CT space either through a direct transform from the surface scanner frame to the CT scanner frame or via composite transforms that may pass through intermediate trackers such as a robot base or patient tracker. Treating the CT scanner as a global reference device in this way enables intraoperative CT updates of anatomy and instrumentation, verification or re-registration of patient position following movement or table motion, and consistency checks between preoperative CT, intraoperative CT, and surface scans. By way of example, this embodiment may employ the concept of transfer registration described above. For example, if the patient does not move after a CT scan, the position of the patient’s anatomy is known. Therefore, registering any one part of the anatomy (one that has more bone exposed than others) with the optical scanner will indicate where the rest of the anatomy is (minimizing the amount of anatomy needed to be cleared). The scan (with semi-rigidly attached tissue) can then be correlated for each vertebra of a spine, for example, to the known bony anatomy. If that semi-rigidly attached tissue is not changed, navigation can continue in realtime. By way of example, this is a transfer registration from CT to optical scanning. Then, if anatomy changes in stages little by little, transfer registrations from optical scan to optical scan can be done as previously described.

[0437] Figs 49-50 are example block diagrams of computer-implemented electronic devices 500, 550 that may be used to implement the systems and methods described in this document, as either a client or as a server or plurality of servers. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device 550 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart-phones, and other similar computing devices. In this example, computing device 500 may represent stationary computer or hub 12 and / or other computing systems referenced in this disclosure. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations described and / or claimed in this document.

[0438] Referring to Fig. 49, computing device 500 includes a processor 502, memory 504, a storage device 506, a high-speed interface 508 connecting to memory 504 and high-speed expansion ports 510, and a low-speed interface 512 connecting to low-speed bus 514 and storage device 506. Each of the components 502, 504, 506, 508, 510, and 512, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor 502 can process instructions for execution within the computing device 500, including instructions stored in the memory 504 or on the storage device 506 to display graphical information for a graphic user interface (GUI) on an external input / output device, such as display 516 coupled to high-speed interface 508. In other implementations, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices 500 may be connected, with each device providing portions of the necessary operations (e g., as a server bank, a group of blade servers, or a multi-processor system).

[0439] The memory 504 stores information within the computing device 500. By way of example only, the memory 504 may be a volatile memory unit, non-volatile memory unit, or another form of computer-readable medium, such as a magnetic or optical disk (for example).

[0440] The storage device 506 is capable of providing mass storage for the computing device 500. In one implementation, the storage device 506 may be or contain a non-transitory computer-readable medium (e.g., any and all computer-readable media except transitory, propagating signals), such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. A computer program product can be tangibly embodied in an information carrier. The computer program product may also contain instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 504, the storage device 506, or memory on processor 502.

[0441] The high-speed interface 508 manages bandwidth-intensive operations for the computing device 500, while the low-speed interface 512 manages lower bandwidth-intensive operations. Such allocation of functions is by way of example only. In one implementation, thehigh-speed interface 508 is coupled to memory 504, display 516 (e.g., through a graphics processor or accelerator), and to high-speed expansion ports 510, which may accept various expansion cards (not shown). In the implementation, low-speed interface 512 is coupled to storage device 506 and low-speed expansion port 514. The low-speed expansion port may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) and may be coupled to one or more input / output devices, such as a keyboard 518, a printer 520, a scanner 522, or a networking device such as a switch or router 524, e.g., through a network adapter.

[0442] The computing device 500 may be implemented in a number of different forms. For example, it may be implemented as a standard server, or multiple times in a group of such servers. It may also be implemented as part of a rack server system. In addition, it may be implemented in a personal computer such as a laptop computer. Alternatively, components from computing device 500 may be combined with other components in a mobile device, such as device 550 (Fig. 50). Each of such devices may contain one or more of computing device 500, 550, and an entire system may be made up of multiple computing devices 500, 550 communicating with each other.

[0443] Referring to Fig. 50, computing device 550 includes a processor 552, memory 554, an input / output device such as a display 556, a communication interface 558, and a transceiver 560, among other components. The device 550 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components 550, 552, 554, 556, 558, and 560, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

[0444] The processor 552 can execute instructions within the computing device 550, including instructions stored in the memory 554. The processor may be implemented as a chipset of chips that include separate and multiple analog and digital processors. Additionally, the processor may be implemented using any of a number of architectures. For example, the processor 552 may be a CISC (Complex Instruction Set Computers) processor, a RISC (Reduced Instruction Set Computer) processor, or a MISC (Minimal Instruction Set Computer) processor. The processor may provide, for example, for coordination of the other components of the device550, such as control of user interfaces, applications run by device 550, and wireless communication by device 550.

[0445] The processor 552 may communicate with a user through control interface 562 and display interface 564 coupled to a display 556. The display 556 may be, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display) display or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface 564 may comprise appropriate circuitry for driving the display 556 to present graphical and other information to a user. The control interface 562 may receive commands from a user and convert them for submission to the processor 552. In addition, an external interface 566 may be provided in communication with processor 552, so as to enable near area communication of device 550 with other devices. External interface 566 may provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used.

[0446] The memory 554 stores information within the computing device 550. The memory 554 can be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. Expansion memory 568 may also be provided and connected to device 550 through expansion interface 570, which may include, for example, a SIMM (Single In Line Memory Module) card interface. Such expansion memory 568 may provide extra storage space for device 550 or may also store applications or other information for device 550. Specifically, expansion memory 568 may include instructions to carry out or supplement the processes described above and may include secure information also. Thus, for example, expansion memory 568 may be provided as a security module for device 550 and may be programmed with instructions that permit secure use of device 550. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner. In some embodiments, and by way of example only, the memory 554 may also include one or more of a log of scanner use, calibration information, hardware and software versions, date of assembly, hardware configuration, date last calibrated, service history, and the like.

[0447] The memory may include, for example, flash memory and / or NVRAM memory, as discussed below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, cause performance of one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 554, expansion memory 568, or memory on processor 552 that may be received, for example, over transceiver 560 or external interface 566.

[0448] Device 550 may communicate wirelessly through communication interface 558, which may include digital signal processing circuitry where necessary. Communication interface 558 may provide for communications under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication may occur, for example, through radio-frequency transceiver 560. In addition, short-range communication may occur, such as using a Bluetooth, WiFi, or other such transceiver (not shown). In addition, GPS (Global Positioning System) receiver module 572 may provide additional navigation- and location-related wireless data to device 550, which may be used as appropriate by applications running on device 550.

[0449] Device 550 may also communicate audibly using for example audio codec 574, which may receive spoken information from a user and convert it to usable digital information. Audio codec 574 may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of device 550. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music fdes, etc.) and may also include sound generated by applications operating on device 550.

[0450] The computing device 550 may be implemented in a number of different forms, some of which are shown in the figure. For example, it may be implemented as a cellular telephone. It may also be implemented as part of a smart-phone, personal digital assistant, or other similar mobile device.

[0451] Additionally computing device 500 or 550 can include Universal Serial Bus (USB) flash drives. The USB flash drives may store operating systems and other applications. The USBflash drives can include input / output components, such as a wireless transmitter or USB connector that may be inserted into a USB port of another computing device.

[0452] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0453] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor.

[0454] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0455] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interactwith an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), peer-to-peer networks (having ad-hoc or static members), grid computing infrastructures, and the Internet.

[0456] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0457] Additionally, by way of example, the computer system may include one or more specialized components such as GPU, field programmable gate array (FPGA), digital signal processor (DSP), neural processing unit (NPU), high speed camera interfaces, PCIe, etc.

[0458] Definitions

[0459] As used herein, a “3D Scanner” is a device that captures three-dimensional (3D) topological data of the surgical field, including anatomical structures and surgical tools, in real time. While there are multiple types of 3D scanners, this disclosure focuses on a proprietary 3D structured light scanner. However, any 3D topological scanner capable of collecting similar data may be used, as this disclosure emphasizes the use of the scanner and its data rather than the emphasizing the scanner itself.

[0460] As used herein, “continuous anatomical automatic tracking (CAAT)” is a system feature that continuously updates the position and orientation of anatomical structures and instruments by collecting data with a 3D scanner. In some embodiments, this process eliminates the need for physical optical markers, offering real-time tracking throughout the surgical procedure.

[0461] As used herein, “gap balancing” is a technique used to ensure equal spacing (gaps) between bones in both flexion and extension during surgeries like Total Knee Arthroplasty (TKA). Accurate gap balancing helps maintain proper joint stability and alignment.

[0462] As used herein, a “hub” is a computer responsible for processing raw data from the 3D scanner and automatically registering it to the Surgical Coordinate System.

[0463] As used herein, an “iterative closest point (ICP)” is an algorithm that refines the registration process by minimizing the distance between corresponding points in two datasets for precise alignment.

[0464] As used herein, an “osteotomy” is a surgical procedure that involves cutting and reshaping bones, commonly used in orthopedic surgeries to correct deformities or prepare the bone for implants.

[0465] As used herein, a “point cloud” is a set of points in 3D space, each representing a specific location on a scanned surface, used to generate a topological map of the surgical field. By way of example, the 3D scanner described herein provides a particularly dense point cloud, offering high-resolution data that enhances precision in surgical procedures, and it does it at >25 rps, with high accuracy, optionally in the near-infrared, and with an advantageous illumination coverage and depth of field in the pattern projection.

[0466] As used herein, a “random sample consensus (RANSAC)” is an algorithm used for initial coarse registration by selecting key points in the dataset and filtering out outliers.

[0467] As used herein, “real-time tracking” is defined as continuous registration of anatomical structures or tools, updating multiple times per second to reflect movements and changes in position within the surgical field. This ensures that data is captured and updated in real time without noticeable delays.

[0468] As used herein, a “robot-mounted scanner” is a 3D scanner attached to the robotic arm, or to the end effector of a robotic arm that allows the scanner to dynamically track anatomical structures as the robot moves during surgery.

[0469] As used herein, “sphere fitting” is a method used to estimate the center of rotation of a joint, such as the hip, by fitting a mathematical sphere to points collected during motion tracking of the femur.

[0470] As used herein, a “structured light pattern” is a method of projecting a predefined light pattern (such as grids, stripes, or dots) onto surfaces in order to capture 3D topological data.

[0471] As used herein, a “user interface (UI)” is a visual interface that provides real-time visualizations and guides the surgeon through key steps in the procedure.

[0472] As used herein, “varus angles” and “valgus angles” are the angles that represent the inward (varus) or outward (valgus) tilting of bones, often measured during surgeries to assess and correct misalignment in joints like the knee.

[0473] As used herein, the term “sagittal plane” is an anatomical plane that divides the body into left and right halves and is typically used to measure medial-lateral alignment. As used herein, the term “coronal or frontal plane” is an anatomical plane that divides the body into anterior (front) and posterior (back) sections, separating the joint into anterior and posterior components. As used herein, the term “transverse or axial plane” is an anatomical plane that divides the body into superior (upper) and inferior (lower) sections, often used for rotational alignment assessments.

[0474] As used herein, the term “flexion / extension (F / E) angle” is an angular measurement in the sagittal plane between the bones of the joint, with flexion being a positive angle representing bending, and extension being closer to or at 0°, representing a straightened j oint. A negative number would indicate a hyperextended knee joint. As used herein, the term “internal / external rotation angle” is an angular measurement of rotational alignment in the transverse plane.Internal rotation is a positive value, indicating inward rotation, while external rotation is negative, indicating outward rotation. As used herein, the term “varus / valgus (V / V) angle” is an angular measurement between the bones in the joint in the coronal plane. A positive V / V angle indicates a varus (medial / inward) orientation, while a negative V / V angle represents a valgus (lateral / outward) orientation.

[0475] As used herein, the term “contact distance (D)” is the measured or estimated distance between two key contact points on the joint surfaces, such as the medial and lateral edges of the tibial plateau or femoral condyles. This distance serves as a lever arm in trigonometriccalculations of joint gap changes under applied stress. This value can be preset in the software UI but would be changeable by the surgeon.

[0476] As used herein, the term “joint laxity” is the flexibility or looseness of a joint under applied stress, reflecting the joint’s capacity to withstand movement without compromising stability. Joint laxity measurements are critical for ensuring balanced load distribution and stability, particularly in joint replacement and repair procedures. Joint laxity is not normally a direct measurement but inferred by measuring gaps between bones.

[0477] As used herein, the term “peak gap” is the maximum gap achieved between two articulating surfaces under applied stress. Real-time tracking of peak gaps is essential for an accurate assessment of joint laxity, especially when continuous force application is challenging.

[0478] As used herein, the term “topological data collection” is a non-invasive scanning technique that captures the 3D surface geometry of anatomical structures, creating high- resolution data that maps out contours, ridges, and other surface features. This data provides a basis for real-time measurement and tracking of relative positions without requiring external markers or optical trackers.

[0479] As used herein, the term “zero-stress condition” is a baseline position where no additional forces (such as varus, valgus, or rotational stresses) are applied to the joint. In this state, the gap between the articulating bones is assumed to be neutral or zero, serving as a reference for all subsequent relative measurements under stress.

[0480] As used herein, the term “camera space” refers to the coordinate system within which the 3D scann...

Claims

CLAIMSWhat is claimed is:

1. A hybrid surgical navigation system configured for registering and tracking one or more anatomic structures of a patient and one or more artificial objects within a common coordinate system in an operating room environment, comprising: a first data acquisition apparatus comprising a three-dimensional scanning device having an image capture component and a first optical tracker rigidly coupled to the first data acquisition apparatus; a second data acquisition apparatus comprising a stereoscopic camera configured to detect optical trackers; a computer system including a memory and at least one processor; and computer readable media embodied in a non-transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a three-dimensional data representation of a surface of an anatomic structure of the patient using a first dataset, the first dataset derived from one or more images captured by the first data acquisition apparatus without optical trackers or applied fiducials, the first dataset being expressed in a first coordinate system associated with the first data acquisition apparatus; compute a first transformation matrix representing a position and orientation of the first data acquisition apparatus in a second coordinate system associated with the second data acquisition apparatus, the first transformation matrix derived from a second dataset comprising one or more stereoscopic images of the first optical tracker captured by the second data acquisition apparatus; compute a second transformation matrix representing a position and orientation of an artificial object having an attached optical tracker in the second coordinate system, the second transformation matrix derived from a third dataset comprising one or more stereoscopic images of the attachedoptical tracker captured by the second data acquisition apparatus; derive a third transformation matrix by multiplying an inverse of the first transformation matrix with the second transformation matrix; and apply the third transformation matrix to transform positional information of the artificial object from the second coordinate system to the first coordinate system.

2. The hybrid surgical navigation system of claim 1, wherein the artificial object is a surgical instrument, an article of equipment located within the operating room environment, or an optical tracker.

3. The hybrid surgical navigation system of claim 1, wherein the three-dimensional scanning device is a structured light three-dimensional scanner.

4. The hybrid surgical navigation system of claim 3, wherein the structured light three- dimensional scanner includes a light source and a pattern generator.

5. The hybrid surgical navigation system of claim 4, wherein the pattern generator is a liquid crystal matrix.

6. The hybrid surgical navigation system of claim 1, wherein the computer readable media includes further instructions that, when executed by the one or more processors, cause the computer system to: receive a fourth dataset representing a three-dimensional anatomical model of the same anatomical structure represented in the first dataset, the fourth dataset being derived from preoperative or intraoperative medical imaging and expressed in a virtual coordinate system; generate a virtual surgical plan defined in the virtual coordinate system using the fourth dataset, the virtual surgical plan comprising one or more surgical targets or surgical cut planes; register the first dataset to the fourth dataset to compute a fourth transformation matrix representing a transformation from the virtual coordinate system to the first coordinate system; andapply the fourth transformation matrix to transform the virtual surgical plan from the virtual coordinate system to the first coordinate system.

7. The hybrid surgical navigation system of claim 6, wherein the computer readable media includes further instructions that, when executed by the one or more processors, cause the computer system to: compute a deviation between a position and orientation of the artificial object expressed in the first coordinate system and one or more transformed surgical cut planes expressed in the first coordinate system.

8. The hybrid surgical navigation system of claim 7, wherein the computer readable media includes further instructions that, when executed by the one or more processors, cause the computer system to: provide a control signal to a robotic actuator configured to reduce the deviation and assist a surgeon in performing a surgical procedure.

9. The hybrid surgical navigation system of claim 8, wherein the artificial object is attached to the robotic actuator.

10. The hybrid surgical navigation system of claim 6, wherein the medical imaging comprises computed tomography (CT) or magnetic resonance imaging (MRI) scan data.

11. The hybrid surgical navigation system of claim 1, wherein the one or more images are captured by the image capture device using infrared light or near infrared light.

12. The hybrid surgical navigation system of claim 1, wherein the one or more stereoscopic images are captured by the stereoscopic camera using infrared light or near infrared light.

13. A hybrid surgical navigation system configured for registering and tracking one or more anatomic structures of a patient and one or more artificial objects within a common coordinate system in an operating room environment, comprising: a first data acquisition apparatus comprising a three-dimensional scanning device having an image capture component and a first optical tracker rigidly coupled to the first data acquisition apparatus;a second data acquisition apparatus comprising a stereoscopic camera configured to detect optical trackers; a computer system including a memory and at least one processor; and computer readable media embodied in a non-transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a three-dimensional data representation of a surface of an anatomic structure of the patient using a first dataset, the first dataset derived from one or more images captured by the first data acquisition apparatus without optical trackers or applied fiducials, the first dataset being expressed in a first coordinate system associated with the first data acquisition apparatus; time-stamp the first dataset with a first time of acquisition, the first time of acquisition being the time acquisition of the first dataset; compute a first transformation matrix representing a position and orientation of the first data acquisition apparatus in a second coordinate system associated with the second data acquisition apparatus, the first transformation matrix derived from a second dataset comprising one or more stereoscopic images of the first optical tracker captured by the second data acquisition apparatus; time-stamp the first transformation matrix with a second time of acquisition, the second time of acquisition being the time acquisition of the second dataset; compute a second transformation matrix representing a position and orientation of an artificial object having an attached optical tracker in the second coordinate system, the second transformation matrix derived from a third dataset comprising one or more stereoscopic images of the attached optical tracker captured by the second data acquisition apparatus; time-stamp the second transformation matrix with a third time of acquisition, the third time of acquisition being the time acquisition of the third dataset;derive a third transformation matrix by multiplying an inverse of the first time-stamped transformation matrix with the second time-stamped transformation matrix; and apply the third transformation matrix to transform positional information of the artificial object from the second coordinate system to the first coordinate system.

14. The hybrid surgical navigation system of claim 13, wherein the artificial object is a surgical instrument, an article of equipment located within the operating room environment, or an optical tracker.

15. The hybrid surgical navigation system of claim 13, wherein the three-dimensional scanning device is a structured light three-dimensional scanner.

16. The hybrid surgical navigation system of claim 15, wherein the structured light three- dimensional scanner includes a light source and a pattern generator.

17. The hybrid surgical navigation system of claim 16, wherein the pattern generator is a liquid crystal matrix.

18. The hybrid surgical navigation system of claim 13, wherein the computer readable media includes further instructions that, when executed by the one or more processors, cause the computer system to: receive a fourth dataset representing a three-dimensional anatomical model of the same anatomical structure represented in the first dataset, the fourth dataset being derived from preoperative or intraoperative medical imaging and expressed in a virtual coordinate system; generate a virtual surgical plan defined in the virtual coordinate system using the fourth dataset, the virtual surgical plan comprising one or more surgical targets or surgical cut planes; register the first dataset to the fourth dataset to compute a fourth transformation matrix representing a transformation from the virtual coordinate system to the first coordinate system; andapply the fourth transformation matrix to transform the virtual surgical plan from the virtual coordinate system to the first coordinate system.

19. The hybrid surgical navigation system of claim 18, wherein the computer readable media includes further instructions that, when executed by the one or more processors, cause the computer system to: compute a deviation between a position and orientation of the artificial object expressed in the first coordinate system and one or more transformed surgical cut planes expressed in the first coordinate system.

20. The hybrid surgical navigation system of claim 19, wherein the computer readable media includes further instructions that, when executed by the one or more processors, cause the computer system to: provide a control signal to a robotic actuator configured to reduce the deviation and assist a surgeon in performing a surgical procedure.

21. The hybrid surgical navigation system of claim 20, wherein the artificial object is attached to the robotic actuator.

22. The hybrid surgical navigation system of claim 18, wherein the medical imaging comprises computed tomography (CT) or magnetic resonance imaging (MRI) scan data.

23. The hybrid surgical navigation system of claim 13, wherein the one or more images are captured by the image capture device using infrared light or near infrared light.

24. The hybrid surgical navigation system of claim 13, wherein the one or more stereoscopic images are captured by the stereoscopic camera using infrared light or near infrared light.

25. The hybrid surgical navigation system of claim 13, wherein the computer readable media comprises a further set of instructions that, when executed by the one or more processors, cause the computer system to: use interpolation to generate one or more intermediate transformation matrices to align the time-stamped first dataset, the time-stamped first transformation matrix, and thetime-stamped second transformation matrix to align the first and third datasets within the first coordinate system.

26. A hybrid surgical navigation system configured for registering and tracking one or more anatomic structures of a patient and one or more artificial objects within a common coordinate system in an operating room environment, comprising: a first data acquisition apparatus comprising a three-dimensional scanning device having an image capture component configured to reconstruct geometric information of objects within its field of view; a second data acquisition apparatus comprising a stereoscopic camera system configured to detect optical trackers; a hybrid reference object comprising an optical tracking array and a distinct geometric structure configured for detection by both the first data acquisition apparatus and the second data acquisition apparatus; a computer system including a memory and at least one processor; and computer readable media embodied in a non-transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a three-dimensional data representation of a surface of an anatomic structure of the patient using a first dataset, the first dataset derived from one or more images captured by the first data acquisition apparatus without optical trackers or applied fiducials, the first dataset being expressed in a first coordinate system associated with the first data acquisition apparatus; compute a first transformation matrix representing a position and orientation of the hybrid reference object in the first coordinate system based on a second dataset comprising one or more images of the hybrid reference object captured by the first data acquisition apparatus; compute a second transformation matrix representing a position and orientation of the hybrid reference object in a second coordinate system associated with the stereoscopic camera system based on a third datasetcomprising one or more stereoscopic images of the hybrid reference object captured by the second data acquisition apparatus; compute a third transformation matrix representing a position and orientation of an artificial object having an attached optical tracker in the second coordinate system based on a fourth dataset comprising one or more stereoscopic images of the artificial object captured by the second data acquisition apparatus; and derive a fourth transformation matrix expressing the artificial object in the first coordinate system by multiplying the first transformation matrix by an inverse of the second transformation matrix and by the third transformation matrix, and apply the fourth transformation matrix to transform positional information of the artificial object from the second coordinate system to the first coordinate system.

27. The hybrid surgical navigation system of claim 26, wherein the artificial object is a surgical instrument, an article of equipment located within the operating room environment, or an optical tracker.

28. The hybrid surgical navigation system of claim 26, wherein the three-dimensional scanning device is a structured light three-dimensional scanner.

29. The hybrid surgical navigation system of claim 28, wherein the structured light three- dimensional scanner includes a light source and a pattern generator.

30. The hybrid surgical navigation system of claim 29, wherein the pattern generator is a liquid crystal matrix.

31. The hybrid surgical navigation system of claim 26, wherein the computer readable media includes further instructions that, when executed by the one or more processors, cause the computer system to: receive a fifth dataset representing a three-dimensional anatomical model of the same anatomical structure represented in the first dataset, the fifth dataset being derivedfrom preoperative or intraoperative medical imaging and expressed in a virtual coordinate system; generate a virtual surgical plan defined in the virtual coordinate system using the fifth dataset, the virtual surgical plan comprising one or more surgical targets or surgical cut planes; register the first dataset to the fifth dataset to compute a fifth transformation matrix representing a transformation from the virtual coordinate system to the first coordinate system; and apply the fifth transformation matrix to transform the virtual surgical plan from the virtual coordinate system to the first coordinate system.

32. The hybrid surgical navigation system of claim 31, wherein the computer readable media includes further instructions that, when executed by the one or more processors, cause the computer system to: compute a deviation between positional information of the artificial object expressed in the first coordinate system and one or more surgical cut planes of the transformed virtual surgical plan expressed in the first coordinate system.

33. The hybrid surgical navigation system of claim 32, wherein the computer readable media includes further instructions that, when executed by the one or more processors, cause the computer system to: provide a control signal to a robotic actuator configured to move at least one of the artificial object or a surgical guide in a manner that reduces the deviation and assists a surgeon in performing a surgical procedure.

34. The hybrid surgical navigation system of claim 33, wherein the artificial object is attached to the robotic actuator.

35. The hybrid surgical navigation system of claim 31, wherein the medical imaging comprises computed tomography (CT) or magnetic resonance imaging (MRI) scan data.

36. The hybrid surgical navigation system of claim 26, wherein the one or more images are captured by the image capture device using infrared light or near infrared light.

37. The hybrid surgical navigation system of claim 26, wherein the one or more stereoscopic images are captured by the stereoscopic camera using infrared light or near infrared light.

Images