Surgical navigation system and related methods

Abstract

A system and method for registering and tracking anatomical structures during surgical procedures by leveraging continuously updated 3D topological data of objects within the surgical field. Registration allows different datasets such as preoperative CT scans and intraoperative 3D scans to be aligned, combined, and transformed into a common coordinate space, enabling more precise surgical planning and execution. Tracking involves continuously registering these datasets over time, monitoring changes in position or orientation, and keeping objects aligned within the common coordinate space, updating the spatial alignment of registered objects multiple times per second, which allows for dynamic monitoring of anatomical structures and surgical tools as they move throughout the procedure. Unlike traditional systems that rely on physical optical markers (OMs) and optical tracker constellations (OTCs), the system and method disclosed herein utilizes the data captured by a 3D scanner to perform Continuous Anatomic Auto Tracking (CAAT).

Description

VIS-004-W01Surgical Navigation System and Related MethodsCROSS-REFERENCES TO RELATED APPLICATIONS

[0001] The present application claims the benefit of priority from U.S. Provisional Application Serial No. 63 / 716,932, filed November 6, 2024 and entitled “Registration & Tracking by 3D Scanning the Topology of Anatomic Structures”, U.S. Provisional Application Serial No. 63 / 717,085, filed November 6, 2024 and entitled “Articular Surface Matching”, U.S. Provisional Application Serial No. 63 / 769,659, filed March 10, 2025 and entitled “System and Method for Real-Time Object Tracking Without Pre-Existing Models”, and U.S. Provisional Application Serial No. 63 / 844,018, filed July 14, 2025 and entitled “3D Scanner”, the entire contents of which are hereby incorporated by reference into this disclosure as if set forth fully herein.FIELD

[0002] The present disclosure relates generally to object scanning, and more specifically to using three-dimensional (3D) scanning to register and track 3D objects in a real environment relative to objects in a virtual environment.BACKGROUND

[0003] In modern surgical practice, particularly in orthopedic procedures like Total Knee Arthroplasty (TKA) and Hip Arthroplasty, precise alignment and tracking of anatomical structures are crucial for successful outcomes. Over the past two decades, surgical navigation systems have become standard tools for enhancing accuracy during these procedures. These systems typically rely on optical trackers and stereoscopic cameras to guide surgeons through intricate surgical steps, ensuring correct positioning of instruments and implants.

[0004] By way of example, an optical tracker (OT) is a system used to track the position of anatomical structures, instruments, or equipment in the surgical field. An optical tracker is composed of a collection of optical markers (OMs) arranged in a specific geometric pattern, or a "constellation." These constellations allow the system to precisely identify and differentiate between tracked objects based on the unique arrangement of their optical markers.VIS-004-W01

[0005] By way of example, an optical marker (OM) is a marker used in traditional surgical navigation systems to track the position of anatomical structures and tools in real-time via stereoscopic cameras. One common example is a reflective sphere, though multiple types of optical markers can be used depending on the system and application.

[0006] However, despite their effectiveness, these systems present several challenges. The setup of optical trackers is often time-consuming and technically complex, requiring precise calibration and positioning of cameras and tracking equipment. Additionally, the physical nature of optical trackers introduces mechanical vulnerabilities — trackers can become dislodged, damaged, or obstructed during surgery (e.g., affecting line of sight between the camera and the OTs), leading to potential inaccuracies in tracking and necessitating recalibration. These issues add to overall procedural time, which may increase patient exposure to anesthesia and complicate workflow.

[0007] Moreover, the reliance on optical trackers and complex equipment adds significant costs and requires highly trained personnel to operate the systems. These challenges have limited the widespread adoption of such systems, particularly in smaller or resource-constrained healthcare facilities. As a result, there is a pressing need for more robust and efficient solutions that maintain or improve the accuracy of surgical navigation while reducing complexity and cost.

[0008] The following disclosure generally describes methods currently used in the field of surgical navigation, for example a navigation system using optical markers and stereoscopic cameras. By way of example only, the methods discussed and / or disclosed herein are done so in the context of a Total Knee Arthroplasty (TKA), however it should be understood that the systems and methods disclosed herein may be used in any scenario (surgical, medical, or otherwise) to generate a reconstructed 3D model of an object.

[0009] Optical markers (OMs) are small reflective markers that are easily segmented from other elements in an image. These markers retro-reflect infrared light emitted by stereoscopic cameras, making them highly detectable to cameras that are located close to the near infrared (NIR) source. OMs are essential for tracking anatomical landmarks, surgical instruments, and equipment during surgery, ensuring reliable visibility even in complex surgical environments.VIS-004-W01

[0010] Optical markers (OMs) are often mechanically grouped into specific rigid geometric patterns, known as optical trackers (OTs). These unique arrangements allow the system to differentiate between various anatomical structures, instruments, or equipment being tracked. By way of example, OTs can be attached to patient anatomy (e.g., bones) near the surgical site, mounted on surgical instruments, (e.g., pointers or measurement tools), and / or fixed to active equipment (e.g., robotic arms, saws, or drills). The distinct geometries of OTs enable the system to identify what each OM represents within the surgical field.

[0011] Stereoscopic cameras are foundational to spatial tracking in surgical navigation systems. These cameras capture images from slightly different angles, similar to human binocular vision. By detecting the reflective OMs in the surgical field, the cameras calculate the 3D positions of all visible optical trackers (OTs).

[0012] Typically, the navigation system compares the relative positions of the detected OMs to the predefined geometric pattern of each OT. Once an OT is identified, it is assigned to its corresponding surgical tool, anatomical structure, or piece of equipment. The identified OTs are then registered in a common coordinate system, or space, providing a comprehensive 3D representation of the surgical field. The system can now track the position and orientation of all OTs in real time.

[0013] Once the positions of the OTs are known in the common coordinate system, they are transformed into a Surgical Coordinate System (SCS), which provides a fixed reference frame for the procedure. The SCS could be equivalent to camera space, CT space, tracker space, or based on another tracked location, such as the base of a robot. By applying a 4x4 rigid transformation matrix 2 (see Figure 1), any OT can be positioned within the SCS, ensuring consistency in tracking across the system. To accurately transform the object’s origin into the SCS, a sequence of transformations may be applied. The process involves transformations between, for example, an object’s local space, OT space, camera space, and a final SCS, which is a chosen common space for all relevant items for navigation.

[0014] By applying these transformations, the system ensures precise positioning of objects within the SCS, providing consistent and accurate tracking across the surgical environment. It should be noted that each transformation matrix introduces error into the equation. Because ofVIS-004-W01 this, the more transformations that are required to move an object into the SCS, the greater the potential for error. For example, in the equation presented above, three transformations are required.

[0015] For surgical navigation systems to be effective, they must track rigidly attached OTs in real-time, updating positions multiple times per second. This continuous updating enables precise monitoring of tool positions, implant alignment, and anatomical movement, ensuring accurate and timely adjustments throughout the surgery.

[0016] Locating anatomy poses unique challenges because the position of the Optical Tracker (OT) relative to the anatomy is not always well defined. When an OT is attached to an instrument, its location relative to the rest of the instrument is precisely engineered and known. However, with anatomical structures, the initial placement of an OT follows general guidelines but is somewhat random within those parameters.

[0017] Another reason this anatomical registration can be difficult is the attachment method. For instruments, the OT can be attached rigidly with screws, ensuring stability and precision. However, for anatomical structures, the attachment must be created during surgery. For TKA, this typically involves drilling two pins bi-cortically (through both sides of the bone) and then attaching the OT to the pins. These pins are subject to flex due to the bone's characteristics, and the bone quality may not be ideal, further complicating the attachment. The OT must always be “visible” to the stereo camera and not be obscured by surgeons, OR staff, instruments, drapes or other equipment to function properly. Additionally, the OT may be bumped or deflected during the procedure by the surgeon or other equipment, leading to inaccuracies in tracking.

[0018] OTs are securely attached to bony landmarks using these pins or clamps. For instance, in procedures such as Total Knee Arthroplasty (TKA), OTs might be placed on the femur and tibia. These OTs serve as proxies for the position of the underlying anatomy, allowing for continuous tracking throughout the procedure.

[0019] Before the OTs attached to anatomy can be used to locate the anatomical structure within the SCS, the anatomy must first be registered. This registration process typically involves identifying key anatomical landmarks, such as specific points on the bone, and matching themVIS-004-W01 with pre-operative imaging (CT or MRT scans). The system then uses the positions of the attached OTs to align the anatomy with the pre-operative 3D model.

[0020] Surgeons often use a pointer or probe (equipped with its own OT) to touch specific anatomical points, the tip-points of which are then recorded by the system. These points help to map (register) the patient’s anatomy with preoperative imaging or a model, allowing for accurate tracking during surgery. However, because the pointer introduces another OT and another transformation matrix into the sequence of transformations required to locate anatomy in the SCS, potential for error is increased.

[0021] Before surgery, detailed imaging of the target anatomy is often obtained using CT (Computed Tomography) or MRI (Magnetic Resonance Imaging). These scans create a 3D model of the anatomy, which serves as a reference during the procedure. This model is crucial for planning surgical interventions and aligning the patient’s real-time anatomy with the preoperative plan.

[0022] However, some systems and robotic surgical systems do not rely on pre-operative imaging. In these cases, a standard 3D model of bones or other anatomy is used during the planning process. These standard models have landmarks identified within them, which can be used to properly locate the standard model. To make the model more visually realistic in surgical planning applications, the standard models can be scaled, deformed, and / or sheared using an “affine” transformation matrix if needed. By way of example, comparing the landmark data in the CT and the landmark data obtained using a pointer, this affine transformation matrix can be computed. For instance, if an OT attached to a pointer is used to locate the medial and lateral epicondyles of the femur, the real distance between these two points could be calculated and used to scale the standard model. In this way, an appropriately sized model can be used to visualize the placement of various implants in proper scale, though it may not be anatomically accurate for that patient. Other transformation matrices noted herein that are for registration or tracking are rigid transformations and do not include scaling, deformation or shearing effects.

[0023] In both cases, whether pre-operative images are available or not, the process is essentially similar and will be treated as such for this disclosure. If pre-operative data, CT data,VIS-004-W01 or similar data is referenced, it is possible that a scaled, deformed, and / or sheared standard model could be used in its place.

[0024] Example Workflow of Optical Tracking Systems for TKA (Prior Art)

[0025] All Total Knee Arthroplasty (TKA) surgeries are different. The patients are different, and the surgeons who perform the procedures have varying training, experience, and methodologies that shape their approach to surgery and their definition of an ideal TKA. There are also many different surgical navigation systems, robotic assistants, and implant systems to choose from. Because of these differences, a standard TKA workflow cannot be universally defined. The procedure that follows is an example of a typical prior art system. By way of example this workflow is represented in Figure 2.

[0026] Prior to surgery, detailed imaging of the target anatomy is often acquired using CT (Computed Tomography) or MRI (Magnetic Resonance Imaging). These scans are segmented to create a 3D model of the relevant structures, such as the femur and tibia in TKA. These 3D models are essential for planning the procedure and aligning the patient's real-time anatomy with the surgical plan. Important anatomical planes and landmarks are also labeled within the model to assist the surgeon during the operation.

[0027] In some systems, pre-operative imaging is not required. This is referred to as "imageless" mode. In this approach, a standard 3D model of the bones is used instead of patientspecific scans. Anatomical landmarks, such as the medial and lateral epicondyles of the femur, are identified during surgery using OTs and a pointer. The distance between these landmarks is used to scale the standard model to match the patient’s unique anatomy, allowing the surgeon to visualize implant placement with appropriate proportions, even in the absence of patient-specific imaging. Simple scaling or uniform scaling may be used but it is likely that more advanced scaling is used where the scaling factor in various directions is different. There may also be some twisting, deformation and / or shearing required to make the landmarks line up best.

[0028] For this workflow, we assume a patient-specific CT scan has been segmented to provide a detailed 3D model of the femur and tibia, though the imageless mode can also be utilized if necessary.VIS-004-W01

[0029] Before surgery begins, the system requires calibration. Stereoscopic cameras are positioned to ensure full visibility of the surgical field, and OTs may be attached to the surgical instruments, the robotic arm, and the robotic base. Calibration ensures that the system aligns correctly with the surgical setup, accounting for distances, camera angles, and the positioning of all equipment.

[0030] Robotic equipment requires additional steps for calibration to ensure that the robot functions correctly. This process includes aligning the robot's physical movements with the virtual model of the surgery to maintain accuracy and ensure that the robot performs as expected during the procedure.

[0031] After setup and calibration, the patient is positioned on the operating table, typically in a supine position with the knee exposed for TKA procedures. The surgical site is prepped and draped in a sterile fashion. Surgical exposure follows, with an incision made to access the knee joint and prepare it for the robotic system to perform the procedure.

[0032] Patient registration starts when OTs are attached to the patient’s anatomy, typically using two pins driven bi-cortically. For femoral registration, the camera system tracks both the OT attached to the femur and the OT attached to a pointer. The system’s user interface guides the surgeon through the process of identifying specific anatomical points on the femur. During this process, the surgeon collects approximately 50 or more points on the surface of the femur, though the number may vary depending on the system.

[0033] These points generate a sparse point cloud representing the surface of the femur. The point cloud is registered to pre-operative imaging (CT, MRI scans, or scaled standard models) using algorithms like RANSAC (Random Sample Consensus) and ICP (Iterative Closest Point). Once registration is complete, the system knows how the pre-operative imaging data must be transformed to align properly in camera space (and vice versa). With trackers also captured in camera space, and their registration to their respective OT virtual spaces, the real-time anatomical model can then be transformed and maintained within the chosen Surgical Coordinate System (SCS), enabling accurate tracking of the patient’s anatomy throughout the procedure.VIS-004-W01

[0034] Following patient registration, the surgeon identifies key anatomical landmarks necessary for establishing the mechanical axis of the leg, for example including the hip center (e.g. identified by rotating the leg while the optical tracking system calculates the center of rotation of the femoral head), knee center (e.g., identified either from pre-operative imaging or by locating the medial and lateral epicondyles of the femur), and / or ankle center (e.g., identified using the optical tracking system on the medial and lateral malleoli (ankle bones) to calculate the center of the ankle joint). These landmarks are crucial for ensuring proper implant alignment. The mechanical axis is one common methodology, but other alignment methods, like kinematic alignment, may use different anatomical landmarks depending on the procedure.

[0035] At this stage, a virtual surgical plan is developed (e.g., by the surgeon based on their experience and surgical philosophy) to guide the robotic system and surgeon throughout the procedure. If pre-operative planning was done, it serves as an initial basis but must be verified and perhaps adjusted in real time as the surgery progresses.

[0036] The virtual plan typically involves the following steps, which may be applied in a different order or iteratively, depending on the system:

[0037] 1. Implant Size Selection. The system assists the surgeon in selecting the optimal implant size based on the patient’s anatomy. Proper implant sizing ensures both the long-term success of the surgery and post-operative joint function.

[0038] 2. Implant Positioning. The system helps the surgeon determine the ideal implant position to achieve accurate joint alignment. The goal is to ensure proper leg alignment and balance between the medial and lateral sides of the knee. The alignment is measured in terms of varus (inward tilt) and valgus (outward tilt), with the system assisting the surgeon in making appropriate adjustments.

[0039] 3. Gap Balancing. By way of example, some surgeons may assess joint laxity by measuring gaps on the medial and lateral sides of the knee at various flexion angles, using this information to guide implant positioning in a process called “gap balancing.” Gap balancing strives for equal tension in the soft tissues (primarily ligaments) around the knee joint throughout its range of motion. The system dynamically tracks the relative positions of the bones andVIS-004-W01 calculates the space between them to guide the surgeon in balancing the medial and lateral gaps during the procedure. However, it should be noted that surgical techniques and philosophies can vary significantly, and some surgeons omit this step entirely if they do not subscribe to a gap balancing philosophy.

[0040] 4. Osteotomy Planning. Based on the selected implant size and position, the system defines the necessary osteotomies (bone cuts). These cuts must be precise to ensure proper implant placement. By way of example, the necessary bone cuts are determined not just by general planning but specifically by the selected implant size and its final planned position, which are derived from the geometry and design specifics of the chosen implant system.

[0041] 5. Robotic-Assisted Bone Resection. During bone resection, the robotic system assists the surgeon in making precise cuts according to the surgical plan. By way of example, the robotic system assists the surgeon by maintaining the saw blade within the correct cutting plane, while still allowing the surgeon to control when the saw is activated and to move the saw freely within that defined plane as needed during the procedure. The system continuously tracks the patient's anatomy and provides feedback, ensuring the cuts are executed accurately.

[0042] 6. Intraoperative Verification and Adjustment. After each bone cut, the accuracy of the cut is verified. By way of example, the surgeon typically performs this cut verification by using an instrument specifically designed for this purpose, which is equipped with its own OT, to manually check the accuracy of the cut. If deviations are detected between the actual cuts and the surgical plan, the surgeon may need to adjust the plan to correct for any differences.

[0043] 7. Trialing and Final Implantation. After bone cuts are verified, trial implants are placed to assess the fit, alignment, and joint movement. If necessary, adjustments are made before the final implant is inserted.

[0044] 8. Post-Operative Evaluation and Closure. The surgeon performs a final assessment to ensure proper alignment and stability of the new joint. Once confirmed, the surgical site is closed, and the patient is prepared for recovery.

[0045] Despite their widespread use, stereoscopic camera and optical tracking systems present several challenges that affect their efficiency and accuracy in surgical applications.VIS-004-W01

[0046] For example, one challenge of stereoscopic camera and optical tracking systems is the complex setup and calibration of the optical tracking system. The setup of optical tracking systems is time-intensive and requires precise positioning of cameras and optical trackers. Even slight misalignment during calibration can lead to errors in tracking the patient's anatomy or surgical instruments, reducing surgical precision. Additionally, setup time adds to the length of the procedure, potentially increasing patient exposure to anesthesia and overall operative time.

[0047] Another challenge of stereoscopic camera and optical tracking systems is their reliance on optical trackers. The physical nature of optical trackers introduces several potential points of failure. For example, tracker placement can affect surgical exposure, requiring larger incisions or altered approaches that may increase invasiveness. Additionally, trackers are prone to physical damage, shifting, or detachment during surgery. Further, if any optical marker on the tracker becomes dirty during surgery, it may affect its registration to the virtual OT space, reducing accuracy. Certainly, if a tracker becomes dislodged or its line of sight is sufficiently obstructed, the system cannot accurately track the anatomy or instruments, leading to error. The specialized optical trackers, cameras, and calibration equipment are expensive, limiting their use to well-funded hospitals and surgical centers. The additional equipment also increases the logistical complexity of the operating room. These systems include a lot of equipment and specific procedures must be followed to ensure their proper use and functioning. Thus, the training and learning curve associated with these systems is substantial. While these systems can save time in some areas, they take time to set up and use properly. Depending on the surgeon this could be a net savings but can also result in a net increase in surgical time.

[0048] Additionally, every time an optical tracker is used, an additional transformation matrix is required to localize it and the additional transformations may introduce additional errors. Such transformations lead to errors depending on the distance the trackers have with the anatomy in question. In many cases trackers are more than 6 inches away from relevant anatomy, so any angular error is magnified by that distance. It is an advantage to navigate based on anatomy to remove this distance error as much as possible.

[0049] As mentioned, with current optical tracking navigation systems the line of sight between the cameras and optical trackers must remain unobstructed for the system to functionVIS-004-W01 effectively. However, during surgery, this line of sight can be blocked by surgical staff or equipment, causing the system to lose track of the optical markers. Limited field of view (FOV) of stereoscopic cameras can cause tracking loss if anatomy or instruments move outside the tracked area, necessitating frequent repositioning. These interruptions necessitate recalibration or repositioning, disrupting the surgical workflow and potentially adding stress and time to the procedure. For example, intraoperative re-registration is required after a tracker becomes dislodged or bumped such that it shifts relative to anatomy. This re-registration can significantly prolong surgery, impacting workflow and anesthesia time.

[0050] Additionally, there are limits to the speed of the system. If OTs are moving too fast, the system can lose registrations. When this happens, the system must notify the surgeon, and the work must slow down.

[0051] High cost and limited adoption is another challenge of current optical tracking navigation systems. The substantial resources required to acquire, maintain, and operate these systems have limited their widespread adoption. Beyond the financial investment, the time and effort needed for training surgical teams, the complexity of setup, and the extended procedural time also present significant barriers. These systems demand highly trained personnel to ensure proper use, and the intricate setup process can increase the overall time required for surgeries. Additionally, the ongoing need for system and / or tool calibration and troubleshooting during procedures adds to the operational complexity, making it challenging for smaller hospitals and clinics to adopt these systems. As a result, accessibility remains limited, particularly in resource- constrained environments.

[0052] While stereoscopic cameras and optical trackers have become valuable tools for improving surgical precision, they come with several limitations. The reliance on physical optical trackers, complex setup, time-intensive calibration, and susceptibility to mechanical failures all point to the need for a more robust and flexible solution. Additionally, the combination of high financial costs, lengthy training requirements, and the complexity of operating these systems has made broader adoption across surgical specialties challenging.VIS-004-W01

[0053] Surgical navigation is crucial for enhancing precision in various surgical procedures, especially in orthopedics and neurology. Currently, two main methods are used: image-based navigation and imageless navigation.

[0054] Image-based navigation utilizes preoperative imaging techniques such as CT scans, MRIs, and X-rays to create a detailed three-dimensional model of the patient's anatomy. This model provides real-time guidance during surgery, with tracking markers placed on the patient and surgical instruments to ensure precision. While this approach offers high accuracy, it has significant drawbacks. The need for extensive preoperative imaging increases procedure costs and preparation time and exposes the patient to considerable amounts of radiation. Additionally, the reliance on these imaging methods can complicate insurance coverage approvals, as not all procedures may be deemed necessary or covered under certain policies.

[0055] In contrast to image-based methods, imageless navigation systems operate without preoperative imaging, relying instead on intraoperative data collection. During surgery, the surgeon manually identifies critical anatomical landmarks, which are then used to build a virtual model that guides the procedure. While this approach eliminates the need for radiation exposure associated with imaging, it heavily depends on the surgeon's ability to accurately identify landmarks. This reliance can introduce significant variability in outcomes, particularly in complex surgeries where precise landmark identification is challenging. Such variability underscores the need for more reliable and automated solutions in surgical navigation.

[0056] Typical Imageless Navigation Surgical Workflow

[0057] Transitioning from traditional image-based methods, imageless navigation represents a significant advancement in surgical technology, minimizing patient exposure to radiation and streamlining preoperative procedures. This section outlines a typical workflow of imageless navigation, emphasizing enhancements provided by the Personalized Alignment™ method using the ROSA® Knee System (owned by Zimmer Biomet).

[0058] Imageless surgical navigation primarily relies on intraoperative data rather than preoperative imaging, focusing on the surgeon's ability to identify anatomical landmarks in realtime. By way of example, a typical workflow may unfold as follows:VIS-004-W01

[0059] Step 1 : Preoperative Patient Positioning and Setup: The patient is positioned supine on the surgical table, with the leg stabilized to maintain the necessary surgical angles. This positioning provides the stable reference needed for accurate data acquisition.

[0060] Step 2: System Calibration and Setup: Surgeons and / or medical center staff calibrate the navigation system including the stereo camera system using reference markers (e.g. optical trackers or OTs) strategically placed on the patient, on the instruments, and on the surgical robot. This step ensures that the system can accurately track surgical instruments relative to the patient’s anatomy throughout the procedure.

[0061] Step 3: Anatomical Landmark Digitization: Key anatomical landmarks are identified and digitized directly on the patient’s joint. These landmarks form the basis for constructing a virtual model that guides the implant placement, emphasizing the need for precision in landmark identification. For example, a surgeon may use a pointer with an OT attached to point to specific anatomic landmarks. The pointer and the OT attached to the patient’s bone must not be within the field of view. Because multiple transformations are required, to bring the anatomic landmark and the bones OT into the same space, error can accumulate.

[0062] Step 4: Virtual Model Creation and Surgical Planning: A 3D model of the joint is created from the digitized data. For example, in a Total Knee Arthroplasty (TKA) procedure, a standard femur model with landmarks is loaded in the computer. After to pointing to the landmarks on the same bone (e.g., during step 3 above), the data is compared and the standard model is scaled and deformed to match the collected data as closely as possible. This process may be referred to as imageless registration. Registration is typically an iterative process where one model is moved (translated) into the same space. In this case, the collected data is moved, scaled and deformed using a very sparsely collected data set of approx. 50 points. This initial registration process is used to update the model. Surgeons use this model to plan the surgical procedure, including where to make bone cuts and how to position the implant optimally.

[0063] Step 5: Bone Cutting and Robotic Arm Guidance: After surgical planning, the cut planes for each bone cut are derived from the selected implant and are positioned relative to the bone so that the implant will fit properly.. The robotic system aids in ensuring that these cuts are executed with high accuracy according to the pre-planned surgical blueprint.VIS-004-W01

[0064] Step 6: Trial Component Placement and Assessment: After the cuts, trial components are placed to test the fit and alignment. This trial insertion is critical to verify that the executed surgical plan was good and executed properly.

[0065] Step 7: Final Implant Placement: Once alignment and fit are verified, the final implants are positioned.

[0066] Step 8: Postoperative Care Protocol: Completing the surgical process, patients are moved to recovery, where their initial responses to the surgery are monitored. A tailored postoperative care plan is initiated based on the specific outcomes of the surgery to optimize recovery.

[0067] Enhancements in Imageless Navigation: Personalized Alignment™ with ROSA System

[0068] Traditional imageless navigation methods rely heavily on the surgeon’s skill in manually identifying anatomical landmarks, which can lead to variability in implant placement and surgical outcomes. The Personalized Alignment™ method emerged as an enhancement to mitigate these limitations by providing a more tailored approach. Its primary goal is to restore the joint to a pre-arthritic state by aligning implants to the patient’s unique anatomy, taking into account cartilage thickness and bone geometry. By integrating these additional anatomical details, Personalized Alignment™ enhances the precision of joint restoration, serving as a bridge between traditional imageless techniques and the advanced automated mapping offered by ASM. As described below, ASM builds upon these enhancements while also addressing the limitations of traditional imageless navigation to further reduce surgeon variability and automate the entire navigation process, setting a new standard in surgical precision and efficiency.

[0069] Objectives of the Personalized Alignment™ Method:

[0070] Restoration of Pre-Arthritic Joint Geometry: The primary goal is to return the joint to its pre-arthritic state, enhancing postoperative functionality and extending the lifespan of the implant.VIS-004-W01

[0071] Precision in Implant Alignment: This method aims to improve patient outcomes by ensuring that the implant aligns accurately with the natural mechanics of each individual’s knee biomechanics, thereby improving functionality and patient satisfaction.

[0072] Additional Data Requirements in Personalized Alignment™:

[0073] Cartilage Thickness Measurement: Unique to the Personalized Alignment™ method, this involves measuring the thickness of the cartilage at various points around the joint. As with typical imageless techniques, the Personalized Alignment method also relies on proper identification of bone structure landmarks. Additionally, this method assesses cartilage thickness to understand the extent of wear or degenerative changes. These measurements help tailor the surgical plan to account for cartilage changes from overuse, trauma, and / or disease, enhancing the precision of the implant placement.

[0074] These enhancements address several limitations found in traditional imageless navigation by ensuring that every aspect of the surgery is informed by detailed, real-time anatomical data. The addition of cartilage thickness measurements allows for a more personalized surgical approach, taking into account individual variations that significantly impact the efficacy of the implant and the overall surgical outcome. By incorporating these precise measurements, the Personalized Alignment™ Methodology replaces other methodologies like mechanical alignment, kinematic alignment, gap balancing, inverse kinematic alignment and others to, in theory, restore the patient knee to a pre-arthritic condition which should yield a better outcome for the patient.

[0075] Accurate real-time tracking of objects in dynamic environments is critical in many fields, particularly in surgical navigation, robotics, and automation. Traditional object tracking methods rely on pre-existing reference models, fiducial markers, or preoperative imaging to register and track objects. While effective, these approaches face limitations in real-world applications where no prior object data is available.

[0076] In surgery, tracking systems often depend on preoperative CT or MRI scans, which provide a 3D model for navigation. However, in many cases, such imaging data is unavailable, or the relevant anatomical structures are obscured, such as cartilage covering bone in orthopedicVIS-004-W01 procedures. Additionally, traditional fiducial markers (optical arrays) can be difficult to use, expensive and error prone, as they require pre-placement and introduce workflow complexity.

[0077] Outside of medicine, real-time object tracking is crucial in robotics and manufacturing, where objects of varying shapes and orientations must be identified and monitored without prior knowledge of their geometry. Conventional methods often require predefined CAD models or visual markers, which limit adaptability in unstructured or rapidly changing environments.SUMMARY

[0078] This disclosure builds upon the innovations presented in U.S. Patent Application No. 18 / 505,973 (U.S. Pub. 2024 / 0102795) (“the ‘973 app”), filed Nov. 9, 2023 and entitled “Generation of One or More Edges of Luminosity to Form Three-Dimensional Models of Scenes”. More specifically, the present disclosure introduces advanced clinical applications for the previously described 3D scanning system. The ‘973 app unveiled a groundbreaking 3D scanner whose capabilities were not previously available in surgical settings. The present disclosure expands on the utility of that technology, particularly emphasizing its application in orthopedic surgeries such as Total Knee Arthroplasty (TKA) and Robotic Assisted TKA (RATKA).

[0079] The enhancements detailed herein expand upon the exemplary clinical uses disclosed in the ‘973 app, with the goal of illustrating how the technology can be highly beneficial and tailored in certain arthroplasty procedures. For example, by leveraging advanced scanning methodologies, the system reduces the need for optical trackers, streamlining setup and improving surgical precision. While this disclosure can be applied broadly across multiple surgical domains, particular emphasis is given to its use in orthopedic procedures such as TKA. The subsequent sections will outline the significant advancements this technology brings to the clinical environment, setting a new standard for precision and efficiency in surgical navigation and instrumentation. Moreover, although described herein in this context as a specific example, it should be understood that the system and methods disclosed herein may be used in any scenario (surgical, medical, or otherwise) to generate a reconstructed 3D model of an object,VIS-004-W01 register it to a reference object in a different coordinate space, and track movement of the object in real space with respect to the reconstructed 3D model of the object within the coordinate space.

[0080] The present disclosure describes a 3D scanner specifically designed for surgical applications, capable of collecting visible topological data from the surgical field in real time. This represents a new class of data that was previously unavailable during surgery. By capturing continuous, high-rate 3D topological data during the procedure, this scanner can replace traditional workflows that rely on optical trackers — without the associated costs, setup time, and limitations. Although described within the context of surgical applications as a specific example, the system and methods disclosed herein may be used in a wide variety of non-surgical applications to register and track an object within a coordinate space.

[0081] This disclosure expands on how this topological data can be applied in a surgical setting to perform various procedures. Although the current disclosure emphasizes applications in TKA, the underlying principles can be adapted for other surgeries with minimal modifications by those skilled in the art. While the 3D scanner described herein is the preferred method for collecting this data, the use of the scanner is not a requirement for this innovation, as other technology capable of capturing 3D surface data during surgery may be utilized to achieve similar results.

[0082] When a continuously updated 3D topological data of sufficient accuracy and rate of the surgical site is available, the data can be used for Continuous Anatomical Auto Tracking (CAAT). This innovation eliminates the need for introduced physical optical markers and reduces the complexity of setup, offering a more streamlined, cost-effective, and accurate approach to surgical navigation. CAAT allows for real-time registration and tracking of anatomical structures, instruments and equipment without using OMs or OTs or other fiducials, marking a significant advancement in surgical technology. CAAT promises greater precision, flexibility, and efficiency in the operating room, benefiting a wide range of surgical procedures.

[0083] Additionally, this disclosure introduces Articular Surface Matching (ASM), a groundbreaking approach to imageless surgical navigation that significantly enhances the accuracy and efficiency of joint replacement procedures, where the goal is to restore a patient’sVIS-004-W01 pre-arthritic joint biomechanics. In some embodiments, ASM addresses critical limitations found in both traditional image-based and existing imageless methods by automating the mapping of joint surfaces without requiring manual identification of anatomical landmarks. Leveraging dense, high-resolution 3D topological mapping, ASM provides both speed and consistency across procedures by generating precise anatomical references that guide optimal implant selection and placement.

[0084] By way of example only, ASM works by analyzing intraoperatively acquired topological data of a patient’s articular surfaces, for example, the medial and lateral femoral condyles and the tibial plateau. In some embodiments, this data is then registered to digital models of available implants to assess which implant design and size will best match the patient’s unique anatomy. By avoiding manual landmark identification, ASM reduces intraoperative time, minimizes variability across surgeons, and mitigates potential errors associated with traditional imageless methods. ASM further improves upon the traditional methods by assessing the cartilage thickness and then estimating the pre-arthritic thicknesses. This additional data is used to refine the registration of the implants to improve the biomechanics.

[0085] ASM enhances patient safety by eliminating the need for preoperative imaging, thus reducing radiation exposure. Additionally, the system’s ability to precisely match and align implants based on individual joint morphology holds significant potential to improve implant longevity, restore pre-arthritic joint biomechanics, and elevate the standard of care for surgeries requiring high anatomical alignment accuracy.

[0086] In some embodiments, this disclosure introduces Zero-Reference Tracking (ZRT), a novel approach that enables real-time 3D tracking of any object, without the need for preexisting models. In some embodiments, ZRT embodies a generalized method for tracking objects in real-time during surgery using a topological 3D scanner as a surgical navigation tool. Unlike traditional tracking systems that rely on pre-existing 3D models, fiducial markers, or preoperative imaging, ZRT allows one or more objects to be identified and tracked entirely from intraoperative scan data. This technique is referred to as “Zero-Reference Tracking (ZRT)” because it can be applied to any object with no prior knowledge of its shape or structure required.VIS-004-W01

[0087] In some embodiments, a method for tracking objects in dynamic environments without requiring pre-existing reference models, fiducial markers, or preoperative imaging data is disclosed herein.

[0088] In some embodiments, a first step of the method is scanning a scene using a structured light scanner or another 3D imaging system, for example like the scanner described in the ‘973 app-

[0089] In some embodiments, a second step of the method is segmenting one or more target objects from the scene, creating a reference model or topology. By way of example, the segmented object(s) become the reference dataset for tracking the target throughout a procedure. In some embodiments, the reference dataset may be continuously updated to reflect the real-time state of the anatomy (changed after a cut, for example). By way of example, this can be done with scans, or there may be a modified version of the initial reference scan, modified by the surgical plan. The result would be to approximate the surface area left by the procedure.

[0090] In some embodiments, a third step of the method is registration and tracking in subsequent scans. By way of example, as new scans are taken, the system registers the reference model to the live scan data, continuously tracking the object(s) over time in 6DOF.

[0091] In some embodiments, the kinematic data for a target / object can be extrapolated to predict an object’s kinematics in six degrees of freedom at a future time point.

[0092] The present disclosure relates to real-time 3D object tracking and spatial registration using topological scanning technology. More specifically, it pertains to a method for tracking objects in dynamic environments without requiring pre-existing reference models, fiducial markers, or preoperative imaging data.

[0093] The method disclosed herein is particularly applicable in surgical navigation, where precise tracking of anatomical structures, surgical instruments, and medical devices is critical. By enabling intraoperative scanning, segmentation, and tracking of objects, ZRT provides a unique solution for real-time localization and motion analysis in medical procedures.VIS-004-W01

[0094] Beyond surgical applications, this invention has broad implications for robotics, manufacturing, and automation, where object tracking in unstructured environments is required. The ability to generate and update a reference model from real-time scan data and track objects dynamically allows for improved automation, enhanced spatial awareness, and greater adaptability in industrial and robotic systems.

[0095] In some embodiments, the CAAT system described herein may employ a process referred to as “transfer registration.” By way of example, transfer registration is a specific implementation of zero reference tracking, wherein each newly acquired optical scan is registered to a prior reference scan representing the immediately preceding anatomical state. By way of example, as that anatomy has not been seen before, it may be considered to have zero reference. It may, however, have been approximated by applying the planned surgical action on the initial (or just-prior) reference, as an initial step to understanding the anatomy for refined registration. In some embodiments, each registration thereby links the current intraoperative anatomical state to the previous anatomical state. Through a sequence of intermediate anatomical states, the current anatomical state is linked to the original anatomic reference model. Because each transformation is based on successive stages of anatomy exhibiting maximum geometric commonality, the resulting registrations can be statistically robust.

[0096] In some embodiments, successful registrations rely on statistical sufficiency of common anatomical features between consecutive stages. It is important in transfer registration to maintain common anatomy sufficient for confident registration in all six degrees of freedom. For this reason, it is likely that scans will take place at standard timepoints in a procedure. Metrics on the number of inliers and their residual errors, and statistical outputs such as covariance matrices could enable the system to quantify the robustness of each stage’s registration.

[0097] In surgical contexts such as TKA or spine surgery, transfer registration may be used to update the reference model as anatomical modifications occur. For example, in TKA, the updated reference may exclude bone that has been resected and include newly exposed cut surfaces. In spine procedures, the reference may similarly exclude removed bony structures while incorporating newly revealed rigid anatomy. Rigidly affixed, relatively immobile, orVIS-004-W01 otherwise untouched tissue may also contribute to stable registration across stages when such tissue remains unaffected by the surgical manipulation.

[0098] In some embodiments, between major stages of anatomical change, the system may acquire new optical scans to update the current reference frame. For example, during stages in which the anatomy remains relatively unchanged (e.g., during drilling or probing operations), the system may rely on continuous motion tracking relative to the most recent reference, maintaining spatial accuracy in real time. In this manner, the system combines stage-based re-registration with intra-stage tracking to preserve registration accuracy throughout the procedure.

[0099] This approach allows intraoperative registration updates without cumulative drift exceeding acceptable limits, provided that each individual registration meets accuracy requirements and sufficient anatomical overlap exists between consecutive states.

[0100] As additional description to the embodiments described below, the present disclosure describes the following embodiments.

[0101] Embodiment 1 is a system for continuous registration and tracking of objects in an operating room environment, comprising: a structured light three-dimensional scanner, including: a light source; a liquid crystal matrix; and an image capture device; a computer system including a memory and at least one processor; and computer readable media embodied in a non- transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a three-dimensional data representation a scene within a field of view of the image capture device using a first dataset, the first dataset derived from one or more initial images captured by the image capture device without optical trackers or applied fiducials, the scene including one or more physical objects; isolate a unique target object from the one or more physical objects within the scene by segmentation of a target reference dataset from the first dataset; register the target reference dataset to a second dataset such that the target reference dataset and the second dataset are aligned within a common coordinate system, the second dataset derived from one or more subsequent images captured by the image capture device without said optical trackers or applied fiducials; and track the unique target object in six degrees of freedom relative to the common coordinate system.VIS-004-W01

[0102] Embodiment 2 is the system of embodiment 1 , wherein the light source is a linear light source.

[0103] Embodiment 3 is the system of embodiments 1 or 2, wherein the three-dimensional data representation is a point cloud or three-dimensional data convertible into a point cloud.

[0104] Embodiment 4 is the system of any of embodiments 1 through 3, wherein the one or more initial captured images and the one or more subsequent captured images are structured light patterned images that encode the surface of the physical object.

[0105] Embodiment 5 is the system of any of embodiments 1 through 4, wherein the three- dimensional structured light scanner further includes a heat reduction assembly.

[0106] Embodiment 6 is the system of any of embodiments 1 through 5, wherein the target object is an anatomical structure.

[0107] Embodiment 7 is the system of any of embodiments 1 through 6, wherein the target object is a surgical instrument.

[0108] Embodiment 8 is the system of any of embodiments 1 through 7, wherein the three- dimensional data representation of the topological surface of the scene is constructed by: decoding light patterns contained in the one or more initial captured images by analyzing brightness history to create camera pixel correspondence to light angle projection; using triangulation to calculate a three-dimensional coordinate for each pixel in the one or more initial captured images based on the decoded light patterns; and combining the triangulated three- dimensional coordinates of each pixel into the three-dimensional point cloud representation of the surface of the scene in physical space.

[0109] Embodiment 9 is the system of any of embodiments 1 through 8, wherein the computer readable media includes a further set of instructions, that, when executed by a processor, cause the computer system to: isolate a second unique target object from the one or more physical objects within the scene by segmentation of a second target reference dataset from the first dataset, register the second target reference dataset to a third dataset such that the second target reference dataset and the third dataset are aligned within the common coordinate system,VIS-004-W01 the third dataset derived from the one or more subsequent images captured by the image capture device; and track the second unique target object relative to the common coordinate system.

[0110] Embodiment 10 is the system of any of embodiments 1 through 9, wherein the unique target object is a first anatomical structure and the second unique target object is a second anatomical structure.

[0111] Embodiment 11 is the system of any of embodiments 1 through 10, wherein the unique target object is an anatomical structure and the second unique target object is a surgical instrument.

[0112] Embodiment 12 is the system of any of embodiments 1 through 11, wherein: the computer system further comprises a user interface and a display device, and the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display the tracked three-dimensional data representation of the unique target object on the display device.

[0113] Embodiment 13 is the system of any of embodiments 1 through 12, wherein the first dataset is captured by the image capture device from multiple different angles or viewpoints.

[0114] Embodiment 14 is the system of any of embodiments 1 through 13, wherein the segmentation of a target reference dataset from the first dataset comprises at least one of manual selection, algorithmic detection, machine learning, or via registration of objects of known shapes.

[0115] Embodiment 15 is the system of any of embodiments 1 through 14, wherein the six degrees of freedom comprise x-coordinate position, y-coordinate position, z-coordinate position, roll orientation, pitch orientation, and yaw orientation.

[0116] Embodiment 16 is the system of any of embodiments 1 through 15, wherein tracking the unique target object relative to the common coordinate system comprises continuously updating the unique target object’s position relative to the common coordinate system by repeatedly acquiring surface data, registering the repeatedly acquired surface data in the common coordinate system, and updating transformation matrices.

[0117] Embodiment 17 is the system of any of embodiments 1 through 16, wherein theVIS-004-W01 computer readable media includes a further set of instructions, that, when executed by a processor, cause the computer system to: compute at least one of velocity, angular velocity, acceleration, and angular acceleration of the target object within the common coordinate system; and extrapolate the values of the computed at least one of velocity, angular velocity, acceleration, and angular acceleration of the target object within the common coordinate system; and estimate at least one of a future x-coordinate position, a future y-coordinate position, a future z-coordinate position, a future roll orientation, a future pitch orientation, and a future yaw orientation of the target object based on the extrapolated values.

[0118] Embodiment 18 is a system for continuous registration and tracking of objects in an operating room environment, comprising: a structured light three-dimensional scanner, including: a light source; a liquid crystal matrix; and an image capture device; a computer system including a memory and at least one processor; and computer readable media embodied in a non- transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a three-dimensional data representation a scene within a field of view of the image capture device using a first dataset, the first dataset derived from one or more initial images captured by the image capture device without optical trackers or applied fiducials, the scene including one or more physical objects; isolate a unique target object from the one or more physical objects within the scene by segmentation of a target reference dataset from the first dataset; register the target reference dataset to a second dataset such that the target reference dataset and the second dataset are aligned within a common coordinate system, the second dataset derived from one or more subsequent images captured by the image capture device without said optical trackers or applied fiducials; track the unique target object in six degrees of freedom relative to the common coordinate system by continuously updating the unique target object’s position relative to the common coordinate system by repeatedly acquiring surface data, registering the repeatedly acquired surface data in the common coordinate system, and updating transformation matrices; compute at least one of velocity, angular velocity, acceleration, and angular acceleration of the target object within the common coordinate system; and extrapolate the values of the computed at least one of velocity, angular velocity, acceleration, and angular acceleration of the target object within the common coordinate system; and estimate at least one of a future x-coordinate position, a future y- coordinate position, a future z-coordinate position, a future roll orientation, a future pitchVIS-004-W01 orientation, and a future yaw orientation of the target object based on the extrapolated values.

[0119] Embodiment 19 is the system of embodiment 18, wherein the light source is a linear light source.[00120J Embodiment 20 is the system of embodiments 18 or 19, wherein the three-dimensional data representation is a point cloud or three-dimensional data convertible into a point cloud.

[0121] Embodiment 21 is the system of any of embodiments 18 through 20, wherein the one or more initial captured images and the one or more subsequent captured images are structured light patterned images that encode the surface of the physical object.

[0122] Embodiment 22 is the system of any of embodiments 18 through 21, wherein the three-dimensional structured light scanner further includes a heat reduction assembly.

[0123] Embodiment 23 is the system of any of embodiments 18 through 22, wherein the target object is an anatomical structure.

[0124] Embodiment 24 is the system of any of embodiments 18 through 23, wherein the target object is a surgical instrument.

[0125] Embodiment 25 is the system of any of embodiments 18 through 24, wherein the three-dimensional data representation of the topological surface of the scene is constructed by: decoding light patterns contained in the one or more initial captured images by analyzing brightness history to create camera pixel correspondence to light angle projection; using triangulation to calculate a three-dimensional coordinate for each pixel in the one or more initial captured images based on the decoded light patterns; and combining the triangulated three- dimensional coordinates of each pixel into the three-dimensional point cloud representation of the surface of the scene in physical space.

[0126] Embodiment 26 is the system of any of embodiments 18 through 25, wherein the computer readable media includes a further set of instructions, that, when executed by a processor, cause the computer system to: isolate a second unique target object from the one or more physical objects within the scene by segmentation of a second target reference dataset from the first dataset; register the second target reference dataset to a third dataset such that the secondVIS-004-W01 target reference dataset and the third dataset are aligned within the common coordinate system, the third dataset derived from the one or more subsequent images captured by the image capture device; and track the second unique target object relative to the common coordinate system.

[0127] Embodiment 27 is the system of any of embodiments 18 through 26, wherein the unique target object is a first anatomical structure and the second unique target object is a second anatomical structure.

[0128] Embodiment 28 is the system of any of embodiments 18 through 27, wherein the unique target object is an anatomical structure and the second unique target object is a surgical instrument.

[0129] Embodiment 29 is the system of any of embodiments 18 through 28, wherein: the computer system further comprises a user interface and a display device, and the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display the tracked three-dimensional data representation of the unique target object on the display device.

[0130] Embodiment 30 is the system of any of embodiments 18 through 29, wherein the first dataset is captured by the image capture device from multiple different angles or viewpoints.

[0131] Embodiment 31 is the system of any of embodiments 18 through 30, wherein the segmentation of a target reference dataset from the first dataset comprises at least one of manual selection, algorithmic detection, machine learning, or via registration of objects of known shapes.

[0132] Embodiment 32 is the system of any of embodiments 18 through 31, wherein the six degrees of freedom comprise x-coordinate position, y-coordinate position, z-coordinate position, roll orientation, pitch orientation, and yaw orientation.

[0133] Embodiment 33 is a system for identification and selection of an optimal surgical implant in an operating room environment, comprising: a structured light three-dimensional scanner, including: a light source: a liquid crystal matrix; and an image capture device; a computer system including a memory and at least one processor; and computer readable media embodied in a non-transitory storage medium comprising a set of instructions that, whenVIS-004-W01 executed by the one or more processors, cause the computer system to: generate a real-time anatomical model of a surgical site that serves as a surface map of one or more articular surfaces of a patient by: constructing a three-dimensional data representation of a topological articular surface of a physical object using a first dataset, the first dataset derived from one or more images captured by the image capture device without optical trackers or applied fiducials; registering the three-dimensional data representation to a second dataset such that the three- dimensional data representation and the second dataset are aligned within a common coordinate system, the registration occurring without said optical trackers or applied fiducials; and tracking the physical object relative to the common coordinate system without said optical trackers or applied fiducials; individually register the constructed three-dimensional data representation to one or more implant models within a library of implant models to create a set of implant model registrations, each of the implant models within the library of implant models representing a unique physical implant configured to engage the articular surface of the physical object; evaluate the quality of each implant model registration in the set of implant model registrations using a computational fitness function; and identify a best-fitting implant based on the evaluation of the quality of each implant model registration in the set of implant model registrations.

[0134] Embodiment 34 is the system of embodiment 33, wherein the light source is a linear light source.

[0135] Embodiment 35 is the system of embodiments 33 or 34, wherein the three-dimensional data representation is a point cloud or three-dimensional data convertible into a point cloud.

[0136] Embodiment 36 is the system of any of embodiments 33 through 35, wherein the captured images are structured light patterned images that encode the surface of the physical object.

[0137] Embodiment 37 is the system of any of embodiments 33 through 36, wherein the physical object is an anatomical structure in a surgical procedure.

[0138] Embodiment 38 is the system of any of embodiments 33 through 37, wherein the three-dimensional data representation of the topological articular surface of the physical object is constructed by: decoding light patterns contained in the one or more captured images byVIS-004-W01 analyzing brightness history to create camera pixel correspondence to light angle projection; using triangulation to calculate a three-dimensional coordinate for each pixel in the one or more captured images based on the decoded light patterns; and combining the triangulated three- dimensional coordinates of each pixel into the three-dimensional point cloud representation of the surface of the physical object in physical space.

[0139] Embodiment 39 is the system of any of embodiments 33 through 38, wherein the three-dimensional data representation of the physical object comprises a first three-dimensional data representation of a first physical object, and the computer readable media includes a further set of instructions, that, when executed by a processor, cause the computer system to: construct a second digital three-dimensional data representation of a topological surface of a second physical object using a third dataset, the third dataset derived from one or more images of surface topology of the second physical object captured by the image capture device without the use of optical trackers or applied fiducials; register the second three-dimensional data representation to a fourth dataset such that the second three-dimensional data representation and the fourth dataset are aligned with one another and the with the first three-dimensional data representation and second dataset within the common coordinate system without the use of optical trackers or applied fiducials; continuously track the second physical object relative to the common coordinate system without the use of optical trackers or applied fiducials.

[0140] Embodiment 40 is the system of any of embodiments 33 through 39, wherein the first physical object is a first anatomical structure and the second physical object is a second anatomical structure.

[0141] Embodiment 41 is the system of any of embodiments 33 through 40, wherein the first physical object is an anatomical structure and the second physical object is a surgical instrument.

[0142] Embodiment 42 is the system of any of embodiments 33 through 41, wherein the second dataset is derived from a preoperative computed tomography scan or magnetic resonance imaging scan.

[0143] Embodiment 43 is the system of any of embodiments 33 through 42, wherein: the computer system further comprises a user interface and a display device, and the computerVIS-004-W01 readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display, on a display device, the tracked three-dimensional data representation of the physical object.

[0144] Embodiment 44 is the system of any of embodiments 33 through 43, wherein tracking the physical object relative to the common coordinate system comprises continuously updating the physical object’s position relative to the common coordinate system by repeatedly acquiring surface data, registering the repeatedly acquired surface data in the common coordinate system, and updating transformation matrices.

[0145] Embodiment 45 is the system of any of embodiments 33 through 44, wherein the fitness function comprises one of least-squares matching, curvature alignment, surface area overlap, multi-factor scoring, fractional area overlap, and stability of registration in six degrees of freedom.

[0146] Embodiment 46 is the system of any of embodiments 33 through 45, wherein the computer readable media includes a further set of instructions, that, when executed by a processor, cause the computer system to: dynamically adjust implant positioning by correcting for changes in cartilage thickness.

[0147] Embodiment 47 is the system of any of embodiments 33 through 46, wherein the implant positioning is manually adjusted by a surgeon during a surgical planning event based on the surgeon’s knowledge and expertise in accounting for disease-related changes in bone shape and cartilage thickness gained through experience.

[0148] Embodiment 48 is the system of any of embodiments 33 through 47, wherein the implant positioning is adjusted automatically based on intraoperative measurements taken with a registered probe instrument.

[0149] Embodiment 49 is the system of any of embodiments 33 through 48, wherein the implant positioning is adjusted automatically based on estimated typical cartilage thickness derived from published statistical data.

[0150] Embodiment 50 is the system of any of embodiments 33 through 49, wherein theVIS-004-W01 implant positioning is adjusted automatically based on a computation of cartilage thickness derived from the registration of the first dataset to the second dataset, wherein the first dataset comprises cartilage and bone surface data of the physical object.

[0151] Embodiment 51 is the system of any of embodiments 33 through 50, wherein the implant positioning is adjusted automatically based on tissue-specific responses to scanner- derived tissue differentiation to estimate cartilage thickness.

[0152] Embodiment 52 is the system of any of embodiments 33 through 51, wherein the implant positioning is manually adjusted by a surgeon during a surgical planning event based on the surgeon’s knowledge and expertise in accounting for disease-related changes in bone shape and cartilage thickness gained through experience and intraoperative measurements taken with a registered probe instrument.

[0153] Embodiment 53 is a system for identification and selection of an optimal surgical implant in an operating room environment, comprising: a structured light three-dimensional scanner, including: a light source: a liquid crystal matrix; and an image capture device; a computer system including a memory and at least one processor; and computer readable media embodied in a non-transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: generate a real-time anatomical model of a surgical site that serves as a surface map of one or more articular surfaces of a patient by: constructing a three-dimensional data representation of a topological articular surface of a physical object using a first dataset, the first dataset derived from one or more images captured by the image capture device without optical trackers or applied fiducials; registering the three-dimensional data representation to a second dataset such that the three- dimensional data representation and the second dataset are aligned within a common coordinate system, the registration occurring without said optical trackers or applied fiducials; and tracking the physical object relative to the common coordinate system without said optical trackers or applied fiducials; individually register the constructed three-dimensional data representation to one or more implant models within a library of implant models to create a set of implant model registrations, each of the implant models within the library of implant models representing a unique physical implant configured to engage the articular surface of the physical object;VIS-004-W01 evaluate the quality of each implant model registration in the set of implant model registrations using a computational fitness function; identify a best-fitting implant based on the evaluation of the quality of each implant model registration in the set of implant model registrations; dynamically adjust implant positioning by correcting for changes in cartilage thickness; and after dynamically adjusting implant positioning by correcting for changes in cartilage thickness, individually re-register the constructed three-dimensional data representation to two or more implant models within the library of implant models to create a set of implant model reregistrations; evaluate the quality of each implant model re-registration in the set of implant model re-registrations using the computational fitness function; and identify or confirm the bestfitting implant based on the evaluation of the quality of each implant model re-registration in the set of implant model re-registrations.

[0154] Embodiment 54 is the system of embodiment 53, wherein the light source is a linear light source.

[0155] Embodiment 55 is the system of embodiments 53 or 54, wherein the three-dimensional data representation is a point cloud or three-dimensional data convertible into a point cloud.

[0156] Embodiment 56 is the system of any of embodiments 53 through 55, wherein the captured images are structured light patterned images that encode the surface of the physical object.

[0157] Embodiment 57 is the system of any of embodiments 53 through 56, wherein the physical object is an anatomical structure in a surgical procedure.

[0158] Embodiment 58 is the system of any of embodiments 53 through 57, wherein the three-dimensional data representation of the topological articular surface of the physical object is constructed by: decoding light patterns contained in the one or more captured images by analyzing brightness history to create camera pixel correspondence to light angle projection; using triangulation to calculate a three-dimensional coordinate for each pixel in the one or more captured images based on the decoded light patterns; and combining the triangulated three- dimensional coordinates of each pixel into the three-dimensional point cloud representation of the surface of the physical object in physical space.VIS-004-W01

[0159] Embodiment 59 is the system of any of embodiments 53 through 58, wherein the three-dimensional data representation of the physical object comprises a first three-dimensional data representation of a first physical object, and the computer readable media includes a further set of instructions, that, when executed by a processor, cause the computer system to: construct a second digital three-dimensional data representation of a topological surface of a second physical object using a third dataset, the third dataset derived from one or more images of surface topology of the second physical object captured by the image capture device without the use of optical trackers or applied fiducials; register the second three-dimensional data representation to a fourth dataset such that the second three-dimensional data representation and the fourth dataset are aligned with one another and the with the first three-dimensional data representation and second dataset within the common coordinate system without the use of optical trackers or applied fiducials; continuously track the second physical object relative to the common coordinate system without the use of optical trackers or applied fiducials.

[0160] Embodiment 60 is the system of any of embodiments 53 through 59, wherein the first physical object is a first anatomical structure and the second physical object is a second anatomical structure.

[0161] Embodiment 61 is the system of any of embodiments 53 through 60, wherein the first physical object is an anatomical structure and the second physical object is a surgical instrument.

[0162] Embodiment 62 is the system of any of embodiments 53 through 61, wherein the second dataset is derived from a preoperative computed tomography scan or magnetic resonance imaging scan.

[0163] Embodiment 63 is the system of any of embodiments 53 through 62, wherein: the computer system further comprises a user interface and a display device, and the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display, on a display device, the tracked three-dimensional data representation of the physical object.

[0164] Embodiment 64 is the system of any of embodiments 53 through 63, wherein tracking the physical object relative to the common coordinate system comprises continuously updatingVIS-004-W01 the physical object’s position relative to the common coordinate system by repeatedly acquiring surface data, registering the repeatedly acquired surface data in the common coordinate system, and updating transformation matrices.

[0165] Embodiment 65 is the system of any of embodiments 53 through 64, wherein the fitness function comprises one of least-squares matching, curvature alignment, surface area overlap, multi-factor scoring, fractional area overlap, and stability of registration in six degrees of freedom.

[0166] Embodiment 66 is the system of any of embodiments 53 through 65, wherein the implant positioning is manually adjusted by a surgeon during a surgical planning event based on the surgeon’s knowledge and expertise in accounting for disease-related changes in bone shape and cartilage thickness gained through experience.

[0167] Embodiment 67 is the system of any of embodiments 53 through 66, wherein the implant positioning is adjusted automatically based on intraoperative measurements taken with a registered probe instrument.

[0168] Embodiment 68 is the system of any of embodiments 53 through 67, wherein the implant positioning is adjusted automatically based on estimated typical cartilage thickness derived from published statistical data.

[0169] Embodiment 69 is the system of any of embodiments 53 through 68, wherein the implant positioning is adjusted automatically using based on tissue-specific responses to scanner- derived tissue differentiation to estimate cartilage thickness.

[0170] Embodiment 70 is the system of any of embodiments 53 through 69, wherein the implant positioning is adjusted automatically based on a computation of cartilage thickness derived from the registration of the first dataset to the second dataset, wherein the first dataset comprises cartilage and bone surface data of the physical object.

[0171] Embodiment 71 is the system of any of embodiments 53 through 70, wherein the implant positioning is manually adjusted by a surgeon during a surgical planning event based on the surgeon’s knowledge and expertise in accounting for disease-related changes in bone shapeVIS-004-W01 and cartilage thickness gained through experience and intraoperative measurements taken with a registered probe instrument.

[0172] Embodiment 72 is a method of updating an anatomic reference model used in executing a surgical plan during a surgical event, comprising the steps of: generating an original anatomic reference model representing a first anatomic state of one or more objects within a surgical site; registering and tracking the one or more objects within a common coordinate system; performing an action resulting in a first changed anatomic state of the one or more tracked objects in the surgical site; acquiring subsequent scan data of the surgical site using a structured light 3D scanner; updating the original reference model with the acquired scan data to generate a first updated reference model representing a second anatomic state of the one or more objects within the surgical site; performing a subsequent action resulting in second changed anatomic state of the one or more tracked objects in the surgical site; acquiring further subsequent scan data of the surgical site using the structured light 3D scanner; updating the first updated reference model with the acquired further subsequent scan data to generate a second updated reference model representing a third anatomic state of the one or more objects within the surgical site; and repeating the scan data capture and reference model updating cycle until the surgical procedure is concluded such that the surgical plan is carried out with a real-time updated reference model before each cut is made.BRIEF DESCRIPTION OF THE DRAWINGS

[0173] Many advantages of the present disclosure will be apparent to those skilled in the art with a reading of this specification in conjunction with the attached drawings, wherein like reference numerals are applied to like elements and wherein:

[0174] Fig. 1 illustrates an example of a prior art rigid transformation matrix for transferring a 3D data set from one coordinate system to another;

[0175] Fig. 2 is a flowchart depicting an example of a typical prior art workflow for a robotically assisted total knee arthroplasty;

[0176] Fig. 3 is a block diagram illustrating essential features of a continuous anatomical automatic tracking (CAAT) system of the present disclosure, according to some embodiments;VIS-004-W01

[0177] Fig. 4 is a perspective view of an example of a surgical environment utilizing the CAAT system of Fig. 3, according to some embodiments;

[0178] Fig. 5 is a perspective view of an example of a 3D scanner forming part of the CAAT system of the Fig. 3, according to some embodiments;

[0179] Fig. 6 is a perspective view of the 3D scanner of Fig. 5 with the cover removed, according to some e embodiments;

[0180] Fig. 7 is a plan view of the 3D scanner of Fig. 6, according to some embodiments;

[0181] Fig. 8 is another perspective view of the 3D scanner of Fig. 6, according to some embodiments;

[0182] Fig. 9 is an perspective view of an example of a rigid scaffolding forming part of the 3D scanner of Fig. 5, according to some embodiments;

[0183] Fig. 10 is an exploded view of an example of a light source assembly forming part of the 3D scanner of Fig. 5, according to some embodiments;

[0184] Figs. 11-13 are perspective, bottom plan, and side plan views, respectively, of an example of a pattern generator forming part of the 3D scanner of Fig. 5, according to some embodiments;

[0185] Fig. 14 is a partial exploded view of an example of a thermal management system forming part of the 3D scanner of Fig. 5, according to some embodiments;

[0186] Fig. 15 is a block diagram depicting an example of a data acquisition process using the CAAT system of Fig. 3, according to some embodiments;

[0187] Fig. 16 is an example of reconstructed point cloud of an exposed knee, simulated with sawbones and using a visible light source generated using the CAAT system of Fig. 3, according to some embodiments;

[0188] Figs. 17A-B depicts a typical prior art stereoscopic camera field of view (on the left) and what the cameras actually capture (on the right), according to some embodiments;VIS-004-W01

[0189] Figs. 18A-B depicts an image and 3D point cloud from a 3D scanner forming part of the CAAT system of Fig. 3, using infrared light instead of visible light, according to some embodiments;

[0190] Fig. 19 depicts a 3D point cloud from a 3D scanner forming part of the CAAT system of Fig. 3, using infrared light instead of visible light and shown at an oblique angle, which illustrates in particular the 3D nature of the regeneration, according to some embodiments;

[0191] Fig. 20 depicts the 3D point cloud of Fig. 60, with some anatomical structures segmented for easy identification, according to some embodiments;

[0192] Fig. 21 is a block diagram depicting an example of a method of registration of anatomical structures and / or surgical instruments using the CAAT system of Fig. 3, according to some embodiments;

[0193] Fig. 22 illustrates three data sets registered into the same coordinate space using the CAAT system of Fig. 3, particularly CT data from the femur and tibia registered to a 3D topological scan, according to some embodiments;

[0194] Fig. 23 is a block diagram depicting an example of a method of calculating a hip center of rotation using the CAAT system of Fig. 3, according to some embodiments;

[0195] Fig. 24 is a block diagram depicting an example of a method of assessing gaps between femoral condyles (e.g., medial and lateral) and a tibial plateau using the CAAT system of Fig. 3, according to some embodiments;

[0196] Fig. 25 is a block diagram depicting an example of a method of tracking movable anatomy and a movable surgical instrument configured to affect the movable anatomy using the CAAT system of Fig. 3, according to some embodiments;

[0197] Fig. 26A-B is a block diagram depicting an example of a method of complex anatomy tracking with robot-controlled instrument in the context of a Total Knee Arthroplasty (TKA) using the CAAT system of Fig. 3, according to some embodiments;VIS-004-W01

[0198] Fig. 27 is a block diagram depicting an example robotically assisted TKA workflow with pre-operative imaging using the CAAT system of Fig. 3, according to some embodiments;

[0199] Fig. 28 is a block diagram illustrating several steps of an example method for tracking objects in dynamic environments without requiring pre-existing reference models, fiducial markers, or preoperative imaging data using the CAAT system of Fig. 3, according to some embodiments.

[0200] Fig. 29 is a block diagram illustrating several steps of an example of a transfer registration method using image data captured by a 3D scanner of the CAAT system of Fig. 3, according to some embodiments;

[0201] Fig. 30 is a block diagram depicting an example robotically assisted TKA workflow without pre-operative imaging using the CAAT system of Fig. 3, according to some embodiments;

[0202] Figs. 31 and 32 are block diagrams illustrating example computer systems with which any of the devices or systems described herein may be implemented, according to some embodimentsDETAILED DESCRIPTION OF A PREFERRED EMBODIMENT

[0203] Illustrative embodiments of the disclosure are described below. In the interest of clarity, not all features of an actual implementation are described in this specification. It will of course be appreciated that in the development of any such actual embodiment, numerous implementation-specific decisions must be made to achieve the developers’ specific goals, such as compliance with system-related and business-related constraints, which will vary from one implementation to another. Moreover, it will be appreciated that such a development effort might be complex and time-consuming but would nevertheless be a routine undertaking for those of ordinary skill in the art having the benefit of this disclosure. The system and method for registration and tracking by 3D scanning the topology of anatomic structures disclosed herein boasts a variety of inventive features and components that warrant patent protection, both individually and in combination.VIS-004-W01

[0204] By way of example, this disclosure introduces a novel system and method for registering and tracking anatomical structures during surgical procedures by leveraging continuously updated 3D topological data of the surgical field. Unlike traditional systems that rely on physical optical markers (OMs) and Optical Tracker Constellations (OTCs), the system and method disclosed herein utilizes the data captured by a 3D scanner to perform Continuous Anatomic Auto Tracking (CAAT). Hereinafter, the system disclosed herein may be referred to as “CAAT system 10”. While shown and described herein within the specific context of a surgical application (specifically a TKA procedure), the CAAT system 10 including the 3D scanner 12 may be used in any scenario (surgical, medical, or otherwise) to generate a reconstructed 3D model of an object.

[0205] Referring to Fig. 3, and by way of example, the CAAT system 10 disclosed herein may feature a 3D scanner 12, hub 14, user interface 16, and distributed computation 18. By way of example, the CAAT system 10 also includes a software program 20 embodied in a non- transitory computer readable storage medium (e.g., the hub 14) and comprising a set of instructions configured to enable the computer to perform the various tasks described herein. In some embodiments, one or more peripheral computing devices 21 may be associated with the CAAT system 10, for example to integrate use and control of certain medical and surgical equipment that may be used in robotics-driven procedures. Such peripheral computing device(s) 21 may be configured to be controlled by the hub 14 and / or user interface 16.

[0206] In some embodiments, the 3D scanner 12 continuously captures topological data of one or more objects within a field of view at a high repetition rate, providing detailed surface information of anything within the field of view (e.g., anatomical structures, surgical equipment, surgical tools, surgical staff, etc.,) and ensuring uninterrupted tracking during dynamic surgical procedures. While the 3D scanner 12 disclosed herein is optimized for this application, the CAAT system 10 is compatible with any 3D scanner capable of optically generating topological information from the surgical site. In some embodiments, the 3D scanner 12 may be enclosed in a sterile drape to maintain sterility, allowing the 3D scanner 12 to function within the sterile surgical environment.VIS-004-W01

[0207] In some embodiments, the hub 14 is a dedicated computer or computer system responsible for receiving raw image data from the 3D scanner 12. Equipped with specialized hardware and software, the hub 14 processes this data in real-time, converting the data into a usable format for tracking anatomical structures and surgical instruments. In some embodiments, the CAAT system 10 performs automatic registration of the data to a Surgical Coordinate System (SCS), eliminating the need for manual optical trackers or registration steps. In some embodiments, the hub 14 may comprise multiple computer systems that communicate with one another so that data can be processed serially or in parallel on separate hardware.

[0208] In some embodiments, the hub 14 transmits transformation matrices for each tracked object to a separate or second computer system, where the data is further processed based on the surgical workflow. This information is presented to the surgeon in an intuitive, user-friendly interface 16, providing real-time visualization of anatomy and surgical tools. In some embodiments, the interface guides the surgeon through key actions such as implant positioning, bone cuts, and other critical tasks.

[0209] In some embodiments, while the hub 14 plays a central role in data processing, the CAAT system 10 architecture allows for flexibility in where computations occur. For example, in some embodiments, some processing tasks, such as initial data filtering or compression and / or reconstruction of a 3D point cloud, may be performed by a processor within the 3D scanner 12 itself to reduce data transmission loads. Additionally, certain computations may be handled by other computers within the system, either in series or parallel with the hub 14, depending on the complexity of the procedure and the system's overall configuration. In some cases, specific computational tasks, such as visualization rendering or more advanced data analysis, may be managed directly within the user interface (UI) 16 system. This distributed computation 20 approach enables scalable performance and ensures that real-time requirements are met across a range of surgical applications.

[0210] In some embodiments, the various methods disclosed herein are configured for use with a structured light scanning system such as the 3D scanner 12. It should be noted that the steps of each method may be performed by a computing device including at least a memory and a processor (which for example may be the hub 14 or peripheral computing device 21), andVIS-004-W01 according to a computer program product (e.g., software 20) embodied in a non-transitory storage medium and including a set of instructions that, when executed by the processor, cause the computing device to perform one or more of the method steps described herein.

[0211] 3D Scanner

[0212] Fig. 4 illustrates an example of a surgical environment 22 which may be a sterile field within an operating room at a hospital or surgical center, in which a patient 24 is positioned on an operating table 26 for a surgical procedure such as a Total Knee Arthroplasty (TKA). By way of example, a surgical instrument 28 (e.g., drill, oscillating saw, etc.) may be provided at the end of a first robotic arm 30, which itself is attached to a first cart 32. A 3D scanner 12 forming part of the CAAT system 10 is provided at the distal end (or “end effector”) of a second robotic arm 34 which is provided on a second cart 36. It should be noted that this surgical environment 22 is provided by way of example only to illustrate one possible setup and positioning of the 3D camera in relation to a patient during a surgical procedure. In some embodiments, the second robotic arm 34 to which the 3D scanner 12 is attached may be attached to a wall, ceiling, immobile table, or other structure.

[0213] Figs. 5-8 illustrate an example of a 3D scanner 12 configured for use with the CAAT system 10 of the present disclosure, according to some embodiments. In some embodiments, the 3D scanner 12 comprises an optical system 42, a light source assembly 44, a thermal management system (TMS) 46, a cover 48, and a rigid scaffolding 50. In some embodiments, the optical system 42 includes a camera, lens, and filter, and is configured for continuously capturing data including high repetition images of patterns projected onto the surgical field (from which topological data is derived and used in reconstructing a 3D point cloud) and communicating the captured data to the hub 14. In some embodiments, the light source assembly 44 is configured to project a structured light pattern onto the target surface, which the CAAT system 10 may then analyze as described below to generate a 3D model of the surface area with the optical system’s field of view, as described herein. In some embodiments, the TMS 46 is configured to remove heat from the 3D scanner 12. More specifically the TMS 46 removes heat generated by the light source and the optical system 42 by using airflow to transfer the heat through a heat sink and ultimately out of the 3D scanner 12 through ventilation tubesVIS-004-W01178, as described in detail below. The TMS 46 shown and described herein is representative of one example of removing heat from the 3D scanner 12. However, the 3D scanner 12 may be modified to incorporate other methods of removing heat in addition to or instead of the TMS 46 described herein.

[0214] In some embodiments, the optical system 42 has a field of view 68 with a central axis Li, and the light source assembly 44 has an illumination field 80 with a central axis L2. In some embodiments, the optical system 42 and light source assembly 44 may be positioned at an angle 0i relative to one another defined as the offset angle between the central axis Li of the optical system 42 and the central axis L2 of the light source assembly 44, as shown by way of example in Fig. 7. This arrangement produces an overlap area (or “working volume”) 69 in which the field of view 68 overlaps with the illumination field 80. Objects within the overlap area 69 may be scanned by the 3D scanner 12. Objects that are not in the overlap area 69 will not be scanned by the 3D scanner 12. In some embodiments, the angle 0i may also be defined as the apex angle in a triangulation between the optical system 42, the light source assembly 44, and the target site. By way of example, the apex location in the triangulation represents the optimal point within the overlap area 69 in which the optical system 42 produces the best imaging results. In some embodiments, the optical system 42 and light source assembly 44 may be positioned such that the angle 0i may be within a range of 10° to 30°. In some embodiments, the optical system 42 and light source assembly 44 may be positioned such that the angle 9i may be within a range of 10° to 20°. In some embodiments, the optical system 42 and light source assembly 44 may be positioned such that the angle 0i = 16.5°. In some embodiments, the 3D scanner 12 may be configured such that the angle 0i may be adjustable during use, for example to adjust the working volume 69 of the 3D scanner 12 or if the positioning of the 3D scanner 12 relative to the target site needs to be moved. In some embodiments, the 3D scanner 12 may be configured such that the focus is operable by a motor. In some embodiments, the 3D scanner 12 may be configured such that the focus is automatically adjustable. In some embodiments, the 3D scanner 12 may be configured such that the working volume 69 may be adjusted by changing the lateral distance between the optical system 42 and the light source assembly 44. In some embodiments, the 3D scanner 12 may be configured such that the focus may be adjusted by changing angle 0i between the optical system 42 and the light source assembly 44. In someVIS-004-W01 embodiments, the 3D scanner 12 may be configured to include an adjustable and / or motorized iris to enable a user to fine tune the level of light entering the camera.

[0215] Fig. 9 illustrates an example of an optical system 42 forming part of the CAAT system 10 of the present disclosure, according to some embodiments. In some embodiments, the optical system 42 may comprise a camera 60, lens 62, filter 64, and a data connection 66. In some embodiments, the camera 60 may be any compatible camera available in the market currently or in the future. In some embodiments, the camera 60 may have a silicon sensor configured to collect the light. In some embodiments, the sensor may be selected to have a high sensitivity to a specific range of wavelengths of light, including but not limited to (and by way of example only) a range of 750-850 nm, or 750-800 nm, or 800-850 nm. In some embodiments, the camera 60 has a field of view 68 defined as the area in front of the camera lens 60 and / or filter 64 that is captured by the camera. Points of interest within the field of view 68 may be captured by the camera 60 and transmitted as data to the hub 14. In some embodiments, the data may be sent directly to the hub 14 by way of the data connection 66. In some embodiments, the data may be sent indirectly to the hub 14 by way of an intermediate component, which may process the data before sending the data to the hub 14. Points of interest that are outside of the field of view 68 will not be captured by the camera 60. In some embodiments, the lens 62 extends from the camera 60 and may be configured to have an adjustable and / or motorized field of view range. Alternatively, the lens 62 may have a fixed field of view and changing the lens 62 would then change the field of view.

[0216] In some embodiments, the filter 64 may be a high pass wavelength filter or spectral filter. In some embodiments, the filter 64 may be selected and / or configured to block light wavelengths below a predetermined threshold value from passing through the filter 64. In some embodiments, the predetermined threshold value may be 800 nm such that wavelength values below 800 nm are blocked and wavelength values above 800 nm (e.g., including infrared light) are able to pass through the filter 64. In some embodiments, the predetermined threshold value may be 750 nm. In some embodiments, the filter 64 may comprise a band pass filter configured to block light wavelengths below a minimum predetermined threshold value and above a maximum predetermined value from passing through the filter 64 such that light having wavelength values within a specified range may pass through the filter 64. In someVIS-004-W01 embodiments, the specified range may be 800-850 nm. In some embodiments, the specified range may be 810-840 nm. By way of example, such specified ranges may block visible light (e.g., light wavelengths below 800 nm) and higher light wavelengths (e.g., light wavelengths above 850 nm) from passing through the filter 64. This helps to maximize the dynamic range of the sensor to capture the pattern projected. For example, in a surgical environment, a NIR wavelength range would not be visible to the surgeon or other staff and the ambient lighting within the operating room would be blocked by the filter, allowing only light reflecting off the target area to be allowed into the camera.

[0217] Fig. 10 illustrates an example of a light source assembly 44 forming part of the CAAT system 10 of the present disclosure, according to some embodiments. In some embodiments the light source assembly 44 includes a linear light source 70, a light shield 72, a light box 74, a transmissive LCD (or pattern generator) 76, and a pattern generator cover 78. In some embodiments, light generated by the light source 70 passes through the light shield 72, light box 74, and pattern generator 76 on its way to the surgical target site. In some embodiments, the light source assembly 44 has an illumination field 80 defined as the maximum possible area in front of the light source assembly 44 that may be illuminated by light emanating from the linear light source 70. In some embodiments, the linear light source 70 comprises an elongated emission component configured to emit light through the light shield 72 and light box 74. In some embodiments, the emitted light may have a wavelength value within a specified range. In some embodiments, the specified range may be 800-850 nm. In some embodiments, the specified range may be 810-840 nm. In some embodiments, the linear light source 70 comprises a plurality of LED emitters mounted to a chip-on-board (COB) printed circuit board (hereinafter “COB 82”) in a linear orientation. In some embodiments, the linear light source 70 may comprise one or more groups of LED emitters, with each of the one or more groups of LED emitters independently powered with voltage and current. In some embodiments, the linear light source 70 may comprise 10 groups of 21 LED emitters.

[0218] Figs. 11-13 illustrate an example of a pattern generator 76 forming part of the 3D scanner 12, according to some embodiments. In some embodiments, the pattern generator 76 comprises a transmissive LCD screen with individually controllable opacity comprising a liquid crystal (LC) layer 140 positioned between first and second polarizing layers 142, 144VIS-004-W01

[0219] In some embodiments, the first polarizing layer 142 ensures light entering the LC layer 140 is polarized. The LC layer’s twist (controlled by voltage) determines whether light’s polarization matches the second polarizer, and how much light is allowed through the LC layer 140. The second polarizing layer 144 acts as a gate that only passes light correctly rotated by the LC layer 140. This arrangement enables each energized line 150 within the LCD layer 140 to precisely modulate light intensity, forming a distinct and sharp pattern of edges. In some embodiments, the first and second polarizing layers 142, 144 may be near infrared (NIR) polarizers (e.g., which allow near infrared light through but block most visible light) instead of more common optically transmissive polarizers. In some embodiments polarizers may also allow for transmission of some visible light (though not as well as the usual polarizers), allowing for dual functionality in the event that the 3D scanner 12 is configured to have both NIR and visible lights present on a single unit.

[0220] Fig. 14 illustrates an example of a thermal management system (TMS) 46 forming part of the 3D scanner 12, according to some embodiments. By way of example, the TMS 46 is provided to remove heat from the major heat sources in the 3D scanner 12, namely the linear light source 70 and the camera 60 to prevent overheating of the light source 70 and the camera 60. In some embodiments, the TMS 46 may include a heat transfer assembly 170, a heat sink 172, an air box 174, a vent box 176, and one or more ventilation tubes 178. In some embodiments, heat is transferred from the linear light source 70 and the camera 60 to the heat sink 172 by the heat transfer assembly 170. Meanwhile, airflow is forcibly directed into one of the ventilation tubes 178 and / or forcibly pulled out of the other of the ventilation tubes 178 by fans that are at distal ends of the ventilation tubes which may be located outside of the sterile field (e.g., external of the sterile drape). In some embodiments, the forcible pushing and forcible pulling operate in concert so that the airflow is in one direction through the TMS 46. In some embodiments, only pulling fans may be used. In some embodiments, only pushing fans may be used. Heat that is pulled from the linear light source 70 and the camera 60 is directed into the heat sink 172, wherein it is transferred to the air that is directed through the heat sinkl72. This heated air is then directed out of the TMS 456 through a ventilation tube 178.

[0221] Scanning and Data Acquisition ProcessVIS-004-W01

[0222] As shown in Fig. 4, by way of example, the 3D scanner 12 may be positioned to continuously capture the surgical field. In some embodiments, as the surgeon manipulates instruments and interacts with the patient’s anatomy, the 3D scanner 12 generates a point cloud of the visible surfaces within the overlap area 69 (e.g., the surfaces that are within both the illumination field 80 of the light source assembly 44 and the field of view 68 of the optical system 42). In some embodiments, assuming data is collected at a rate exceeding 20 Hz, the data stream may be continuously updated in real-time. While faster rates are preferred to reduce system latency, rates above 20 Hz are generally sufficient for functional real-time performance. In some embodiments, the point cloud data is then used to create transforms that can be used to update the position and orientation of 3D reference geometry of the anatomy and any tracked instruments within the Surgical Coordinate System (SCS).

[0223] Fig. 15 is a block diagram depicting a data acquisition process 250 using a 3D scanner 12 described herein. While the 3D scanner 12 disclosed herein introduces several novel features, the general principles of structured light scanning are well established and are described here for clarity. Furthermore, this data acquisition process 250 may be used with other scanners, such as a shadow caster of the type disclosed in in the ‘973 app.

[0224] In some embodiments, a first step 252 in the data acquisition process 250 is to generate and project a structured light pattern onto a surface to be scanned. For example, in some embodiments, the 3D scanner 12 may illuminate the target surface with a predefined light pattern (e.g., commonly grids, stripes, or dots) projected onto the surfaces being scanned. By way of example, this may be accomplished by configuring the pattern generator 76 described above to produce the desired structured light pattern. In some embodiments, the 3D scanner 12 described herein may be configured to use infrared light, ensuring that the operating room lighting does not interfere with the scanning process. In some embodiments, a series of light patterns is projected to encode all the data necessary for reconstructing the 3D point cloud. Multiple pattern series can be used, depending on the specific requirements of data encoding. In some embodiments, variations in intensity, color, or phase may be applied to the pattern to enhance data accuracy or accelerate acquisition.VIS-004-W01

[0225] In some embodiments, a second step 254 in the data acquisition process 250 is image capture. For example, in some embodiments, cameras or sensors within the 3D scanner 12 (e.g., optical system 42 described above) capture the reflected structured light as it interacts with the target surface. In some embodiments, an image may be taken for each pattern projected in the series. By way of example, the captured images reflect deformations in the light pattern caused by the surface's contours and depth. In some embodiments, these images may be collected by one or multiple cameras, with the relative positions and orientations of the cameras being known to the CAAT system 10. In some embodiments, the first step 252 of generating and projecting structured light pattem(s) onto a surface to be scanned and the second step 254 of image capture may be repeated in an iterative loop until all the required images have been acquired.

[0226] In some embodiments, once all required images have been obtained by way of the previous steps, a third step 256 in the data acquisition process 250 is reconstruction of the 3D surface, for example within the hub 14. In some embodiments, reconstruction of the 3D surface may occur on board the 3D scanner 12 in a main control board rather than in a separate hub 14, in which case the following sub-steps also occur within the main control board of the 3D scanner 12. In some embodiments, the reconstruction step 256 includes sub-steps of pattern decoding, triangulation and 3D point cloud generation. In some embodiments, after image capture, the images are sent to the hub 14 for processing. In some embodiments, a first sub-step 258 of the reconstruction step 256 is pattern decoding. By way of example, data can be transferred from the 3D scanner 12 to the hub 14 continuously as images are captured, or in bulk once an entire pattern series is complete. In some embodiments, the reconstruction step 254 cannot be completed until all captured images are transferred to the hub 14, but some computation can be completed on each image as it arrives in the hub 14. In some embodiments, the CAAT system 10 is configured to decode the captured light patterns by analyzing their brightness history to create camera pixel correspondences to light angle projection. This, together with known pixel observation ray, enable via triangulation in-depth information, allowing the CAAT system 10 to reconstruct the 3D surface as a 3D point cloud.

[0227] In some embodiments, a second sub-step 260 of the reconstruction step 256 is triangulation. By way of example, the CAAT system 10 may use triangulation to calculate the 3D coordinates of each point where the light interacts with the surface. With the known positionsVIS-004-W01 of the linear light source 70, camera 60, and pattern generator 76, the CAAT system 10 determines a 3D point for each pixel in the captured images.

[0228] In some embodiments, a third sub-step 262 of the reconstruction step 256 is generation of a 3D point cloud 268 representing the scanned target surface. By way of example, the triangulated points from each frame may then be combined into a 3D point cloud 268, representing the surface in 3D space, with each point assigned x, y, and z coordinates. An example of a 3D point 268 cloud is shown in Fig. 16. In some embodiments, depending on system capabilities, additional data such as color or intensity may be included to enhance detail and visual fidelity.

[0229] In some embodiments, a fourth step 264 in the data acquisition process 250 is data processing and filtering. For example, in some embodiments, the raw point cloud data may undergo post-processing to reduce noise, fill gaps, and improve accuracy. In some embodiments, if multiple scans are taken from different angles, the data may be aligned, merged, or "stitched" together to create a cohesive 3D model.

[0230] In some embodiments, a fifth step 266 in the data acquisition process 250 is to output the data such that the final point cloud 268 is made available for further processes, such as segmentation and registration, which can be used to guide the surgical workflow.

[0231] Differences between optical tracker generated data and 3D scanned topology data

[0232] It is important to distinguish between the data generated by Optical Markers (OMs) and Optical Trackers (OTs) (e.g., as used in prior art system) versus the data produced through 3D scanning of a surgical site, as disclosed herein.

[0233] By way of example, in Figs. 17A-B, the image on the left (Fig. 17A) depicts a typical TKA surgery, showing externally applied optical trackers 270 attached to the robot 272, femur 274, and tibia 276. Each externally applied optical tracker 270 comprises four optical markers 278 in a predetermined arrangement. This image represents the typical field of view for stereoscopic systems. On the right (Fig. 17B), the image shows what one of the stereoscopic cameras captures after segmenting the image to isolate the OMs 278 (which by way of example appear as black markers on a white background). Clearly visible are three OTs 270, eachVIS-004-W01 consisting of four OMs 278. When two such images, taken from slightly different angles, are combined via triangulation, the OTs 270 can be located in space, inferring the positions of the bones and robotic instruments registered to their respective trackers.

[0234] In contrast, the data collected by a 3D topological scanner such as 3D scanner 12 presents a very different picture. By way of example, Figs. 18A-B illustrates images from the 3D scanner 12. By way of example, the image on the left (Fig. 18A) is a scanner image 280 captured by the 3D scanner 12 using infrared light, and the image on the right (Fig. 18B is a 3D point cloud 268 generated from the whole pattern series of collected image data.

[0235] For example, one key difference is the field of view. For a stereoscopic system to capture all the necessary OTs 270 during surgery, the field of view must be quite large, often requiring the cameras to be positioned several meters away, typically at an angle from the side and slightly above the surgical area. From this vantage point, the bones are often not visible to the naked eye, as they may be obstructed by surgeons, instruments, other tissues, etc. By contrast, the 3D scanner 12 provides a much narrower field of view, captured from a closer, top- down perspective, for example as shown in Fig. 4.

[0236] Another key difference is that the data density is markedly different between the two systems. By collecting points one by one, the stereoscopic system provides sparse data, providing around 50 possible data points on the surface of a femur, for example. This must be collected manually by using a pointer to touch a point and then a foot pedal may be activated to capture the point. This is time consuming and can be error prone if the pointer is moving. Also, on the articular surface of the femur and tibia, if the surgeon is trying to register to preoperative CT bone, then the pointer must be sharp and penetrate the cartilage fully without penetrating the bone. Conversely, the 3D scanner 12 of the present disclosure captures a much more robust data set without penetrating patient anatomy. For example, the image shown in Fig. 18B has approximately 4 megapixels, with about 300,000 individual points on the femur and 200,000 on the tibia. Additional points correspond to surrounding soft tissues. In some embodiments, a depth fdter may be applied to the data to remove much of the background (such as the blue surgical drape).VIS-004-W01

[0237] By way of example, to further illustrate the 3D nature of the data, Fig. 19 presents the scan data (e.g., point cloud 268) from an oblique angle, giving a clearer sense of its depth. By way of example, for users familiar with knee anatomy, identifying key anatomical structures is relatively straightforward. However, for clarity, Fig. 20 highlights several major anatomical landmarks, by way of example, including the distal end of the femur 274, the anterior cruciate ligament (ACL) 282, the proximal end of the tibia 276, the patellar tendon 284, and a side view of the patella 286. Other points correspond to skin, muscle, fascia, and fat, though they are not specifically labeled. Note that these images use simulated tissue models commonly used for research, not real tissue.

[0238] By way of example, in real tissue, the visual characteristics will vary due to differences in how materials interact with light. These images are representative, but real tissue tend to introduce more noise, particularly in tissues that are partially transparent to infrared light. In some embodiments, this increased noise can be mitigated by applying more aggressive filtering algorithms, for example. In some embodiments, to help define a more reliable shape at one point in a procedure, a wavelength may be used that has a shorter optical penetration length. That wavelength may overlap with other sources of light in the operating room but may accommodated for the sake of an initial accurate view for reference with subsequent scans taken at wavelengths that introduce more noise, but that do not compete with other sources of light in the operating room. In addition, it may also be a relatively narrow-band source (e.g. 5 to 10 nm bandwidth) that has a relative brightness at that wavelength much greater than that of the room light, such that spectrally filtering for that wavelength results in high-contrast scans with minimal competition with the ambient light.

[0239] Registration Using Topological Data

[0240] Once topological data is available in the form of a 3D point cloud 268, a wide range of capabilities become possible. In some embodiments, multiple 3D point clouds may be aligned and / or combined within a single coordinate space for a more accurate and efficient surgery. By way of example, this aligning and / or combining may occur by way of a process called “registration”, which aligns two or more 3D point cloud datasets, combining them into a common coordinate space. This registration process allows different datasets — such asVIS-004-W01 preoperative CT scans and intraoperative 3D scans — to be aligned and even combined, enabling more precise surgical planning and execution. By way of example, Fig. 21 illustrates an example registration process 290 that may be performed by the CAAT system 10. In some embodiments, the registration process 290 comprises several steps including but not necessarily limited to preprocessing the data for accuracy 292, segmentation 294, registration of anatomical structures 296, registration of instruments and equipment 298, and making precise measurements 300. Although presented as a block diagram, many of these “steps” may be performed in any order, in series or parallel, and / or independently of one another. For example, anatomic registration and instrument registration may occur in any order, independent of one another, or one may occur without the other occurring.

[0241] In some embodiments, a first step 292 in the registration process 290 using topological point cloud data is pre-processing the raw point data cloud to ensure accuracy and reliability. In some embodiments, this pre-processing step 292 may include several sub-steps including noise reduction 301, smoothing 302, upsampling 303 the 3D point cloud, and / or downsampling 304 the 3D point cloud. By way of example, noise in the data can be introduced from various sources, such as light interference or reflections from surgical instruments. In some embodiments, noise reduction techniques 302 like statistical outlier removal and radius-based filtering (for example) are used to clean the dataset, ensuring that only relevant points remain. In some embodiments, smoothing algorithms 303, such as Gaussian filtering or bilateral filtering, may be applied to create a more continuous and realistic surface representation. For example, this step may help to reduce local variations in point locations that are not significant for registration, ensuring a more refined dataset without losing critical anatomical details. In some embodiments, the 3D point cloud may optionally be upsampled 304 by increasing the data points in the point cloud. In some embodiments, the 3D point cloud may optionally be downsampled 304 by decreasing the data points in the point cloud. In some embodiments, further data processing techniques such as outlier removal 305, normal estimation 306, and / or feature extraction 307 may be used.

[0242] In some embodiments, a second step 294 in a registration process 290 using topological point cloud data is segmentation. Segmentation enables the CAAT system 10 to handle multiple objects simultaneously during registration while reducing the computationalVIS-004-W01 load. By way of example only, once the point cloud data has been pre-processed, the segmentation step 294 can be applied to isolate different anatomical structures or surgical tools. Segmentation is an optional step but provides significant benefits for both data handling and computational efficiency. One such benefit is that segmentation allows for the identification of different objects within the dataset. For example, all points belonging to the femur can be isolated from the main dataset, while points belonging to the tibia can be similarly separated. Any remaining data can be segmented or discarded if unnecessary. Another benefit of segmentation is related to speed and latency. For example, registering large point clouds can be computationally intensive. By segmenting and reducing the dataset to only the objects of interest, computational requirements are lowered, improving processing speed and reducing overall system latency. In some embodiments, the segmentation step 294 may occur before the pre-processing step 292 to perform segmentation using raw, unfiltered data.

[0243] In some embodiments, various methods of segmentation 294 may be used, including but not limited to manual segmentation 308, traditional algorithms 309, and artificial intelligence (Al) 310 powered segmentation. By way of example, manual segmentation 308 may be useful in complex cases, as operators can manually label objects in the point cloud to isolate key structures. In some embodiments, traditional algorithmic techniques 309 such as edge detection or region-growing may be used to identify structures based on their shape or intensity. In some embodiments, Al-powered segmentation 310 may be used so that pre-trained deep learning models can automatically segment point cloud data in real-time, offering high accuracy and minimal manual intervention. In some embodiments, the Al-powered segmentation 310 may operate within a 2D image space (e.g., a 2D image or 2D depth-map representation) or a 3D space (e.g., point cloud space).

[0244] In some embodiments, a third step 296 in a registration process 290 using topological data is anatomic registration. In most cases, registration involves aligning two datasets, such as a preoperative CT dataset and a 3D topological scan from the surgical site. By way of example only, the alignment may be performed using algorithms such as RANSAC (Random Sample Consensus) 312 for initial, coarse registration and ICP (Iterative Closest Point) 314 for finetuning. These algorithms compare corresponding points between datasets and compute a transformation matrix that minimizes alignment errors. For example, RANSAC 312 selects keyVIS-004-W01 features in the point cloud, filtering out outliers, and computes an initial transfor ation matrix by focusing on valid, inlier points, and ICP 314 iteratively refines the resulting transformation matrix by minimizing the distance between corresponding points in the two datasets, leading to precise alignment. The resulting transformation is a 4x4 rigid matrix that translates, rotates, and scales one dataset to align with the other, for example similar to the transformation matrix shown in Fig. 1. In the context of a TKA procedure, for instance, this could align a CT scan with a realtime topological scan of the femur 274 and tibia 276.

[0245] In some embodiments, the anatomic registration step 296 can be repeated for multiple objects, such as the femur 274 and tibia 276, allowing simultaneous registration of more than one object assuming the computational horsepower is available. Fig. 22 illustrates the registration of CT data from the femur 274 and tibia 276, along with a 3D topological scan or point cloud 268, into a common coordinate space 316. By using registered datasets, the CAAT system 10 provides surgeons with accurate measurements that aid in making critical decisions during surgery. By way of example only, Fig. 22 shows the registration of three datasets — CT data from the femur and tibia, and a 3D topological scan — into a common coordinate space for this purpose.

[0246] The registration step 296 is a critical step in aligning two 3D point cloud datasets, such as preoperative CT data with a real-time topological scan in a surgical setting. This method achieves the same results as registration done with Optical Markers (OMs) 278 and Optical Trackers (OTs) 270, but without the need for external tools anchored to bones or an additional transformation to account for the indirect determination of a transformation matrix via the trackers. Additionally, the topological dataset is far denser and more detailed than what a surgeon could collect using a pointer with OT.

[0247] In some embodiments, a fourth step 298 in a registration process 290 using topological data is instrument and equipment registration. By way of example, this step is optional and may occur before, during, or independently of anatomic registration in the previous step 296. In addition to anatomical structures, instruments and surgical equipment can be registered using topographical data. By way of example, each instrument has a unique shape that can be used for registration. For instance, a bone saw has a distinct geometry compared to a scalpel, makingVIS-004-W01 identification possible in the topological data. In some embodiments, the same registration methods used for anatomy can be applied to instruments, but some challenges exist. For example, instruments with reflective surfaces may not be well-captured in the 3D scan. In some embodiments, treating surfaces to scatter light can reduce specular reflections and improve scan quality. Another challenge is that some materials absorb light, causing poor visibility in the scan. As with specular reflections, in some embodiments treating surfaces to scatter light can mitigate this issue. By way of example, another challenge with instrument and equipment registration is occlusions, in that instruments and / or equipment may be partially obscured by the surgeon's hand. In some embodiments, by extending the handle or otherwise introducing distinct geometry, the CAAT system 10 can still register the instrument even when partially blocked. In some embodiments, distinct geometry may include (but is not limited to) portions of the tracker that are not optical markers, patterns printed on the instruments (e.g., particularly on their most distal end) and / or attachment of a fiducial to the instrument enabling identification and understanding of the instrument position and / or orientation at the surgical site itself.

[0248] In cases where an anatomical point is not directly visible (e.g., the center of the ankle), a pointer instrument can be used. In some embodiments, for example, registering an end of the pointer to the scanned data, the anatomical point can be inferred accurately. In some embodiments, this could be done by registering a proximal end (e.g., that would extend into camera view) of the pointer in the common coordinate system and, based at least in part on the known attributes or the pointer, the relative positioning of the distal tip (e.g., that would be positioned near the target out-of-view anatomical point) would be known or inferred. In this way, moving the scanner in order to change its field of view to see anatomy at the ankle (for example) may be avoided.

[0249] In some embodiments, a fifth step 300 in a registration process 290 using topological data is measurements. For example, once the objects are registered within the same coordinate space, precise measurements can be made, providing crucial information during surgery. These measurements may include, but are not limited to distance measurements, angle measurements, and / or gap measurements. By way of example, measurements made after registrations are relative measurements between two registered objects within the coordinate space or a single registered object and the scanner reference frame in the coordinate space. In some embodiments,VIS-004-W01 the CAAT system 10 can measure the distance between two points on anatomical structures or instruments. For example, measuring the width of the knee’s sulcus is as simple as selecting two points (either manually, or with the points identified by Al or by registration with a model, or with CT), with the system computing the distance in real-time. In some embodiments, more complex measurements, like assessing the varus or valgus angle of the knee, can be performed by identifying the sagittal planes of the femur and tibia in the registered datasets. Once identified, the system calculates the angle between the planes, providing vital information for implant alignment. In some embodiments, if two objects are registered, measurements between the two objects can be computed. For example, in TKA surgeries, the medial and lateral gaps between the femoral condyles and tibial plateau can be measured to assess ligament balance. Surgeons often aim to balance these gaps to ensure even tension in the knee, which is critical for the success of the implant.

[0250] Also, by measuring known anatomic features of the knee, careful fitting of a parametric model of the knee may be possible directly for implant selection or even real-time manufacture.

[0251] Tracking with Topological Data

[0252] By way of example, tracking and registration are closely related, but they serve distinct purposes in a surgical setting. As previously described, registration is the process of aligning two or more 3D point cloud datasets, bringing them into a single coordinate system. This is typically a one-time event or a relatively infrequent operation. In contrast, tracking involves continuously registering these datasets over time, monitoring changes in position or orientation, and keeping objects aligned within the same coordinate system. Tracking updates the spatial alignment of registered objects multiple times per second, which allows for dynamic monitoring of anatomical structures and surgical tools as they move throughout the procedure.

[0253] Real-time tracking is essentially registration performed in a continuous loop, where the system rapidly processes new data to reflect movements or changes as they occur. The faster this loop is executed, the more accurately the system can represent the current state of the surgical environment. In some embodiments, and by way of example only, registration uses a dense point cloud, while tracking may use “sparse” scans, which are not the highest pixel densityVIS-004-W01 scans, but are much faster to take. The goal of real-time tracking is to create a seamless experience where movements of anatomical structures, instruments, or other objects are captured and reflected without noticeable delay.

[0254] Defining what constitutes "real-time" is challenging because it varies depending on the context and application. In the world of cinema, for example, a standard film is displayed at 24 frames per second (fps). At this rate, the human brain perceives the sequence of still images as continuous motion, which most viewers consider to be real-time. Computer monitors typically operate at higher refresh rates, such as 60Hz or 120Hz, offering smoother visuals. Gaming monitors push this further, with refresh rates of up to 240Hz, though this is generally only advantageous in specific, fast-paced applications.

[0255] From a physiological standpoint, studies suggest that the human brain begins to interpolate between frames at speeds around 24 fps, making discrete images appear as a fluid, continuous motion. For the purposes of this disclosure, we define real-time tracking as performing at least 20 registrations per second (rps). This threshold provides a sufficiently smooth experience, making the data appear continuous to the surgeon. Faster registration speeds are always preferable, as they can further reduce latency, but they come with significant tradeoffs in terms of the system’s requirements. Higher speeds increase the demands on light intensity, the speed of image sensors, and computational power to process the data quickly enough to maintain real-time performance.

[0256] In some embodiments, speed may be increased by bypassing the coarse or global registration step (e.g., RANSAC as described above) during tracking, and instead loop through the fine registration (e.g., ICP) step. This is because typically the tracked object does not move very much or very far in the short time between scan registrations, it is already close enough for successful ICP operation. In some embodiments, the exclusion of a coarse registration step may provide a significant increase in tracking speed. In some embodiments, tracking may be lost during ICP-only operation (e.g., the tracked object moves faster or farther than ICP can successfully register). In such cases, by way of example only, the system 10 may revert to a global registration step which may be RANSAC as described above, or in some embodiments anVIS-004-W01Al-driven analysis of captured images (e.g., from the 3D scanner 12) may be used to quickly determine the location of the tracked object.

[0257] In some embodiments, the CAAT system 10 may increase tracking speed by using a previously registered object position to constrain the considered data points of where the object may move to next for fine registration. In some embodiments, this would reduce the points for consideration during fine registration (e.g., ICP), and fewer points to analyze generally leads to increased processing speed. By way of example, in some embodiments, this data may be constrained based on velocity and / or acceleration. In some embodiments, an object can’t go faster than a maximum determined velocity in any direction, which limits or constrains the potential data points needed for fine registration. In some embodiments, the data points could be constrained based on likely acceleration of the object. In some embodiments, object velocity may be used to extrapolate an initial guess location for the fine registration.

[0258] In some embodiments, the CAAT system 10 may constrain the considered data points of where the object may move to next for fine registration by limiting points of consideration in the acquired topological scan(s) that are a certain distance away from the previously registered object points. In some embodiments, this may include points located within an axis-aligned bounding box in every direction by an amount that is limited by practical physics, for example a set amount (e.g., a guess without first measuring velocity). In some embodiments, the CAAT system 10 may be configured to select a direction for the bounding box growth based on its previous velocity.

[0259] In some embodiments, the CAAT system 10 may constrain the considered data points of where the object may move to next for fine registration by considering only points within a radius from each of the relevant points (and optionally in a particular direction).

[0260] Single object tracking (Hip Center Example)

[0261] The simplest case of tracking is the real-time monitoring of a single object, where its position is continuously updated in the Surgical Coordinate System (SCS). This type of tracking is critical in various surgical procedures where accurate positioning of key anatomical landmarksVIS-004-W01 or surgical tools is required. In this example, the scanner is kept stationary, or its motion is itself tracked relative to the operating room via some other navigation device.

[0262] A practical example of single object tracking in a TKA procedure is the calculation of the hip center of rotation. In some TKA methodologies, such as the Mechanical Alignment approach, determining the precise location of the hip center is essential for establishing the mechanical axis of the leg, which guides the alignment of implants. By way of example, Fig. 23 presents a block diagram depicting steps of a method 320 of determining the hip center of rotation using the CAAT system 10 described herein. In some embodiments, the first eight steps of the method may be repeated in a (rapidly occurring) loop until sufficient data has been accumulated to accurately determine and report a hip center of rotation and femur length. By way of example, a first step 322 in the method 320 of determining the hip center of rotation is using the 3D scanner 12 described herein in tracking mode to acquire topological scan data of the femur 274. In some embodiments, a second step 324 of the method 320 of determining the hip center of rotation is registering the acquired scan data to a 3D model of the femur 274, which for example may be derived from a pre-operative CT scan. In some embodiments, a third step 326 in the method 320 of determining the hip center of rotation is producing a transformation matrix for each time point. This transformation matrix 2 maps the position of the femur into camera space, allowing the system to track the femur's movements in real-time as the surgeon manipulates the leg.

[0263] In some embodiments, a fourth step 328 of the method 320 of determining the hip center of rotation is to use the generated transformation matrix to compute the location of achosen anatomic point for determining the length of the femur in camera space. A fifth step 330 of the method 320 of determining the hip center of rotation is to send coordinates of the tracked chosen anatomic point to the hub 14 (for example) for sphere fitting.

[0264] As previously mentioned, the first eight steps of the method may be repeated in a (rapidly occurring) loop until sufficient data has been accumulated to accurately determine and report a hip center of rotation and femur length. The first five steps identified above describe the CAAT system 10 producing a data point comprising calculated (x, y, z) coordinates for the femur origin location with each scan, which may be happening at least 20 times per second.VIS-004-W01Thus, a sixth step 332 of the method 320 of determining the hip center of rotation is to add a newly determined coordinates data point to the dataset containing the previously calculated coordinates. In some embodiments, a seventh step 334 of the method 320 of determining the hip center of rotation is to calculate the size and location of a sphere to which the accumulated data points were best fit, for example using least squares fitting to find the best-fitting sphere, which minimizes the difference between the points and the surface of the sphere. The center of this sphere represents the hip center of rotation. In some embodiments, an eighth step 336 of the method 320 of determining the hip center of rotation is to compare residual errors of the fit of the sphere to the predetermined acceptance criteria to assess whether the generated best fit sphere is a good fit (e.g., decision box 336). If the sphere is not a good fit, then the workflow reverts to the first step 322 of acquiring topological scan data of the femur 274. In some embodiments, if the generated best fit sphere is determined to be a good fit, then a final step 338 in the method 320 of determining the hip center of rotation is to report the hip center and femur origin length (e.g., based upon the position of the sphere origin and the radius of femur).

[0265] While some noise is inevitably introduced due to factors such as movement artifacts in the pattern images or reconstructed object, or slight variations in scan accuracy, the CAAT system 10 can compensate for these by averaging or applying noise-reduction techniques. Once the hip center is identified in camera space, it can easily be transformed into any other coordinate system (e.g., SCS), ensuring that this critical anatomical point is available for accurate surgical planning and alignment, including but not limited to intraoperative real-time modification of the surgical plan.

[0266] This example highlights the value of single object tracking in real-time during surgery, where critical anatomical landmarks must be continuously monitored and accurately calculated to guide the procedure.

[0267] Two Object Tracking and Measuring (Gap Balancing Example)

[0268] A more complex use case for tracking occurs when two or more objects are tracked simultaneously. While a previous section described how distances can be calculated between two objects through registration, this concept can be extended into a continuous tracking mode, where multiple registrations are performed over time to monitor the relative positions of multipleVIS-004-W01 objects. This is especially valuable in real-time scenarios, where dynamic forces are applied, and accurate measurements must be taken as the objects move in relation to each other.

[0269] An example of this in a Total Knee Arthroplasty (TKA) procedure is when the surgeon uses a gap balancing methodology to assess the gaps between the medial and lateral condyles of the femur and the tibial plateau. By way of example, Fig. 24 is a flowchart depicting an example method 350 of assessing gaps between two objects, for example femoral condyles (e g., medial and lateral) and a tibial plateau. In this method 350, the goal is to ensure that the gaps between the femur and tibia are balanced when the knee is in both extension and flexion, which helps maintain proper knee alignment and functionality after the procedure.

[0270] Measuring these gaps presents a challenge because the surgeon does not perform this measurement passively. Instead, they manually apply forces to the knee in both a varus (inward) and valgus (outward) direction. This creates a dynamic scenario where the knee joint, the entire leg, and the surgeon’s hands are moving as the force is applied. The critical challenge is that the surgeon is not looking for a single static gap value but rather the peak gap under maximum load. Because significant force is applied during this process, the positions of both the femur and tibia are constantly shifting, and these movements happen quickly. The gap measurement needs to be recorded at high speed, tracking the movement of both the femur and tibia in real time to capture the moments where the force is greatest, and the gap is widest.

[0271] In some embodiments, a first step 352 in the method 350 of assessing gaps between femoral condyles and a tibial plateau is to acquire topological scan data of the femur and tibia and generate reconstructed 3D point clouds of each, for example using techniques described above. In some embodiments, a second step 354 of the method 350 of assessing gaps between femoral condyles and a tibial plateau is to independently register the femur and tibia in the CAAT system 10 as described above, for example using the acquired and reconstructed scan data in addition to reference data 356 of the femur and tibia, for example from preoperative imaging (e.g., CT imaging) or data collected intraoperatively. In some embodiments, a third step 358 in the method of 350 assessing gaps between femoral condyles and a tibial plateau is to use transformation matrices to convert reference data to the surgical coordinate system or to another virtual space. In some embodiments, a fourth step in the method 350 of assessing gaps betweenVIS-004-W01 femoral condyles and a tibial plateau is to calculate the distances between key points on the femur and tibia in real time. As the surgeon applies varus and valgus forces, the system tracks the resulting movements and dynamically measures the gaps between the medial and lateral condyles and the tibial plateau. In some embodiments, this includes performing a first sub step 360 of computing the medial gap (e.g., the minimum distance between the femur and tibia on the medial side) and a second sub step 362 of computing the lateral gap (e.g., the minimum distance between the femur and tibia on the lateral side). In some embodiments, a fifth step 364 in the method 350 of assessing gaps between femoral condyles and a tibial plateau is to consolidate the data and present it to the surgeon in a usable format. In some embodiments, the next step 366 is to determine whether there is enough data to proceed with the procedure. If the answer is no, then the workflow reverts to the first step 352 of acquiring and reconstructing topological scans of the femur and tibia, and the workflow continues to cycle through the steps until enough data has been obtained to assess the gaps between femoral condyles and a tibial plateau, at which point the tracking may end (box 368).

[0272] In some embodiments, by performing multiple registrations rapidly (at a rate defined as real-time, e.g., 20+ registrations per second), the CAAT system 10 can record the gap values continuously as the knee moves under load. In some embodiments, recorded measurements may be analyzed to determine the peak gap value at the moment when the maximum force is applied in each direction. Because these forces are only applied for a brief period, rapid and continuous tracking is essential to ensure the correct measurements are captured at the precise moment.

[0273] There are several benefits to real-time gap measurement. For example, the ability to track the femur and tibia in real time allows for dynamic gap measurements under varying loads, providing more accurate and clinically relevant data than static measurements. Additionally, real-time tracking ensures that the CAAT system 10 captures the peak gap values, which are critical in determining the balance between the medial and lateral sides of the knee.Furthermore, since the leg and knee joint are moving throughout the measurement process, continuous tracking allows the system to compensate for these movements and still produce accurate results.

[0274] Tracking Anatomy with a Robot-Assisted Surgical InstrumentVIS-004-W01

[0275] A more complex use case for tracking arises when a surgical instrument is attached as the end effector of a robot. In this scenario, the position and orientation of the instrument must be continuously updated to accommodate any anatomy movement. An example of this in TKA is when an oscillating saw is attached to a robotic system, and the robot assists the surgeon by positioning the saw in the correct plane for a planned osteotomy. By way of example, Fig. 4 illustrates an operating environment 22 in which a surgical instrument 28 is positioned on the end of a robotic arm 30.

[0276] By way of example, the robot’s function is to prevent the saw blade from drifting outside of the predefined cut plane, maintaining specific translational and angular tolerances. This process is straightforward if the knee is rigidly fixed. However, in practice, the knee may move slightly due to various factors such as patient movement or surgeon manipulation. As the knee moves, the robot must continuously adjust to follow the anatomy and ensure the osteotomy remains accurate. Traditionally, this can be achieved using Optical Markers (OMs) and Optical Trackers (OTs), but it can also be accomplished using real-time topological data collected from the 3D scanner 12 without the use of OMs and / or OTs. For example, Fig. 25 is a block diagram illustrating an example method 370 of tracking movable anatomy and a movable surgical instrument configured to affect the movable anatomy, according to some embodiments of the disclosure. In some embodiments, this method 370 is essentially an extension of the multi-object tracking described previously, but with the additional complexity of having to communicate the anatomy’s movement to the robot so that the position of the end effector can be updated in real time.

[0277] In this example scenario, multiple objects are tracked simultaneously. Thus, in some embodiments, a first step 372 in the method 370 is to acquire topological scan data of the target anatomy (e.g., a femur) and the surgical instrument 28 and generate reconstructed 3D point clouds of each, for example using techniques described above. In some embodiments, a second step 374 of the method 370 of tracking movable anatomy and a movable surgical instrument configured to affect the movable anatomy is to independently register the target anatomy and the surgical instrument 28 on the end of the robot arm 30 in the CAAT system 10 as described above, for example using the acquired and reconstructed scan data in addition to reference data 366 of the target anatomy and surgical instrument. In some embodiments, the reference data 366VIS-004-W01 of the target anatomy may be acquired from preoperative imaging (e.g., CT imaging) or data collected intraoperatively. In some embodiments, the reference data 366 of the surgical instrument 28 may be acquired from preoperative CAD data, preoperative scans, intraoperative scans, and / or a fiducial marker if the instrument does not scan well. In some embodiments, the origin point (or other point of interest) of the instrument 28 is identified in the data set or calibrated at the time of surgery.

[0278] In some embodiments, a third step 378 in the method 370 is to use transformation matrices to convert the target anatomy position and orientation into robot space. By way of example, the robot's base remains fixed, and a point on the base is defined as the origin of the surgical coordinate system (or “SCS”), which in this example may also be referred to as "robot space." In some embodiments, all other objects (3D scanner 12, instrument (e.g., saw) 28, femur 274, etc.) may be transformed into the robot space to ensure alignment. By way of example, by tracking the femur, its position can be continuously transformed into the robot space. In some embodiments, an optional step 380 of the method 370 may be performed to smooth the input data to reduce the effects of noise before the data is sent to the robot, for example using a moving average and / or Kalman filter.

[0279] In some embodiments, a fourth step 382 in the method 370 is to send the target anatomy position and orientation to the robot. By way of example, the instrument 28 (e.g., saw blade) can be tracked either directly using the 3D scanner 12 as disclosed herein or by relying on data from the robot's internal control system, which already knows the position of its end effector (and hence the saw blade) in robot space.

[0280] In some embodiments, a practical result of the method 370 is that the position of the target anatomy is continuously tracked as it moves during surgery, for example using the CAAT system 10 and the 3D scanner 12. In some embodiments, the updated position of the target anatomy is continuously communicated to the robot 30, which as a fifth step 384 in the method 370, may intraoperatively plan the optimal kinematic motion path to maintain relative position to the target. Stated differently, the robot may adjust the instrument 28 (e.g., saw blade) movement in real time to maintain alignment with the target anatomy and the predefined cut plane. By wayVIS-004-W01 of example, the robot effectively compensates for any slight anatomical shifts, ensuring the accuracy of the osteotomy.

[0281] This real-time adjustment can be performed within predefined translational and angular tolerances, providing the surgeon with a more precise and reliable way of cutting within a moving surgical field. Depending on the design of the robot and its positional control system, it may be necessary to update the registration data at a rate faster than 20 registrations per second (rps) to achieve a finely tuned control system.

[0282] In some embodiments, a sixth step 386 in the method 370 is to execute the motion path of the instrument with minimal latency between the time the 3D topological data is captured to when an updated transformation matrix is sent to the robot. In such a use case, latency becomes critical. Any delay in updating the robot’s position relative to the anatomy can lead to the saw blade falling out of alignment with the femur, reducing the accuracy of the cut. Minimizing latency between the time the 3D topological data is captured and when the updated transformation matrix 2 is sent to the robot is essential for maintaining real-time performance. For instance, if the system operates at 30 registrations per second (rps) and the latency is 50 milliseconds (ms), then the robot will get data on its new position 50 ms after the bone being tracked was there. Additionally, there may be additional latency introduced by the robot’s control system (e.g., the robot needs to calculate an optimal path, engage motors, accelerate, and finally reach the intended point). If, for example, it takes the robot 20 ms to perform these tasks, the total lag would be 70 ms. Thus, it is essential to reduce both sources of lag in a real-time tracking system to ensure that the robot can respond in real time to anatomical movements.

[0283] In some embodiments, a seventh step 388 in the method 370 is to determine whether tracking of the target anatomy and surgical instrument needs to continue. If the determination is affirmative, then the workflow cycles back to step 372 of acquiring and reconstructing topological scan data of the target anatomy. If the answer is no (e.g., the osteotomy has been completed), then the system may end the tracking (e.g., box 389).

[0284] By increasing the registration rate and reducing computational latency, the system can deliver accurate, real-time adjustments to the robot, ensuring that the saw blade remains in theVIS-004-W01 correct plane even as the target anatomy moves. This approach maintains the precision of the osteotomy and helps prevent unintended deviations, improving surgical outcomes.

[0285] Anatomy Tracking and Robot Following with a Mounted Scanner[00286J In previous examples, the 3D scanner 12 was positioned at a fixed location. However, mounting the scanner on the end effector of a robotic arm 34 (e.g., as shown by way of example in Fig. 4) allows the 3D scanner’s 12 position and field of view 68 to change dynamically as the robot 34 moves. This configuration offers significant advantages during surgical procedures where the anatomy or surgical site frequently shifts.

[0287] For instance, in a Total Knee Arthroplasty (TKA) procedure, the knee is often moved by the surgeon, transitioning from full extension to 90 degrees of flexion, and sometimes into full flexion. While a fixed 3D scanner 12 can capture the surgical site in one position, maintaining a clear view becomes challenging as the anatomy moves. If the knee or other anatomical structures shift out of the 3D scanner 12’ s field of view 68, manual repositioning would be required, interrupting the workflow.

[0288] By mounting the 3D scanner 12 on the robot’s end effector, the system can continuously track bones like the femur or tibia and automatically adjust the scanner’s position to maintain a fixed perspective on the surgical site. As the knee moves through different angles, the robot dynamically updates the scanner's position, keeping the anatomy within the field of view. This automation reduces the need for manual intervention, saving time and effort. In some embodiments, the robot may be provided with one or more degrees of freedom to enable movement of the 3D scanner 12. By way of example, the robot shown in Fig. 4 includes seven (7) degrees of freedom, however robots with fewer degrees of freedom (e.g., 1, 2, 3, etc.) or more degrees of freedom (e g., 8, 9, 10, etc.) may be used as may be needed or available.

[0289] In addition to maintaining a fixed view, the robot-mounted scanner synchronizes its movements with the anatomy in real-time. This ensures optimal scanner positioning throughout the procedure, allowing the surgeon to focus on the operation without worrying about viewpoint adjustments. Multiple viewpoints can be pre-programmed or customized for different stages ofVIS-004-W01 the surgery, with the surgeon easily switching between them via the user interface (UI), further enhancing procedural efficiency.

[0290] Complex Anatomy Tracking with Robot-Controlled Instrument (Oscillating Saw Example)

[0291] In this use case, both a 3D scanner 12 and an oscillating saw 28 are mounted to a robot arm. By way of example, the 3D scanner 12 tracks the anatomy while the robot positions the saw 28 to make precise cuts along a predefined cut plane, which may be modified intraoperatively based on the surgeon’s real-time observations. In some embodiments, if the 3D scanner 12 position changes as the robot moves, it may be necessary to apply one or more transformations to ensure that the anatomical data, 3D scanner 12, robot base, and saw 28 remain aligned in the same coordinate space. In some embodiments, the common coordinate system may be referenced to the patient, the 3D scanner 12, the robot base, or to another object.

[0292] By way of example only, Fig. 26A-B depicts a block diagram illustrating a method 390 of complex anatomy tracking with robot-controlled instrument in the context of a TKA, where the robot assists with cutting the femur.

[0293] In some embodiments, a first step 392 of the example method 390 of complex anatomy tracking with robot-controlled instrument is to mount the scanner 12 and the saw or saw blade 28 to a robot end effector 28, which by way of example may be an articulating arm having multiple degrees of freedom. In some embodiments, a next step 393 in the method 390 is to move (or initiate the robot to move) the robot arm to a known position in which the scanner 12 faces the robot base. In some embodiments, a next step 394 in the method 390 is to use the scanner 12 to acquire a dataset comprising structured light image data of the robot base (“robot base image dataset”). In some embodiments, a next step 395 in the method 390 is to upload a dataset comprising the robot base CAD geometry to the CAAT system 10 (“robot base CAD dataset”). In some embodiments, a next step 396 in the method 390 is to register the robot base image dataset to the uploaded robot base CAD dataset. In some embodiments, a next step 397 in the method 390 is to compute a transformation matrix T(B / S) (transformation matrix of the robot base dataset to the common coordinate system) at this fixed position to relate the robot base to the common coordinate system. In some embodiments, a next step 398 in the method 390 mayVIS-004-W01 be to track or continuously monitor robot position sensors and update the transformation matrix T(B / S) in real time.

[0294] By way of example, the above steps establish a robot base position in the common coordinate system. As previously mentioned, all other objects (3D scanner 12, instrument (e.g., saw) 28, femur 274, etc.) are transformed into the common coordinate system to ensure alignment. By way of example, the robot's base remains fixed, and in some embodiments a point the common coordinate system may be referenced to the base.

[0295] In some embodiments, the following steps of the method 390 of complex anatomy tracking with robot-controlled instrument may be followed to relate the position of the saw 28 or saw blade on the robot end effector to the common coordinate system. In some embodiments, this may be accomplished by expressing the position of the robot end effector in real time as a transformation matrix T(SB / S) (transformation matrix of the saw blade dataset to the common coordinate system). By way of example, the position of the robot's end effector may be known in real-time through the robot control system. This transformation matrix T(SB / S) is used to relate the end effector’s (and therefore the sawblade’s) position to the common coordinate system.

[0296] In some embodiments, a next step 399 in the method 390 is to use the 3D scanner 12 to acquire a structured light image dataset of the saw blade (“saw data set”) if the saw blade is positioned within the camera’s field of view or a saw blade calibration tool if the saw blade is not within the camera’s field of view. In some embodiments, a next step 400 in the method 390 is to upload a saw blade or saw blade calibration tool CAD geometry dataset to CAAT system 10. In some embodiments, a next step 401 in the method 390 is to register the saw dataset to the uploaded CAD geometry dataset. In some embodiments, a next step 402 in the method 390 is to compute a transformation matrix T(SB / S) (transformation matrix of the saw dataset to the scanner space) at the fixed position. In some embodiments, a next step 404 in the method 390 is to track or continuously monitor robot position sensors and update the transformation matrix T(SB / S) in real time.

[0297] In some embodiments, the following steps of the method 390 of complex anatomy tracking with robot-controlled instrument may be followed to register and track the femurVIS-004-W01 position in real-time within the common coordinate system. Tn some embodiments, the CAAT system 10 may accomplish this by producing the transformation matrix T(F / S) (transformation matrix of the femur to the common coordinate system).

[0298] In some embodiments, a next step 406 in the method 390 is to manually move the robot to position where the scanner faces the femur. In some embodiments, a next step 408 in the method 390 is to use the 3D scanner 12 to acquire a structured light image dataset of the femur (“femur dataset”), which will include a subset data representative of femoral cartilage. In some embodiments, a next step 410 in the method 390 is to determine whether a reference dataset (e.g., preoperative CT scan) of the femur is available. If yes, then in some embodiments, a next step 412 in the method 390 is to upload a segmented reference dataset of the femur to the CAAT system 10. In some embodiments, a next step 414 in the method 390 is to register the femur / femoral cartilage scan dataset to the reference (e.g., CT) dataset, for example using one or more methods described herein. If there is no available reference dataset, then in some embodiments, a next step 416 in the method 390 is to create a femur / femur cartilage model dataset. In some embodiments, a next step 418 in the method 390 is to register the created model dataset to the femur / femoral cartilage scan dataset, for example using one or more methods described herein. In some embodiments, a next step 420 in the method 390 is to compute a transformation matrix T(F / S) to transform this registration into the common coordinate system. In some embodiments, a next step 422 in the method 390 is to continuously repeat the previous steps (e g., scan femur / femoral cartilage, register the scan dataset to the reference dataset, and recompute the transformation matrix T(F / S) in real time to track or monitor the position of the femur in the common coordinate system.

[0299] In some embodiments, the registrations described above may be performed in any order, since the goal of registering each of the robot, target anatomy (e.g., femur), surgical instrument (e.g., saw blade) and / or scanner into a common coordinate system is the same regardless of which occurs first. For example, in some embodiments, it may be advantageous to register the patient anatomy first, using the patient anatomy as the origin of the common coordinate system to minimize the need to re-register the patient anatomy in the event of movement.VIS-004-W01

[0300] In some embodiments, the following steps of the method 390 of complex anatomy tracking with robot-controlled instrument may be followed to register the surgical plan (e.g., cut plane) into the common coordinate space and perform the surgical cut. By way of example, the cut plane is initially determined during preoperative planning and adjusted during surgery based on patient-specific anatomy and knee kinematics. In some embodiments, the cut plane may be defined relative to the femur.

[0301] In some embodiments, a next step 424 in the method 390 is to load a preoperative CT dataset and / or femur image data into a surgical planning tool and determine proper cut planes in the common coordinate system. In some embodiments, a next step 426 in the method 390 is to continuously convert robot base, scanner, and saw blade into the common coordinate system by applying the correct respective transformation matrices and / or their inverses as needed. In some embodiments, a next step 428 in the method 390 is to continuously update the robot’s position to validate the position of the robot with respect to the anatomy, for example to ensure the saw blade remains in the cut plane at all times even with movement of the femur. In some embodiments, a next step 429 in the method 390 is to manually power and move the saw blade within the cut plane as needed to execute the cut as planned.

[0302] By way of example, all of these transformation matrices are computed in real-time, and the data is continuously fed into the robot's control system, to maintain the saw’s alignment within the predefined cut plane (for example) while adjusting its position based on real-time anatomical movement. This allows the surgeon to follow the operative plan, for example to manipulate the saw within the plane while the robot ensures that the blade remains within the precise boundaries of the cut plane.

[0303] The linear algebra described previously and in this section is crucial for handling these complex tracking scenarios. It is important to note that the same principles of transforming anatomy and objects into a common coordinate space apply to all the previous use cases described but the equations were not explicitly given. Whether it’s tracking a single object like the femur, maintaining a stable field of view, or synchronizing a scanner and robot, the underlying process of sequentially applying transformation matrices remains a key part of the system. These transformations ensure that all components of the system — whether anatomy,VIS-004-W01 surgical instruments, or the robot — are correctly aligned within the SCS, enabling precise, realtime interactions.

[0304] By way of example, Fig. 27 is a block diagram depicting an example novel TKA workflow 430 with pre-operative imaging using the CAAT system 10 of the present disclosure. By way of example, the workflow 430 may be divided into several portions, including a preoperative portion 432, OR setup portion 434, initial registration portion 436, surgical planning portion 438, Bone resection portion 440, and implantation portion 442.

[0305] Regarding the pre-operative portion 432, prior to surgery, detailed imaging 444 of the target anatomy is often acquired using CT (Computed Tomography) or MRI (Magnetic Resonance Imaging). In some embodiments, these scans may be segmented to create a 3D model of the relevant structures, such as the femur and tibia in TKA. These 3D models are essential for pre-operative planning 446 of the procedure and aligning the patient's real-time anatomy with the surgical plan. Important anatomical planes and landmarks are also labeled within the model to assist the surgeon during the operation.

[0306] For this workflow, we assume a patient-specific CT scan has been segmented to provide a detailed 3D model of the femur and tibia, though the imageless mode can also be utilized if necessary.

[0307] By way of example, the OR setup portion 434 may include patient positioning 448, 3D scanner setup 450, robot setup 452, and instrument setup 454. Before surgery begins, the CAAT system 10 requires calibration. In some embodiments, scanner setup 450 comprises one or more 3D scanners 12 being positioned to ensure full visibility of the surgical field. Calibration ensures that the CAAT system 10 aligns correctly with the surgical setup, accounting for distances, camera angles, and the positioning of all equipment.

[0308] Robotic equipment requires additional steps for calibration to ensure that the robot functions correctly. The robot setup 452 process includes aligning the robot's physical movements with the virtual model of the surgery to maintain accuracy and ensure that the robot performs as expected during the procedure.VIS-004-W01

[0309] In some embodiments, patient positioning 448 occurs after setup and calibration. By way of example, in a TKA procedure the patient is positioned on the operating table, typically in a supine position with the knee exposed for TKA procedures, as shown by way of example in Fig. 4. The surgical site is prepped and draped in a sterile fashion.

[0310] By way of example, the initial registration portion 436 may include surgical exposure 456, scanning 458, bone to cartilage registration 460, hip center calculation 462 via tracking, and ankle center calculation 464 using a pointer. In some embodiments, upon completion of the OR setup 434 portion, surgical exposure 456 may occur, with an incision made to access the knee joint and prepare it for the robotic system to perform the procedure.

[0311] In some embodiments, femoral registration may occur as described above, by using the 3D scanner to generate a 3D point cloud model registered to the SCS. This real-time anatomical model is maintained within the Surgical Coordinate System (SCS), enabling accurate tracking of the patient’s anatomy throughout the procedure. In some embodiments, the CAAT system 10 may perform bone to cartilage registration 460, including misalignment correction as described below.

[0312] Following patient registration, the surgeon may identify key anatomical landmarks necessary for establishing the mechanical axis of the leg, for example including the hip center 462 (e.g. identified by rotating the leg while the CAAT system 10 calculates the center of rotation of the femoral head as described above), and / or ankle center 464 (e.g., identified using the optical tracking system on the medial and lateral malleoli (ankle bones) to calculate the center of the ankle joint). These landmarks, or others, or measured curvatures, for example, are crucial for ensuring proper implant alignment. The mechanical axis is one common methodology, but other alignment methods, like kinematic alignment, may use different anatomical landmarks depending on the procedure.

[0313] At this stage, a virtual surgical plan 438 is developed to guide the robotic system and surgeon throughout the procedure. If pre-operative planning was done, it serves as an initial basis but must be verified and adjusted in real time as the surgery progresses.VIS-004-W01

[0314] By way of example, the virtual surgical plan 438 may involve the implant selection 466, initial implant positioning 468, gap balancing via tracking 470, positioning adjustments 472, and cut plane definitions 474. Regarding implant selection 466, in some embodiments the CAAT system 10 may assist the surgeon in selecting the optimal implant size based on the patient’s anatomy. Proper implant sizing ensures both the long-term success of the surgery and post-operative joint function. Regarding implant positioning, in some embodiments the CAAT system 10 may help the surgeon determine the ideal implant position to achieve accurate joint alignment. The goal is to ensure proper leg alignment and balance between the medial and lateral sides of the knee. The alignment is measured in terms of varus (inward tilt) and valgus (outward tilt), with the system assisting the surgeon in making appropriate adjustments. In some embodiments, gap balancing 470 ensures equal tension in the soft tissues (primarily ligaments) around the knee joint throughout its range of motion. By way of example the CAAT system 10 dynamically tracks the relative positions of the bones and calculates the space between them to guide the surgeon in balancing the medial and lateral gaps during the procedure, as described above. At this time, any positioning adjustments 472 should be made. Regarding defining the cut plane 474, in some embodiments based on the selected implant size and position, the system defines the necessary osteotomies (bone cuts). These cuts must be precise to ensure proper implant placement.

[0315] In some embodiments, during the bone resection 440 portion, the CAAT system 10 assists the surgeon in making precise cuts according to the surgical plan. The system continuously tracks the patient's anatomy and provides feedback, ensuring the cuts are executed accurately. For example, during and / or after each bone cut, the system verifies the accuracy of the cuts, implant alignment, and gap balancing. If deviations are detected, the system alerts the surgeon, allowing for real-time adjustments to be made. In some embodiments, the CAAT system 10 may also update the active reference model after each cut so that the robot is executing the surgical plan according to a real-time representation of the anatomical state of the surgical site. (See, e.g., below discussion of transfer registration). By way of example, this cut and verify process occurs for each osteotomy cut, including anterior 476, posterior 478, distal 480, posterior chamfer 482, anterior chamfer 484, and / or tibia 486.VIS-004-W01

[0316] In some embodiments, after bone cuts are verified, the workflow 430 proceeds with the implantation 442 portion. In some embodiments, trial implants 488 are placed to assess the fit, alignment, and joint movement. If necessary, adjustments are made before the final implant 490 is inserted. In some embodiments, the surgeon then performs a final assessment (e.g., including range of motion tests 492) to ensure proper alignment and stability of the new joint. Once confirmed, the surgical site is closed 494, and the patient is prepared for recovery.

[0317] Zero-Reference Tracking (ZRT)

[0318] Fig. 28 is a block diagram illustrating several steps of an example method 810 for tracking objects in dynamic environments without requiring pre-existing reference models, fiducial markers, or preoperative imaging data according to some embodiments of the disclosure, which may be referred to herein as “zero reference tracking method 810” or “ZRT method 810”. The registration and tracking capabilities of the CAAT system 10 have been described herein as transforming at least two different data sets into a common coordinate space, where one of the data sets (e g., the reference data set) may be derived from preoperative data such as a MRI or CT scan. However, in some embodiments, the CAAT system 10 may be configured to use an initial reconstructed 3D point cloud generated from scanned topological data as a reference data set, obviating the need for preoperative data.

[0319] In some embodiments, a first step 820 in the ZRT method 810 is scene scanning to create a 3D reference model of the entire scene. In some embodiments, this process may follow the scanning and data acquisition process 250 and / or registration process 290 described with reference to Figs. 15-21 above, but without the use of preoperative data. In some embodiments, the CAAT system 10 may capture topological data of the target object(s) through a single scan or multiple scans from different angles or viewpoints, according to a first sub-step 822 of the ZRT method 810. In some embodiments, if a single scan provides sufficient accuracy and visibility of the target object(s), then the reference dataset may be derived from the single scan. In some embodiments, the CAAT system 10 may capture topological dataset of the target object(s) through multiple scans which may be stitched together according to a second sub-step 824 of the ZRT method 810 to improve accuracy by combining data from various angles, provide a wider field of view for larger or complex objects, and eliminate occlusions by capturing previouslyVIS-004-W01 hidden surfaces. By way of example, the resulting scan dataset is a high-fidelity three- dimensional reference model of the surgical scene, which can be filtered or manipulated to improve usability according to a third (optional) sub-step 826 of the ZRT method 810.

[0320] In some embodiments, a second step 830 in the ZRT method 810 is segmentation of target objects to isolate the target reference dataset(s). In some embodiments, segmentation results in the target object(s) being identified and isolated from the background. By way of example, the segmentation step 814 in the ZRT method 810 may be similar or identical to the segmentation step 294 in the registration process 290 described above with reference to Fig. 21. For example, as described above, in some embodiments, segmentation can be performed using various approaches, including but not limited to manual selection 832 (user-defined segmentation), algorithmic detection 834 (shape, contrast, or depth-based recognition), and machine learning-based segmentation 836 (automated object identification). In some embodiments, segmentation may be achieved by registering a plane to the table-top (for example), and isolate all items above the plane (e.g., between the registered plane and the 3D scanner. In some embodiments, this can be automatic, is registration-based (for the plane), and can simplify the scene. In some embodiments, a first object can be registered and then segmented out (e.g., differentiating by proximity to the reference scan after registration), which simplifies the scene by leaving only the additional objects to be registered. By way of example, this method is a form of segmentation by elimination of irrelevant items (e.g., any item below the registered plane) from the scanned scene altogether. In some embodiments, the segmented dataset for each object then becomes the target reference dataset, which can be filtered or otherwise manipulated to improve usability according to an optional sub-step 838 of the ZRT method 810.

[0321] In some embodiments, a third step 840 in the ZRT method 810 is registration and tracking in subsequent scans. In some embodiments, as new scans are taken in a real-time tracking application, the CAAT system 10 may register the target reference datasets to each new scan dataset, according to a first sub-step 842 of the ZRT method 810. By way of example, each new registration provides an accurate position and orientation for the object at that time point. In some embodiments, by repeating this at several time points, the CAAT system 10 can compute velocity, angular velocity, acceleration, and / or angular acceleration, according to a second sub-VIS-004-W01 step 844 of the ZRT method 810. Knowing the position, velocity, and acceleration (both translational and angular), the CAAT system 10 can extrapolate these values to estimate the future position and orientation of the object, according to a third sub-step 846 of the ZRT method 810. This predictive tracking (or “kinematic extrapolation”) reduces reliance on computationally expensive coarse registration algorithms such as RANSAC, improving real-time performance. In some embodiments, if the object is partially occluded in any scan, the CAAT system 10 can continue tracking as long as some portion of the reference geometry remains visible. In some embodiments, the visible portion must be sufficient to contain enough features to lock registration in six degrees of freedom so that the reference model can be updated in real time.

[0322] In some embodiments, the ZRT method 810 boasts several innovations and advantages over prior art systems. For example, on key advantage is that pre-existing object data is not required to establish a baseline registration. The ZRT method 810 disclosed herein works with any object that is scannable by the 3D scanner 12, even if no preoperative imaging exists. Additionally, the ZRT method 810 eliminates the need for fiducial markers or tracking devices. In some embodiments, another key advantage of the ZRT method 810 disclosed herein is in object-agnostic tracking. For example, the ZRT method 810 disclosed herein can be applied to any object, including bones, cartilage, instruments, or unknown objects in a scene. By way of example, another key advantage of the ZRT method 810 disclosed herein is elimination of occlusions with multi-angle scanning, which helps to ensure stable tracking as angles change. By way of example, another key advantage of the ZRT method 810 disclosed herein is more accurate registration over time. For example, the stitched reference model allows for stronger alignment with new scans, reducing tracking drift. By way of example, another key advantage of the ZRT method 810 is kinematic extrapolation for future position prediction. In some embodiments, the method enables motion path prediction, reducing reliance on computationally expensive registration methods and increasing registration speed. By way of example, another key advantage of the ZRT method 810 is its applicability to various industries, including but not limited to surgical navigation (tracking anatomy and instruments in real-time), manufacturing & automation (monitoring objects in assembly lines), and robotics & security (real-time tracking of unknown objects).

[0323] Example: Tracking a surgical instrument without a pre-existing 3D modelVIS-004-W01

[0324] In a first practical example, a surgeon wants to track the position and orientation of a surgical instrument, but no pre-existing 3D model of the instrument is available. Following the ZRT method 810 disclosed herein, the surgeon first performs an initial scan 820 of the instrument (e.g., in this case the instrument is the scene) by placing the instrument on a table and scanning the instrument from multiple angles to acquire one or more topological datasets of the instrument’s surface (e.g., sub-step 822 of the ZRT method 810). In some embodiments, the CAAT system 10 then stitches the individual topological datasets together (if necessary) to form a complete reconstructed 3D model of the instrument (e.g., sub-step 824 of the ZRT method 810). By way of example, in the segmentation step 830, the CAAT system 10 isolates the instrument scan data from the background scan data (e.g., of the table and any other objects in the scan) to create a target reference dataset comprising only the instrument scan data. During the third step 840 of registering and tracking in subsequent scans, as the instrument moves, the CAAT system 10 continuously registers new scan data to the reference model (e g., sub-step 842 of ZRT method 810), tracking position and orientation in real time.

[0325] Several advantages of the ZRT method 810 disclosed herein are manifested in this first practical example. For example, because the ZRT method 810 requires no pre-scanned model, the method works with any surgical tool. Additionally, a scanning system using the ZRT method 810 may adapt to new instruments more quickly enabling surgeons to scan and track tools dynamically. Furthermore, the ZRT method 810 allows tracking with full six degrees of freedom, ensuring that position (e.g., x, y, z coordinates) and orientation (e.g., roll, pitch, and yaw) of the instrument within the common coordinate system are precisely monitored. Finally, kinematic extrapolation ensures that the motion path of the target can be extrapolated to future time points, estimating its future location and orientation.

[0326] Example: Tracking cartilage in a TKA procedure

[0327] In a second practical example (e.g., a TKA procedure), the femoral and tibial cartilage surfaces are exposed. If no MRI or CT data is available, traditional tracking methods fail because (1) there is no pre-op reference model of cartilage, and (2) the bone beneath the cartilage is obscured, preventing bone-based tracking.VIS-004-W01

[0328] Following the ZRT method 810 disclosed herein, the surgeon first performs an initial scene scan (e.g., first step 820) using the 3D scanner 12 or other structured light scanner to capture topological data of the cartilage surfaces, which for example may be a single scan or multiple scans taken from different angles (e.g., sub-step 822). If multiple scans are taken, the CAAT system 10 stitches the data together into a complete reconstructed 3D model of the surgical site (e.g., sub-step 824). In the segmentation and reference model creation step, the system segments the femoral cartilage scan data (e.g., including both femur and femoral cartilage) as one reference target dataset and the tibial cartilage scan data (e.g., including both tibia and tibial cartilage) as another target reference dataset, and then saves these segmented object surface datasets as target reference geometry for tracking. During the third step 840 of registering and tracking in subsequent scans, as the knee is manipulated, new scans are taken and registered to the reference model (e.g., sub-step 842). In some embodiments, the CAAT system 10 can track multiple objects (e.g., femur with femoral cartilage and tibia with tibial cartilage) independently. Relative motion analysis between the two bones is possible, providing real-time feedback on joint kinematics.

[0329] Several advantages of the ZRT method 810 disclosed herein are manifested in this first practical example. For example, the CAAT system 10 using the ZRT method 810 tracks cartilage without pre-op imaging (e.g., no CT or MRI is needed). Additionally, the CAAT system 10 using the ZRT method 810 performs stable tracking throughout the procedure, and cartilage remains a reliable reference. Furthermore, the CAAT system 10 using the ZRT method 810 calculates real-time joint motion, which can enhance surgical decision-making. Finally, kinematic extrapolation helps the motion path of the target to be extrapolated to future time points, estimating its future location and orientation.

[0330] Transfer Registration

[0331] In some embodiments, the CAAT system 10 may employ a process referred to as “transfer registration.” By way of example, transfer registration is a specific implementation of zero reference tracking, wherein each newly acquired optical scan is registered to a prior reference scan (and currently active reference model) representing the immediately preceding anatomical state. In some embodiments, each registration thereby links the current intraoperativeVIS-004-W01 anatomical state to the previous anatomical state. Through a sequence of intermediate anatomical states, the current anatomical state is linked to the original anatomic reference model. Because each transformation is based on successive stages of anatomy exhibiting maximum geometric commonality, the resulting registrations are statistically robust. In some embodiments, successful registrations rely on statistical sufficiency of common anatomical features between consecutive stages. It is important in transfer registration to maintain common anatomic structure sufficient for confident registration in all six degrees of freedom. For this reason, it is likely that scans will take place at standard timepoints in a procedure. Metrics on the number of inliers and their distance, and statistical outputs such as covariant matrices could enable the system to quantify the robustness of each stage’s registration.

[0332] In surgical contexts such as TKA or spine surgery, transfer registration may be used to update the active reference model as anatomical modifications occur. For example, in TKA, the updated reference may exclude bone and / or cartilage that has been resected and include newly exposed cut surfaces. In spine procedures, for example, the reference may similarly exclude removed bony structures while incorporating newly revealed rigid anatomy. Rigidly affixed, relatively immobile, or otherwise untouched tissue may also contribute to stable registration across stages when such tissue remains unaffected by the surgical manipulation. Although described primarily in the context of a TKA, transfer registration is suitable for many procedures where anatomy is affected, for example resection of bone, and is not limited to those discussed here.

[0333] In some embodiments, between major stages of anatomical change, the system may acquire new optical scans to update the current reference frame. For example, during stages in which the anatomy remains relatively unchanged (e.g., during drilling or probing operations), the system may rely on continuous motion tracking relative to the most recent reference, maintaining spatial accuracy in real time. In this manner, the system combines stage-based re-registration with intra-stage tracking to preserve registration accuracy throughout the procedure.

[0334] This approach allows intraoperative registration updates without cumulative drift exceeding acceptable limits, provided that each individual registration meets accuracy requirements and sufficient anatomical overlap exists between consecutive states.VIS-004-W01

[0335] By way of example, Fig. 29 illustrates one example of a transfer registration method 850 employed by the CAAT system 10, according to some embodiments.

[0336] In some embodiments, a first step 852 in the transfer registration method 850 is to generate an original anatomic reference model representing a first anatomic state of one or more objects in the surgical site. By way of example, the original reference model may be generated as described above, based on pre-operative imaging (e.g., CT, MRI, etc.) or by the ZRT method 810. In some embodiments, this original anatomic reference model serves as the first active reference model in the procedure.

[0337] In some embodiments, a second step 854 in the transfer registration method 850 is to register and track the one or more objects within a common coordinate system. By way of example, the registration and tracking may occur in accordance with any of the registration and tracking methods disclosed herein.

[0338] In some embodiments, a third step 856 in the transfer registration method 850 is performing an action resulting in (or the CAAT system 10 otherwise detecting) a first changed anatomic state of at least one of the one or more tracked objects in the surgical site. In some embodiments, this action may be performing an osteotomy cut, for example.

[0339] In some embodiments, a fourth step 858 in the transfer registration method 850 is acquiring subsequent scan data by scanning the surgical site using the 3D scanner 12, for example.

[0340] In some embodiments, a fifth step 860 in the transfer registration method 850 is to update the active reference model (e.g., the original reference model) to generate a first updated reference model representing a second anatomic state of the surgical site. By way of example, this may occur by registering the subsequent scan data to the original reference model and creating a first updated reference model. In some embodiments, the generated first updated reference model serves as the active reference model for the procedure moving forward.

[0341] In some embodiments, a sixth step 862 in the transfer registration method 850 is performing another action resulting in (or the CAAT system 10 otherwise detecting) a secondVIS-004-W01 changed anatomic state of at least one of the one or more tracked objects in the surgical site. In some embodiments, this action may be performing another osteotomy cut, for example.

[0342] In some embodiments, a seventh step 864 in the transfer registration method 850 is acquiring further subsequent scan data by scanning the surgical site using the 3D scanner 12, for example.

[0343] In some embodiments, an eighth step 866 in the transfer registration method 850 is to update the currently active reference model (e.g., the generated first updated reference model) with further subsequent scan data to generate a second updated reference model representing a third anatomic state of the one or more tracked objects in the surgical site. By way of example, this may occur by registering the further subsequent scan data to the first updated reference model and creating a second updated reference model. In some embodiments, the generated second updated reference model then serves as the active reference model for the procedure moving forward.

[0344] In some embodiments, a ninth step 868 in the transfer registration method 850 is to repeat the scanning and updating events until the procedure is concluded such that the surgical plan is carried out with a real-time updated reference model of the surgical site before each cut is made.

[0345] Articular Surface Matching (ASM)

[0346] As discussed above, existing surgical navigation techniques, both image-based and imageless, face limitations that impact surgical precision and outcomes. Traditional methods often rely on preoperative imaging or manual landmark identification, each with inherent challenges that can lead to variability in patient outcomes and prolonged intraoperative time.

[0347] By way of example, in some embodiments, the CAAT system 10 may be configured to perform an articular surface matching (ASM) technique that overcomes these limitations by introducing a data-driven, automated approach to joint mapping that eliminates the need for optical trackers and manual landmark identification. By harnessing high-resolution topological data, the ASM technique performed by the CAAT system 10 brings a new level of accuracy and efficiency to surgical navigation, building upon the advancements in imageless navigation (suchVIS-004-W01 as the Personalized Alignment™ method discussed above) to further optimize surgical outcomes. By way of example, the primary objective of the ASM technique is to restore the joint to ideal or near-ideal biomechanics (e.g., which may be a balance between ideal biomechanics and the patient’s specific or tolerable biomechanics), optimizing postoperative functionality and extending the longevity of the implant.

[0348] By way of example, a key concept of the ASM technique is the precise matching of the patient’s articular surfaces to a best-fitting implant. In some embodiments, during a surgical procedure (e.g., a TKA), the patient’s articular surfaces of the femur and tibia may be scanned to collect image data to produce a dense 3D topological map of the cartilage on the femoral condyles and the tibial plateau, for example using the methods described above. In some embodiments, the CAAT system 10 may be configured to compare this topological data against 3D surface maps of the available implants for the procedure, which may include multiple sizes and designs. For instance, if five sizes are available for one implant design and four sizes for another, the CAAT system 10 registers each of the nine available femoral components to the topological map of the patient’s femoral articular cartilage surfaces.

[0349] In some embodiments, the CAAT system 10 further includes a fitness function which then assesses the quality of each registration, helping determine the implant that best fits the patient’s anatomy and optimally restores the biomechanics of the diseased joint to ideal or nearideal biomechanics. In its initial stage, the ASM technique provides optimal positioning for the implant based on the diseased joint condition. To achieve a pre-arthritic joint alignment, further adjustments are made using one of several methods to compensate for the bone shape and cartilage thickness changes associated with disease or trauma, including but not limited to (and by way of example only) manual adjustment by the surgeon, cartilage thickness measurement by probe, statistical correction based on published data, and correction through expanded topological data analysis.

[0350] In some instances, the surgeon may manually adjust the implant positioning, using the surgical planning software 20, to a pre-arthritic condition based on their experience examining many examples of both healthy and diseased joints. These adjustments account for the alteredVIS-004-W01 bone shape and cartilage thickness resulting from disease or trauma, ensuring alignment that approximates the joint's pre-arthritic condition.

[0351] In some embodiments, the surgeon may use an instrument to probe and measure cartilage thickness around visible defects. For example, if this instrument is registered or tracked by the CAAT system 10, data can be automatically captured at each probing point, enabling a more precise alignment adjustment based on actual cartilage thickness. By way of example, this is the method used by the “Personalized Alignment” method, and it works equally well if used with the CAAT system 10 performing the ASM technique.

[0352] In some embodiments, the CAAT system 10 performing the ASM technique can leverage statistical data from published studies on joint degeneration patterns. By way of example, in this approach, the system 10 may compare the collected cartilage map to standard patterns of osteoarthritis progression, and apply a correction based on a statistical model. This method provides a generalized correction that is likely a good correction but also likely not the ideal correction for each individual patient.

[0353] In some embodiments, the system 10 performing the ASM technique uses the topological data gathered by the 3D scanner 12 (or other advanced and / or structured light scanners), which can differentiate tissue types based on unique interactions with light. For example, cartilage’s semi-transparency to infrared (IR) light allows IR to penetrate beyond its surface before scattering back, providing a signal regarding optical penetration depth in areas with cartilage. This extra data may enable the CAAT system 10 to determine tissue types as well as estimate cartilage thickness comprehensively across the joint, improving implant positioning accuracy based on disease-compromised cartilage thickness. By way of example, commonly owned and co-pending PCT Application No. PCT / US25 / 24625 entitled “System and Related Methods for the Characterization of Objects Through Subsurface Scattering Across One or More Edges of Luminosity” describes several methods by which tissue differentiation can be accomplished with an advanced 3D scanner.

[0354] Advantages of ASMVIS-004-W01

[0355] With its comprehensive approach to articular surface matching, the ASM technique offers distinct advantages that address the limitations of traditional surgical navigation and elevate the precision of joint restoration. By way of example, one such advantage of the ASM technique is a reduced setup complexity and elimination of patient-mounted trackers. By way of example, the ASM technique eliminates the need for optical trackers on the patient, which are commonly required in traditional imageless navigation systems. By using high-resolution topological mapping instead of manual tracking, the CAAT system 10 using the ASM technique reduces setup complexity and avoids potential errors from shifted or mispositioned markers. This streamlined setup improves workflow efficiency, minimizes patient discomfort, healing time, and risk, and enhances overall surgical accuracy.

[0356] Another advantage of the ASM technique is that it utilizes implant design intent and geometry. For example, the ASM technique leverages the specific design intent and geometry of the implants selected by the surgeon, enabling accurate, customized determination of implant size, placement, and necessary bone resections. By aligning the implant with the joint's natural anatomy, utilization of the ASM technique enhances the reliability and precision of surgical outcomes, improving implant longevity and joint function.

[0357] Yet another advantage of the ASM technique is the elimination of manual landmark identification. Traditional imageless navigation methods rely on surgeons manually identifying anatomical landmarks. In contrast, the ASM technique eliminates this requirement by employing advanced algorithms that use topological data to automatically generate an accurate map of the joint surface. This automation reduces intraoperative time, minimizes surgeon variability, and lowers the risk of errors associated with manual landmarking.

[0358] It should be noted that there may be other decisions a surgeon wants to make that do require anatomical landmarks. Since the system 10 does not probe these points during the ASM technique, in some embodiments, the system 10 can obtain them by essentially inferring them from the design intent of the implant. For example, every implant is designed based on an assumption of where these landmarks are relative to various features on the implant. Since the implant is aligned to the patient anatomy, the data inherent in the implant design can be used to identify where these landmarks are. This is an inference but if the fit is good, the inference isVIS-004-W01 likely good. In theory, this may be comparable in accuracy to using a pointer. Tn some embodiments, the assumed positions of the anatomical landmarks may need to be provided by implant manufacturers with the implant geometry data.

[0359] Another advantage of the ASM technique is correction for changes in cartilage thickness due to disease or trauma. By way of example, the ASM technique adjusts for changes in cartilage thickness caused by disease or trauma, using one of four flexible methods. These include manual adjustments by the surgeon, cartilage measurements taken with a probe, statistical corrections based on published data, computation of cartilage thickness based on registration of topological data from a scanner (such as the 3D scanner 12 disclosed herein) which includes cartilage and bone surface data to a pre-operative CT scan (for example) which includes only bone surface data and then calculating the distance between the cartilage and bone surface, or advanced analysis of topological data from a scanner (such as the 3D scanner 12 disclosed herein) that estimates cartilage thickness through tissue differentiation. This adaptability allows the ASM technique to tailor implant positioning to each patient’s specific anatomical changes.

[0360] Yet another advantage of the ASM technique is its adaptability to multiple sources of topological data: the ASM technique is designed to integrate with a variety of topological data sources, including structured light scanners (such as the scanner 12 disclosed herein) and other high-resolution mapping technologies. While structured light technology is one example, the CAAT system 10 utilizing the ASM technique can incorporate data from other devices or emerging technologies as long as they provide detailed topographical data. This versatility makes the ASM technique both adaptable and future ready.

[0361] By way of example the ASM technique overcomes many limitations inherent in previous navigation technologies by providing a more reliable, efficient, and precise approach to joint mapping and implant positioning. Its automated and comprehensive mapping capabilities reduce variability in surgical outcomes, shorten intraoperative time, and enhance patient safety by eliminating the need for preoperative imaging. The ASM technique’s accuracy in joint surface mapping positions it as a transformative solution in surgical navigation, offering a new standard of care for surgeries requiring precise anatomical alignment and implant placement.VIS-004-W01

[0362] By way of example, Fig. 30 is a block diagram depicting an example TKA workflow 630 using the CAAT system 10 of the present disclosure without pre-operative imaging and using the ASM technique. In some embodiments, the workflow 630 demonstrates how the ASM technique enhances a Robot-Assisted Total Knee Arthroplasty (RA-TKA) procedure, optimizing implant positioning and alignment through precise articular surface matching and automated mapping techniques. It should be noted that none of the steps that follow require an optical tracker or manual anatomical landmark identification by the surgeon. By way of example, the workflow 630 may be divided into several portions, including a pre-operative portion 632, OR setup portion 634, data collection portion 636, surgical planning portion 638, bone resection portion 640, and implantation portion 642.

[0363] Regarding the pre-operative portion 632, prior to surgery, a standard 3D model 646 of the bones and implants may be used instead of patient-specific scans described in the previous workflow 430. Anatomical landmarks, such as the medial and lateral epicondyles of the femur, are identified during surgery using the 3D scanner 12 as described above. In some embodiments, the distance between these landmarks is used to scale the standard model to match the patient’s unique anatomy, allowing the surgeon to visualize implant placement with appropriate proportions, even in the absence of patient-specific imaging. These 3D models 646 may be used for pre-operative planning 644 of the procedure and aligning the patient's real-time anatomy with the surgical plan. Important anatomical planes and landmarks are also labeled within the model to assist the surgeon during the operation.

[0364] By way of example, the OR setup portion 634 may include scanner setup 648, robot setup 650, instruments setup 652, patient positioning 654, and surgical exposure. In some embodiments, scanner setup 648 comprises one or more 3D scanners 12 being positioned to ensure full visibility of the surgical field and registration of the one or more 3D scanners 12 in a common coordinate system. In some embodiments, robot setup 650 may include registration of the robot base in the common coordinate system. In some embodiments, instrument setup 652 may include registration of a surgical instrument within the common coordinate system, including but not limited to a saw blade (for example) attached as a robot end effector. By way of example, registration ensures that the various components of the CAAT system 10 align correctly to one another within the surgical setup, accounting for distances, camera angles, andVIS-004-W01 the positioning of all equipment such that the scanner 12 captures accurate topological data and the robot aligns precisely with the anatomical reference points established by the CAAT system 10 using ASM methodology. No optical trackers are attached to the patient, as the CAAT system 10 using ASM methodology does not rely on manual landmarking or traditional tracking markers. Furthermore, no OTs are required on the robot because the robot’s position can be verified by scanning a fixed part of the robot, usually the robot base. In some embodiments, the scanned data is compared to a 3D model of the robot base to obtain the needed transformation matrices to place objects into any coordinate system needed. In some embodiments, the robot base is the reference of the common coordinate system so using this transform, anything in scanner space can be transformed into the common coordinate system. In some embodiments, no OTs are required if all that is required is to determine the position of the robot end-effector with respect to the anatomy, as that may be achieved in certain situations by scanning the tool near the anatomy. The anatomy can be registered to the “tool space,” which the robot knows relative to its base. The robot can then make the appropriate motions.

[0365] In some embodiments, patient positioning 654 may occur after setup and calibration. In some embodiments, patient positioning 654 may occur before setup and calibration. By way of example, in a TKA procedure the patient is positioned on the operating table, typically in a supine position with the leg stabilized to maintain the necessary surgical angles, as shown by way of example in Fig. 4. By way of example, this setup is critical to ensure consistent reference points during the surgical procedure. In some embodiments, the RA-TKA procedure utilizing the ASM technique begins by preparing the joint for topological data collection to create an accurate 3D map of the patient’s target articular surfaces. Unlike traditional imageless systems, the ASM technique bypasses the need for patient-mounted trackers or manual identification of anatomical landmarks. This streamlined setup ensures that the joint surface can be mapped accurately and consistently, forming the foundation for precise implant selection and positioning. In some embodiments, the surgical site may be prepped and draped in a sterile fashion. In some embodiments, upon completion of patient positioning 654, surgical exposure 656 may occur, with an incision made to access the knee joint and prepare it for the robotic system to perform the procedure.VIS-004-W01

[0366] By way of example, the data collection portion 636 may include ZRT scanning and segmentation 656, registering all implant models and grade fit 660 and automatic implant selection and positioning 662, ankle center tracking 664, hip center tracking 666, and gap balancing 668.

[0367] In some embodiments, femoral registration may occur by way of the ZRT method 810 as described above, by using the 3D scanner 12 to generate a 3D point cloud model registered to the common coordinate system. This real-time anatomical model is maintained within the common coordinate system, enabling accurate tracking of the patient’s anatomy throughout the procedure. In some embodiments, in particular, the structured light 3D scanner 12 or a similar high-resolution imaging device is used to capture detailed 3D topological data of the femoral and tibial articular surfaces. The CAAT system 10 utilizing the ASM technique, generates a comprehensive map (e.g., the real-time anatomical model) that includes the contours and unique anatomical details of the joint surfaces and serves as the basis for the subsequent matching process, ensuring that the implant will be aligned with the patient’s natural anatomy.

[0368] By way of example, the steps of registering all implant models and grade fit 660 and automatic implant selection and positioning 662 represent a crucial component of the ASM technique, in that the CAAT system 10 analyzes intraoperatively acquired topological data of the patient’s articular surfaces, and automatically compares them to digital models of available implants, assessing which implant design and size will best match the patient’s unique anatomy. For example, in some embodiments the CAAT system 10 may register each available femoral and tibial implant to the patient's anatomy, assessing the quality of each registration through a fitness function (e.g., grade fit). In some embodiments, this fitness function identifies the bestfitting implant size and design based on the initial diseased joint condition, determining an optimal implant for anatomical compatibility. Because this is done with a registration process, when the best fitting implant is found, it is also properly positioned relative to the CT data.Thus, the CAAT system 10 using ASM technique is configured to automatically select the best implant and place it in the best location for the patient. Additionally (or alternatively), CAAT system 10 may be configured to display the results of the fitness analysis (or at least a certain number of top scoring options) so that the surgeon can see all the grades and manually select their desired implant. In some embodiments, this may allow the surgeon to see multiple implantsVIS-004-W01 having similar fitness results and determine if any of the modifications they might want to make based on other data or their experience favors an implant that otherwise has a near equivalent fitness results or grade.

[0369] By way of example, the data collection portion 636 may further include ankle center tracking 664, hip center tracking 666, and gap balancing 668, for example as described herein above.

[0370] At this stage, a virtual surgical plan 638 is developed to guide the robotic system and surgeon throughout the procedure. If pre-operative planning was done, it serves as an initial basis but may be verified and adjusted in real time as the surgery progresses. By way of example, after the data collection 636 has been completed, the virtual surgical plan 638 portion may involve modification of the surgical plan 670 (e.g., based on collected data compared with the pre-operative planning 644), cut plane definitions 672, and switching to scanning and tracking mode 674.

[0371] For example, in some embodiments, after identifying the best-fitting implant based on the diseased joint’s structure, the CAAT system 10 using ASM technique may be configured to adjust the implant positioning to approximate the pre-arthritic state. This adjustment can be made using one of several methods. For example, in some embodiments, the surgeon may manually adjust the implant position in the planning software, informed by their expertise in accounting for disease-related changes in bone shape and cartilage thickness. In some embodiments, if a registered probe is available, the surgeon measures cartilage thickness around affected areas, allowing the CAAT system 10 using ASM technique to adjust the implant position automatically based on these data points. In some embodiments, the CAAT system 10 can also apply published statistical data to estimate typical cartilage thickness and make appropriate corrections. In some embodiments, if using a scanner capable of tissue differentiation, the CAAT system 10 may interpret tissue-specific responses to estimate cartilage thickness across the joint surface. For example, the CAAT system 10 can interpret this data to locate diseased areas and modify the patient’s topological data set to eliminate these defects by adding thickness in regions where defects are noted.VIS-004-W01

[0372] By way of example, after these adjustments are made to the patient’s data set to account for any cartilage thickness corrections, implant registration may be repeated so that the ideal position can be determined. Additionally, other implant sizes and designs should be reregistered to ensure the changes do not change the implant selection.

[0373] In some embodiments, with the cartilage thickness corrections applied, the surgeon may optionally choose to make additional adjustments to address any other biomechanical conditions unique to the patient. For example, the surgeon may wish to restore the pre-arthritic condition of the knee closely but also account for any other condition, like a congenital varus / valgus (V / V) misalignment. This customization allows the surgeon to choose, based on their experience, a joint alignment that is the right balance between ideal biomechanics and restoration of the patient’s specific biomechanics. By way of example only, the additional adjustments may include (but are not limited to) V / V alignment (e.g., correcting for any medial or lateral alignment deviations to improve overall knee stability), flexion / extension balance (e.g., adjusting implant orientation to maintain or correct flexion and extension balance, improving joint movement range), internal / external rotation (e.g., ensuring the implant aligns correctly with the rotational axis of the joint, optimizing rotational stability), joint line elevation (e.g., adjusting the elevation of the joint line to restore the anatomical joint height and preserve normal knee kinematics), and / or posterior slope (e.g., modifying the posterior slope of the tibial component to optimize load distribution and reduce strain on the implant). By way of example, these optional adjustments provide the surgeon with flexibility to address any unique biomechanical characteristics, maximizing postoperative joint function.

[0374] In some embodiments, with the implant registered to the patient's anatomy and the pre-arthritic and optional biomechanical adjustments applied, the CAAT system 10 using ASM technique may create a precise virtual model of the planned joint replacement. This model provides the surgical team with a detailed visualization of the implant’s placement and orientation relative to the patient’s joint structure, enabling fine-tuning of the surgical plan.

[0375] Regarding defining the cut plane 672, in some embodiments based on the selected implant size and position, the CAAT system 10 defines the necessary osteotomies (bone cuts). These cuts must be precise to ensure proper implant placement. For example, if cut planes areVIS-004-W01 defined in the implant data, once the CAAT system 10 using the ASM technique selects and positions the implant, the cut planes are essentially defined. By way of example, for knee implants, these cut planes are essentially the planar surfaces on the implant that are designed to approximate the bone cuts. In some embodiments, this data is then used by the CAAT system 10 to perform the cuts using the robot to aid the surgeon by maintaining cut planes. In some embodiments, the 3D scanned ZRT models used in transfer registration can be anticipated using this plan (e.g., the planned planes can be made to affect an initial ZRT model). This anticipated model may be modified based on the scan of the actual cut.

[0376] In some embodiments, during the bone resection 640 portion, the CAAT system 10 assists the surgeon in making precise cuts according to the surgical plan. For example, the CAAT system 10 provides instructions to the robot to adjust the cutting tool position per the surgical plan (box 676) if needed. The surgeon then executes the cut with robotic assistance (box 678). In some embodiments, the CAAT system 10 continuously tracks the patient's anatomy and provides feedback during the cutting, ensuring the cuts are executed accurately 680 (e.g., according to the surgical plan). In some embodiments, after each cut is made, the CAAT system 10 may scan the target joint to verify if the newly created planar surface is correctly located relative to the rest of the bone. If an error was made, for any reason, this verification can alert the surgeon quickly so that adjustments to other cuts can be made to accommodate the error. By way of example, this cut and verify process may repeat until all cuts are complete (box 682). In some embodiments, reference model updating for transfer registration may also occur at this time.

[0377] In some embodiments, after all bone cuts have been completed, the workflow 630 proceeds with the implantation 642 portion. In some embodiments, trial implants 688 are placed to assess the fit, alignmentjoint movement, and stability of the planned implant. By way of example, the virtual model generated by the CAAT system 10 using the AMS technique provides a reference to verify alignment accuracy and allows for further adjustments to optimize functionality and comfort. In some embodiments, the surgeon may perform range of motion, laxity, and patella tracking tests 686 to ensure proper alignment and stability of the new joint. In some embodiments, following verification of the trial components, the final implants may be placed by the surgeon (box 688). With the implant positioned based on the patient’s pre-arthriticVIS-004-W01 anatomy, this step ensures that the joint mechanics are restored as closely as possible to the patient’s natural biomechanics. Once confirmed, the surgical site is closed 690, and the patient is prepared for recovery.

[0378] In some embodiments, upon completion of the procedure, the CAAT system 10 may be configured to provide a detailed report of the implant’s positioning and alignment metrics, aiding in postoperative assessments. This documentation supports ongoing care, for example by recording that the surgical goals have been achieved and that the joint alignment and function are optimized for long-term success.

[0379] By way of example, the ASM workflow 630 in RA-TKA demonstrates a transformative approach to surgical navigation by integrating detailed articular surface matching, advanced topological data analysis, and precise implant alignment. By eliminating traditional manual landmark identification, patient-mounted trackers, and the need for preoperative imaging, the ASM technique reduces radiation exposure, enhances workflow efficiency, and standardizes outcomes across surgical teams. This approach not only streamlines the procedure but also improves patient safety, allowing surgeons to make refined, patient-specific adjustments that replicate pre-arthritic joint biomechanics with higher consistency. As a result, the ASM technique establishes a new standard in knee arthroplasty, empowering surgeons with tools for safer, more precise procedures that can yield lasting improvements in joint function and implant longevity.

[0380] Figs 31-32 are example block diagrams of computer-implemented electronic devices 500, 550 that may be used to implement the systems and methods described in this document, as either a client or as a server or plurality of servers. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device 550 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart-phones, and other similar computing devices. In this example, computing device 500 may represent stationary computer or hub 12 and / or other computing systems referenced in this disclosure. The components shown here, their connectionsVIS-004-W01 and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations described and / or claimed in this document.

[0381] Referring to Fig. 31, computing device 500 includes a processor 502, memory 504, a storage device 506, a high-speed interface 508 connecting to memory 504 and high-speed expansion ports 510, and a low-speed interface 512 connecting to low-speed bus 514 and storage device 506. Each of the components 502, 504, 506, 508, 510, and 512, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor 502 can process instructions for execution within the computing device 500, including instructions stored in the memory 504 or on the storage device 506 to display graphical information for a graphic user interface (GUI) on an external input / output device, such as display 516 coupled to high-speed interface 508. In other implementations, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices 500 may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

[0382] The memory 504 stores information within the computing device 500. By way of example only, the memory 504 may be a volatile memory unit, non-volatile memory unit, or another form of computer-readable medium, such as a magnetic or optical disk (for example).

[0383] The storage device 506 is capable of providing mass storage for the computing device 500. In one implementation, the storage device 506 may be or contain a non-transitory computer-readable medium (e.g., any and all computer-readable media except transitory, propagating signals), such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. A computer program product can be tangibly embodied in an information carrier. The computer program product may also contain instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 504, the storage device 506, or memory on processor 502.VIS-004-W01

[0384] The high-speed interface 508 manages bandwidth-intensive operations for the computing device 500, while the low-speed interface 512 manages lower bandwidth-intensive operations. Such allocation of functions is by way of example only. In one implementation, the high-speed interface 508 is coupled to memory 504, display 516 (e.g., through a graphics processor or accelerator), and to high-speed expansion ports 510, which may accept various expansion cards (not shown). In the implementation, low-speed interface 512 is coupled to storage device 506 and low-speed expansion port 514. The low-speed expansion port may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) and may be coupled to one or more input / output devices, such as a keyboard 518, a printer 520, a scanner 522, or a networking device such as a switch or router 524, e.g., through a network adapter.

[0385] The computing device 500 may be implemented in a number of different forms. For example, it may be implemented as a standard server, or multiple times in a group of such servers. It may also be implemented as part of a rack server system. In addition, it may be implemented in a personal computer such as a laptop computer. Alternatively, components from computing device 500 may be combined with other components in a mobile device, such as device 550 (Fig. 32). Each of such devices may contain one or more of computing device 500, 550, and an entire system may be made up of multiple computing devices 500, 550 communicating with each other.

[0386] Referring to Fig. 32, computing device 550 includes a processor 552, memory 554, an input / output device such as a display 556, a communication interface 558, and a transceiver 560, among other components. The device 550 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components 550, 552, 554, 556, 558, and 560, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

[0387] The processor 552 can execute instructions within the computing device 550, including instructions stored in the memory 554. The processor may be implemented as a chipset of chips that include separate and multiple analog and digital processors. Additionally, the processor may be implemented using any of a number of architectures. For example, theVIS-004-W01 processor 552 may be a CISC (Complex Instruction Set Computers) processor, a RISC (Reduced Instruction Set Computer) processor, or a MISC (Minimal Instruction Set Computer) processor. The processor may provide, for example, for coordination of the other components of the device 550, such as control of user interfaces, applications run by device 550, and wireless communication by device 550.

[0388] The processor 552 may communicate with a user through control interface 562 and display interface 564 coupled to a display 556. The display 556 may be, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display) display or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface 564 may comprise appropriate circuitry for driving the display 556 to present graphical and other information to a user. The control interface 562 may receive commands from a user and convert them for submission to the processor 552. In addition, an external interface 566 may be provided in communication with processor 552, so as to enable near area communication of device 550 with other devices. External interface 566 may provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used.

[0389] The memory 554 stores information within the computing device 550. The memory 554 can be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. Expansion memory 568 may also be provided and connected to device 550 through expansion interface 570, which may include, for example, a SIMM (Single In Line Memory Module) card interface. Such expansion memory 568 may provide extra storage space for device 550 or may also store applications or other information for device 550. Specifically, expansion memory 568 may include instructions to carry out or supplement the processes described above and may include secure information also. Thus, for example, expansion memory 568 may be provided as a security module for device 550 and may be programmed with instructions that permit secure use of device 550. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner. In some embodiments, and by way of example only, the memory 554 may also include one or more of aVIS-004-W01 log of scanner use, calibration information, hardware and software versions, date of assembly, hardware configuration, date last calibrated, service history, and the like.

[0390] The memory may include, for example, flash memory and / or NVRAM memory, as discussed below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, cause performance of one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 554, expansion memory 568, or memory on processor 552 that may be received, for example, over transceiver 560 or external interface 566.

[0391] Device 550 may communicate wirelessly through communication interface 558, which may include digital signal processing circuitry where necessary. Communication interface 558 may provide for communications under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication may occur, for example, through radio-frequency transceiver 560. In addition, short-range communication may occur, such as using a Bluetooth, WiFi, or other such transceiver (not shown). In addition, GPS (Global Positioning System) receiver module 572 may provide additional navigation- and location-related wireless data to device 550, which may be used as appropriate by applications running on device 550.

[0392] Device 550 may also communicate audibly using for example audio codec 574, which may receive spoken information from a user and convert it to usable digital information. Audio codec 574 may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of device 550. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, etc.) and may also include sound generated by applications operating on device 550.

[0393] The computing device 550 may be implemented in a number of different forms, some of which are shown in the figure. For example, it may be implemented as a cellular telephone. It may also be implemented as part of a smart-phone, personal digital assistant, or other similar mobile device.VIS-004-W01

[0394] Additionally computing device 500 or 550 can include Universal Serial Bus (USB) flash drives. The USB flash drives may store operating systems and other applications. The USB flash drives can include input / output components, such as a wireless transmitter or USB connector that may be inserted into a USB port of another computing device.

[0395] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0396] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor.

[0397] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0398] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middlewareVIS-004-W01 component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), peer-to-peer networks (having ad-hoc or static members), grid computing infrastructures, and the Internet.

[0399] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0400] Additionally, by way of example, the computer system may include one or more specialized components such as GPU, field programmable gate array (FPGA), digital signal processor (DSP), neural processing unit (NPU), high speed camera interfaces, PCle, etc.

[0401] Definitions

[0402] As used herein, a “3D Scanner” is a device that captures three-dimensional (3D) topological data of the surgical field, including anatomical structures and surgical tools, in real time. While there are multiple types of 3D scanners, this disclosure focuses on a proprietary 3D structured light scanner. However, any 3D topological scanner capable of collecting similar data may be used, as this disclosure emphasizes the use of the scanner and its data rather than the emphasizing the scanner itself.

[0403] As used herein, “continuous anatomical automatic tracking (CAAT)” is a system feature that continuously updates the position and orientation of anatomical structures and instruments by collecting data with a 3D scanner. In some embodiments, this process eliminates the need for physical optical markers, offering real-time tracking throughout the surgical procedure.VIS-004-W01

[0404] As used herein, “gap balancing” is a technique used to ensure equal spacing (gaps) between bones in both flexion and extension during surgeries like Total Knee Arthroplasty (TKA). Accurate gap balancing helps maintain proper joint stability and alignment.

[0405] As used herein, a “hub” is a computer responsible for processing raw data from the 3D scanner and automatically registering it to the Surgical Coordinate System.

[0406] As used herein, an “iterative closest point (ICP)” is an algorithm that refines the registration process by minimizing the distance between corresponding points in two datasets for precise alignment.

[0407] As used herein, an “osteotomy” is a surgical procedure that involves cutting and reshaping bones, commonly used in orthopedic surgeries to correct deformities or prepare the bone for implants.

[0408] As used herein, a “point cloud” is a set of points in 3D space, each representing a specific location on a scanned surface, used to generate a topological map of the surgical field. By way of example, the 3D scanner described herein provides a particularly dense point cloud, offering high-resolution data that enhances precision in surgical procedures, and it does it at >25 rps, with high accuracy, optionally in the near-infrared, and with an advantageous illumination coverage and depth of field in the pattern projection.

[0409] As used herein, a “random sample consensus (RANSAC)” is an algorithm used for initial coarse registration by selecting key points in the dataset and filtering out outliers.

[0410] As used herein, “real-time tracking” is defined as continuous registration of anatomical structures or tools, updating multiple times per second to reflect movements and changes in position within the surgical field. This ensures that data is captured and updated in real time without noticeable delays.

[0411] As used herein, a “robot-mounted scanner” is a 3D scanner attached to the robotic arm, or to the end effector of a robotic arm that allows the scanner to dynamically track anatomical structures as the robot moves during surgery.VIS-004-W01

[0412] As used herein, “sphere fitting” is a method used to estimate the center of rotation of a joint, such as the hip, by fitting a mathematical sphere to points collected during motion tracking of the femur.

[0413] As used herein, a “structured light pattern” is a method of projecting a predefined light pattern (such as grids, stripes, or dots) onto surfaces in order to capture 3D topological data.

[0414] As used herein, a “user interface (UI)” is a visual interface that provides real-time visualizations and guides the surgeon through key steps in the procedure.

[0415] As used herein, the term “zero-reference tracking (ZRT)” refers to a method for tracking objects in real time without requiring pre-existing reference models, fiducial markers, or preoperative imaging. In some embodiments, ZRT generates reference data dynamically by scanning, segmenting, and registering objects entirely from intraoperative or real-time scan data.

[0416] As used herein, the term “reference model” refers to a dataset representing the shape and topology of an object, created from a scene scan, and used for tracking in subsequent scans.

[0417] As used herein, the term “segmentation” refers to a process of identifying and isolating one or more target objects from a scene scan, which may be performed manually, algorithmically, or through machine learning.

[0418] As used herein, the term “registration” refers to a process of aligning new scan data with a reference model to determine the object's position and orientation at a given time point.

[0419] As used herein, the term “multi-angle scanning” refers to using a 3D scanner to capture image data of an object from multiple viewpoints to improve accuracy, expand the field of view, and reduce occlusions.

[0420] As used herein, the term “kinematic extrapolation” refers to using position, velocity, and / or acceleration data to estimate an object's future location and orientation, reducing reliance on computationally intensive registration methods.

[0421] As used herein, the term "coupled" is defined as connected, although not necessarily directly, and not necessarily mechanically. The use of the word "a" or "an" when used inVIS-004-W01 conjunction with the term "comprising" in the claims and / or the specification may mean "one," but it is also consistent with the meaning of "one or more" or "at least one." The term "about" means, in general, the stated value plus or minus 5%. The use of the term "or" in the claims is used to mean "and / or" unless explicitly indicated to refer to alternatives only or the alternative are mutually exclusive, although the disclosure supports a definition that refers to only alternatives and "and / or."

[0422] As used herein, the terms "comprise" (and any form of comprise, such as "comprises" and "comprising"), "have" (and any form of have, such as "has" and "having"), "include" (and any form of include, such as "includes" and "including") and "contain" (and any form of contain, such as "contains" and "containing") are open-ended linking verbs. As a result, a method or device that "comprises," "has," "includes" or "contains" one or more steps or elements, possesses those one or more steps or elements, but is not limited to possessing only those one or more elements. Likewise, a step of a method or an element of a device that "comprises," "has," "includes" or "contains" one or more features, possesses those one or more features, but is not limited to possessing only those one or more features. Furthermore, a device or structure that is configured in a certain way is configured in at least that way but may also be configured in ways that are not listed.

[0423] All patents and publications mentioned in this specification are indicative of the levels of those skilled in the art to which the disclosure pertains. It is to be understood that while a certain form of the disclosure is illustrated, it is not to be limited to the specific form or arrangement herein described and shown. It will be apparent to those skilled in the art that various changes may be made without departing from the scope of the disclosure and the disclosure is not to be considered limited to what is shown and described in the specification and any drawings / figures included herein.

[0424] One skilled in the art will readily appreciate that the present disclosure is well adapted to carry out the objectives and obtain the ends and advantages mentioned, as well as those inherent therein. The embodiments, methods, procedures and techniques described herein are presently representative of the preferred embodiments, are intended to be exemplary and are not intended as limitations on the scope. Changes therein and other uses will occur to those skilledVIS-004-W01 in the art which are encompassed within the spirit of the disclosure and are defined by the scope of the appended claims. Although the disclosure has been described in connection with specific preferred embodiments, it should be understood that the disclosure as claimed should not be unduly limited to such specific embodiments. Indeed, various modifications of the described modes for carrying out the disclosure which are obvious to those skilled in the art are intended to be within the scope of the following claims. too

Claims

VIS-004-W01CLAIMSWhat is claimed is:

1. A system for continuous registration and tracking of objects in an operating room environment, comprising: a structured light three-dimensional scanner, including: a light source; a liquid crystal matrix; and an image capture device; a computer system including a memory and at least one processor; and computer readable media embodied in a non-transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a three-dimensional data representation a scene within a field of view of the image capture device using a first dataset, the first dataset derived from one or more initial images captured by the image capture device without optical trackers or applied fiducials, the scene including one or more physical objects; isolate a unique target object from the one or more physical objects within the scene by segmentation of a target reference dataset from the first dataset; register the target reference dataset to a second dataset such that the target reference dataset and the second dataset are aligned within a common coordinate system, the second dataset derived from one or more subsequent images captured by the image capture device without said optical trackers or applied fiducials; and track the unique target object in six degrees of freedom relative to the common coordinate system.

2. The system of claim 1, wherein the light source is a linear light source.

3. The system of claim 1, wherein the three-dimensional data representation is a point cloud or three-dimensional data convertible into a point cloud.

4. The system of claim 1, wherein the one or more initial captured images and the one orVIS-004-W01 more subsequent captured images are structured light patterned images that encode the surface of the physical object.

5. The system of claim 1, wherein the three-dimensional structured light scanner further includes a heat reduction assembly.

6. The system of claim 1, wherein the target object is an anatomical structure.

7. The system of claim 1, wherein the target object is a surgical instrument.

8. The system of claim 1, wherein the three-dimensional data representation of the topological surface of the scene is constructed by: decoding light patterns contained in the one or more initial captured images by analyzing brightness history to create camera pixel correspondence to light angle projection; using triangulation to calculate a three-dimensional coordinate for each pixel in the one or more initial captured images based on the decoded light patterns; and combining the triangulated three-dimensional coordinates of each pixel into the three-dimensional point cloud representation of the surface of the scene in physical space.

9. The system of claim 1, wherein the computer readable media includes a further set of instructions, that, when executed by a processor, cause the computer system to: isolate a second unique target object from the one or more physical objects within the scene by segmentation of a second target reference dataset from the first dataset. register the second target reference dataset to a third dataset such that the second target reference dataset and the third dataset are aligned within the common coordinate system, the third dataset derived from the one or more subsequent images captured by the image capture device; and track the second unique target object relative to the common coordinate system.

10. The system of claim 9, wherein the unique target object is a first anatomical structure and the second unique target object is a second anatomical structure.

11. The system of claim 9, wherein the unique target object is an anatomical structure and theVIS-004-W01 second unique target object is a surgical instrument.

12. The system of claim 1, wherein: the computer system further comprises a user interface and a display device, and the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display the tracked three- dimensional data representation of the unique target object on the display device.

13. The system of claim 1, wherein the first dataset is captured by the image capture device from multiple different angles or viewpoints.

14. The system of claim 1, wherein the segmentation of a target reference dataset from the first dataset comprises at least one of manual selection, algorithmic detection, machine learning, or via registration of objects of known shapes.

15. The system of claim 1, wherein the six degrees of freedom comprise x-coordinate position, y-coordinate position, z-coordinate position, roll orientation, pitch orientation, and yaw orientation.

16. The system of claim 1 , wherein tracking the unique target object relative to the common coordinate system comprises continuously updating the unique target object’s position relative to the common coordinate system by repeatedly acquiring surface data, registering the repeatedly acquired surface data in the common coordinate system, and updating transformation matrices.

17. The system of claim 16, wherein the computer readable media includes a further set of instructions, that, when executed by a processor, cause the computer system to: compute at least one of velocity, angular velocity, acceleration, and angular acceleration of the target object within the common coordinate system; and extrapolate the values of the computed at least one of velocity, angular velocity, acceleration, and angular acceleration of the target object within the common coordinate system; and estimate at least one of a future x-coordinate position, a future y-coordinate position, a future z-coordinate position, a future roll orientation, a future pitch orientation,VIS-004-W01 and a future yaw orientation of the target object based on the extrapolated values.

18. A system for continuous registration and tracking of objects in an operating room environment, comprising: a structured light three-dimensional scanner, including: a light source: a liquid crystal matrix; and an image capture device; a computer system including a memory and at least one processor; and computer readable media embodied in a non-transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a three-dimensional data representation a scene within a field of view of the image capture device using a first dataset, the first dataset derived from one or more initial images captured by the image capture device without optical trackers or applied fiducials, the scene including one or more physical objects; isolate a unique target object from the one or more physical objects within the scene by segmentation of a target reference dataset from the first dataset; register the target reference dataset to a second dataset such that the target reference dataset and the second dataset are aligned within a common coordinate system, the second dataset derived from one or more subsequent images captured by the image capture device without said optical trackers or applied fiducials; track the unique target object in six degrees of freedom relative to the common coordinate system by continuously updating the unique target object’s position relative to the common coordinate system by repeatedly acquiring surface data, registering the repeatedly acquired surface data in the common coordinate system, and updating transformation matrices; compute at least one of velocity, angular velocity, acceleration, and angular acceleration of the target object within the common coordinate system; and extrapolate the values of the computed at least one of velocity, angularVIS-004-W01 velocity, acceleration, and angular acceleration of the target object within the common coordinate system; and estimate at least one of a future x-coordinate position, a future y- coordinate position, a future z-coordinate position, a future roll orientation, a future pitch orientation, and a future yaw orientation of the target object based on the extrapolated values.

19. The system of claim 18, wherein the light source is a linear light source.

20. The system of claim 18, wherein the three-dimensional data representation is a point cloud or three-dimensional data convertible into a point cloud.

21. The system of claim 18, wherein the one or more initial captured images and the one or more subsequent captured images are structured light patterned images that encode the surface of the physical object.

22. The system of claim 18, wherein the three-dimensional structured light scanner further includes a heat reduction assembly.

23. The system of claim 18, wherein the target object is an anatomical structure.

24. The system of claim 18, wherein the target object is a surgical instrument.

25. The system of claim 18, wherein the three-dimensional data representation of the topological surface of the scene is constructed by: decoding light patterns contained in the one or more initial captured images by analyzing brightness history to create camera pixel correspondence to light angle projection; using triangulation to calculate a three-dimensional coordinate for each pixel in the one or more initial captured images based on the decoded light patterns; and combining the triangulated three-dimensional coordinates of each pixel into the three-dimensional point cloud representation of the surface of the scene in physical space.

26. The system of claim 18, wherein the computer readable media includes a further set of instructions, that, when executed by a processor, cause the computer system to:VIS-004-W01 isolate a second unique target object from the one or more physical objects within the scene by segmentation of a second target reference dataset from the first dataset; register the second target reference dataset to a third dataset such that the second target reference dataset and the third dataset are aligned within the common coordinate system, the third dataset derived from the one or more subsequent images captured by the image capture device; and track the second unique target object relative to the common coordinate system.

27. The system of claim 26, wherein the unique target object is a first anatomical structure and the second unique target object is a second anatomical structure.

28. The system of claim 26, wherein the unique target object is an anatomical structure and the second unique target object is a surgical instrument.

29. The system of claim 18, wherein: the computer system further comprises a user interface and a display device, and the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display the tracked three- dimensional data representation of the unique target object on the display device.

30. The system of claim 18, wherein the first dataset is captured by the image capture device from multiple different angles or viewpoints.

31. The system of claim 18, wherein the segmentation of a target reference dataset from the first dataset comprises at least one of manual selection, algorithmic detection, machine learning, or via registration of objects of known shapes.

32. The system of claim 18, wherein the six degrees of freedom comprise x-coordinate position, y-coordinate position, z-coordinate position, roll orientation, pitch orientation, and yaw orientation.

33. A system for identification and selection of an optimal surgical implant in an operating room environment, comprising: a structured light three-dimensional scanner, including:VIS-004-W01 a light source: a liquid crystal matrix; and an image capture device; a computer system including a memory and at least one processor; and computer readable media embodied in a non-transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: generate a real-time anatomical model of a surgical site that serves as a surface map of one or more articular surfaces of a patient by: constructing a three-dimensional data representation of a topological articular surface of a physical object using a first dataset, the first dataset derived from one or more images captured by the image capture device without optical trackers or applied fiducials; registering the three-dimensional data representation to a second dataset such that the three-dimensional data representation and the second dataset are aligned within a common coordinate system, the registration occurring without said optical trackers or applied fiducials; and tracking the physical object relative to the common coordinate system without said optical trackers or applied fiducials; individually register the constructed three-dimensional data representation to one or more implant models within a library of implant models to create a set of implant model registrations, each of the implant models within the library of implant models representing a unique physical implant configured to engage the articular surface of the physical object; evaluate the quality of each implant model registration in the set of implant model registrations using a computational fitness function; and identify a best-fitting implant based on the evaluation of the quality of each implant model registration in the set of implant model registrations.

34. The system of claim 33, wherein the light source is a linear light source.

35. The system of claim 33, wherein the three-dimensional data representation is a pointVIS-004-W01 cloud or three-dimensional data convertible into a point cloud.

36. The system of claim 33, wherein the captured images are structured light patterned images that encode the surface of the physical object.

37. The system of claim 33, wherein the physical object is an anatomical structure in a surgical procedure.

38. The system of claim 33, wherein the three-dimensional data representation of the topological articular surface of the physical object is constructed by: decoding light patterns contained in the one or more captured images by analyzing brightness history to create camera pixel correspondence to light angle projection; using triangulation to calculate a three-dimensional coordinate for each pixel in the one or more captured images based on the decoded light patterns; and combining the triangulated three-dimensional coordinates of each pixel into the three-dimensional point cloud representation of the surface of the physical object in physical space.

39. The system of claim 33, wherein the three-dimensional data representation of the physical object comprises a first three-dimensional data representation of a first physical object, and the computer readable media includes a further set of instructions, that, when executed by a processor, cause the computer system to: construct a second digital three-dimensional data representation of a topological surface of a second physical object using a third dataset, the third dataset derived from one or more images of surface topology of the second physical object captured by the image capture device without the use of optical trackers or applied fiducials; register the second three-dimensional data representation to a fourth dataset such that the second three-dimensional data representation and the fourth dataset are aligned with one another and the with the first three-dimensional data representation and second dataset within the common coordinate system without the use of optical trackers or applied fiducials; continuously track the second physical object relative to the common coordinateVIS-004-W01 system without the use of optical trackers or applied fiducials.

40. The system of claim 39, wherein the first physical object is a first anatomical structure and the second physical object is a second anatomical structure.

41. The system of claim 39, wherein the first physical object is an anatomical structure and the second physical object is a surgical instrument.

42. The system of claim 33, wherein the second dataset is derived from a preoperative computed tomography scan or magnetic resonance imaging scan.

43. The system of claim 33, wherein: the computer system further comprises a user interface and a display device, and the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display, on a display device, the tracked three-dimensional data representation of the physical object.

44. The system of claim 33, wherein tracking the physical object relative to the common coordinate system comprises continuously updating the physical object’s position relative to the common coordinate system by repeatedly acquiring surface data, registering the repeatedly acquired surface data in the common coordinate system, and updating transformation matrices.

45. The system of claim 33, wherein the fitness function comprises one of least-squares matching, curvature alignment, surface area overlap, multi-factor scoring, fractional area overlap, and stability of registration in six degrees of freedom.

46. The system of claim 33, wherein the computer readable media includes a further set of instructions, that, when executed by a processor, cause the computer system to: dynamically adjust implant positioning by correcting for changes in cartilage thickness.

47. The system of claim 46, wherein the implant positioning is manually adjusted by a surgeon during a surgical planning event based on the surgeon’s knowledge and expertise in accounting for disease-related changes in bone shape and cartilage thickness gainedVIS-004-W01 through experience.

48. The system of claim 46, wherein the implant positioning is adjusted automatically based on intraoperative measurements taken with a registered probe instrument.

49. The system of claim 46, wherein the implant positioning is adjusted automatically based on estimated typical cartilage thickness derived from published statistical data.

50. The system of claim 46, wherein the implant positioning is adjusted automatically based on a computation of cartilage thickness derived from the registration of the first dataset to the second dataset, wherein the first dataset comprises cartilage and bone surface data of the physical object.

51. The system of claim 46, wherein the implant positioning is adjusted automatically based on tissue-specific responses to scanner-derived tissue differentiation to estimate cartilage thickness.

52. The system of claim 46, wherein the implant positioning is manually adjusted by a surgeon during a surgical planning event based on the surgeon’s knowledge and expertise in accounting for disease-related changes in bone shape and cartilage thickness gained through experience and intraoperative measurements taken with a registered probe instrument.

53. A system for identification and selection of an optimal surgical implant in an operating room environment, comprising: a structured light three-dimensional scanner, including: a light source: a liquid crystal matrix; and an image capture device; a computer system including a memory and at least one processor; and computer readable media embodied in a non-transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: generate a real-time anatomical model of a surgical site that serves as aVIS-004-W01 surface map of one or more articular surfaces of a patient by: constructing a three-dimensional data representation of a topological articular surface of a physical object using a first dataset, the first dataset derived from one or more images captured by the image capture device without optical trackers or applied fiducials; registering the three-dimensional data representation to a second dataset such that the three-dimensional data representation and the second dataset are aligned within a common coordinate system, the registration occurring without said optical trackers or applied fiducials; and tracking the physical object relative to the common coordinate system without said optical trackers or applied fiducials; individually register the constructed three-dimensional data representation to one or more implant models within a library of implant models to create a set of implant model registrations, each of the implant models within the library of implant models representing a unique physical implant configured to engage the articular surface of the physical object; evaluate the quality of each implant model registration in the set of implant model registrations using a computational fitness function; identify a best-fitting implant based on the evaluation of the quality of each implant model registration in the set of implant model registrations; dynamically adjust implant positioning by correcting for changes in cartilage thickness; and after dynamically adjusting implant positioning by correcting for changes in cartilage thickness, individually re-register the constructed three-dimensional data representation to two or more implant models within the library of implant models to create a set of implant model re-registrations; evaluate the quality of each implant model re-registration in the set of implant model re-registrations using the computational fitness function; and identify or confirm the best-fitting implant based on the evaluation of the quality of each implant model re-registration in the set of implant model reregistrations.VIS-004-W0154. The system of claim 53, wherein the light source is a linear light source.

55. The system of claim 53, wherein the three-dimensional data representation is a point cloud or three-dimensional data convertible into a point cloud.

56. The system of claim 53, wherein the captured images are structured light patterned images that encode the surface of the physical object.

57. The system of claim 53, wherein the physical object is an anatomical structure in a surgical procedure.

58. The system of claim 53, wherein the three-dimensional data representation of the topological articular surface of the physical object is constructed by: decoding light patterns contained in the one or more captured images by analyzing brightness history to create camera pixel correspondence to light angle projection; using triangulation to calculate a three-dimensional coordinate for each pixel in the one or more captured images based on the decoded light patterns; and combining the triangulated three-dimensional coordinates of each pixel into the three-dimensional point cloud representation of the surface of the physical object in physical space.

59. The system of claim 53, wherein the three-dimensional data representation of the physical object comprises a first three-dimensional data representation of a first physical object, and the computer readable media includes a further set of instructions, that, when executed by a processor, cause the computer system to: construct a second digital three-dimensional data representation of a topological surface of a second physical object using a third dataset, the third dataset derived from one or more images of surface topology of the second physical object captured by the image capture device without the use of optical trackers or applied fiducials; register the second three-dimensional data representation to a fourth dataset such that the second three-dimensional data representation and the fourth dataset are aligned with one another and the with the first three-dimensional data representation and secondVIS-004-W01 dataset within the common coordinate system without the use of optical trackers or applied fiducials; continuously track the second physical object relative to the common coordinate system without the use of optical trackers or applied fiducials.

60. The system of claim 59, wherein the first physical object is a first anatomical structure and the second physical object is a second anatomical structure.

61. The system of claim 59, wherein the first physical object is an anatomical structure and the second physical object is a surgical instrument.

62. The system of claim 53, wherein the second dataset is derived from a preoperative computed tomography scan or magnetic resonance imaging scan.

63. The system of claim 53, wherein: the computer system further comprises a user interface and a display device, and the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display, on a display device, the tracked three-dimensional data representation of the physical object.

64. The system of claim 53, wherein tracking the physical object relative to the common coordinate system comprises continuously updating the physical object’s position relative to the common coordinate system by repeatedly acquiring surface data, registering the repeatedly acquired surface data in the common coordinate system, and updating transformation matrices.

65. The system of claim 53, wherein the fitness function comprises one of least-squares matching, curvature alignment, surface area overlap, multi-factor scoring, fractional area overlap, and stability of registration in six degrees of freedom.

66. The system of claim 53, wherein the implant positioning is manually adjusted by a surgeon during a surgical planning event based on the surgeon’s knowledge and expertise in accounting for disease-related changes in bone shape and cartilage thickness gained through experience.VIS-004-W0167. The system of claim 53, wherein the implant positioning is adjusted automatically based on intraoperative measurements taken with a registered probe instrument.

68. The system of claim 53, wherein the implant positioning is adjusted automatically based on estimated typical cartilage thickness derived from published statistical data.

69. The system of claim 53, wherein the implant positioning is adjusted automatically using based on tissue-specific responses to scanner-derived tissue differentiation to estimate cartilage thickness.

70. The system of claim 53, wherein the implant positioning is adjusted automatically based on a computation of cartilage thickness derived from the registration of the first dataset to the second dataset, wherein the first dataset comprises cartilage and bone surface data of the physical object.

71. The system of claim 53, wherein the implant positioning is manually adjusted by a surgeon during a surgical planning event based on the surgeon’s knowledge and expertise in accounting for disease-related changes in bone shape and cartilage thickness gained through experience and intraoperative measurements taken with a registered probe instrument.

72. A method of updating an anatomic reference model used in executing a surgical plan during a surgical event, comprising the steps of: generating an original anatomic reference model representing a first anatomic state of one or more objects within a surgical site; registering and tracking the one or more objects within a common coordinate system; performing an action resulting in a first changed anatomic state of the one or more tracked objects in the surgical site; acquiring subsequent scan data of the surgical site using a structured light 3D scanner; updating the original reference model with the acquired scan data to generate a first updated reference model representing a second anatomic state of the one or moreVIS-004-W01 objects within the surgical site; performing a subsequent action resulting in second changed anatomic state of the one or more tracked objects in the surgical site; acquiring further subsequent scan data of the surgical site using the structured light 3D scanner; updating the first updated reference model with the acquired further subsequent scan data to generate a second updated reference model representing a third anatomic state of the one or more objects within the surgical site; and repeating the scan data capture and reference model updating cycle until the surgical procedure is concluded such that the surgical plan is carried out with a real-time updated reference model before each cut is made.

Images