Surgical navigation system and related methods

Abstract

A system and method for registering and tracking anatomical structures during surgical procedures by leveraging continuously updated 3D topological data of objects within the surgical field. Registration allows different datasets such as preoperative CT scans and intraoperative 3D scans to be aligned, combined, and transformed into a common coordinate space, enabling more precise surgical planning and execution. Tracking involves continuously registering these datasets over time, monitoring changes in position or orientation, and keeping objects aligned within the common coordinate space, updating the spatial alignment of registered objects multiple times per second, which allows for dynamic monitoring of anatomical structures and surgical tools as they move throughout the procedure. Unlike traditional systems that rely on physical optical markers (OMs) and optical tracker constellations (OTCs), the system and method disclosed herein utilizes the data captured by a 3D scanner to perform Continuous Anatomic Auto Tracking (CAAT).

Description

VIS-007-W01Surgical Navigation System and Related MethodsCROSS-REFERENCES TO RELATED APPLICATIONS

[0001] The present application claims the benefit of priority from U.S. Provisional Application Serial No. 63 / 716,932, filed November 6, 2024 and entitled “Registration & Tracking by 3D Scanning the Topology of Anatomic Structures”, U.S. Provisional Application Serial No. 63 / 717,108, filed November 6, 2024 and entitled “Joint Laxity and Gap Measurement with Topological Data”, U.S. Provisional Application Serial No. 63 / 717,118, filed November 6, 2024 and entitled “Calculation of Hip Center of Rotation Using Dynamic Topological Data”, U.S. Provisional Application Serial No. 63 / 717,129, filed November 6, 2024 and entitled “Instrument-Based Anatomical Point Location Using Topology Registration without Optical Trackers”, U.S. Provisional Application Serial No. 63 / 717,185, filed November 6, 2024 and entitled “Anatomical Verifications Using Topological Data Instead of Optical Tracking Systems”, and U.S. Provisional Application Serial No. 63 / 844,018, filed July 14, 2025 and entitled “3D Scanner”, the entire contents of which are hereby incorporated by reference into this disclosure as if set forth fully herein.FIELD

[0002] The present disclosure relates generally to surgical navigation, and more specifically to using intraoperative topological data acquired from three-dimensional (3D) scanning without optical trackers or applied fiducials to intraoperatively modify a pre-operative surgical plan.BACKGROUND

[0003] In modern surgical practice, particularly in orthopedic procedures like Total Knee Arthroplasty (TKA) and Hip Arthroplasty, precise alignment and tracking of anatomical structures are crucial for successful outcomes. Over the past two decades, surgical navigation systems have become standard tools for enhancing accuracy during these procedures. These systems typically rely on optical trackers and stereoscopic cameras to guide surgeons through intricate surgical steps, ensuring correct positioning of instruments and implants.

[0004] By way of example, an optical tracker (OT) is a system used to track the position of anatomical structures, instruments, or equipment in the surgical field. An optical tracker isVIS-007-W01 composed of a collection of optical markers (OMs) arranged in a specific geometric pattern, or a "constellation." These constellations allow the system to precisely identify and differentiate between tracked objects based on the unique arrangement of their optical markers.

[0005] By way of example, an optical marker (OM) is a marker used in traditional surgical navigation systems to track the position of anatomical structures and tools in real-time via stereoscopic cameras. One common example is a reflective sphere, though multiple types of optical markers can be used depending on the system and application.

[0006] However, despite their effectiveness, these systems present several challenges. The setup of optical trackers is often time-consuming and technically complex, requiring precise calibration and positioning of cameras and tracking equipment. Additionally, the physical nature of optical trackers introduces mechanical vulnerabilities — trackers can become dislodged, damaged, or obstructed during surgery (e.g., affecting line of sight between the camera and the OTs), leading to potential inaccuracies in tracking and necessitating recalibration. These issues add to overall procedural time, which may increase patient exposure to anesthesia and complicate workflow.

[0007] Moreover, the reliance on optical trackers and complex equipment adds significant costs and requires highly trained personnel to operate the systems. These challenges have limited the widespread adoption of such systems, particularly in smaller or resource-constrained healthcare facilities. As a result, there is a pressing need for more robust and efficient solutions that maintain or improve the accuracy of surgical navigation while reducing complexity and cost.

[0008] The following disclosure generally describes methods currently used in the field of surgical navigation, for example a navigation system using optical markers and stereoscopic cameras. By way of example only, the methods discussed and / or disclosed herein are done so in the context of a Total Knee Arthroplasty (TKA), however it should be understood that the systems and methods disclosed herein may be used in any scenario (surgical, medical, or otherwise) to generate a reconstructed 3D model of an object.

[0009] Optical markers (OMs) are small reflective markers that are easily segmented from other elements in an image. These markers retro-reflect infrared light emitted by stereoscopicVIS-007-W01 cameras, making them highly detectable to cameras that are located close to the near infrared (NIR) source. OMs are essential for tracking anatomical landmarks, surgical instruments, and equipment during surgery, ensuring reliable visibility even in complex surgical environments.

[0010] Optical markers (OMs) are often mechanically grouped into specific rigid geometric patterns, known as optical trackers (OTs). These unique arrangements allow the system to differentiate between various anatomical structures, instruments, or equipment being tracked. By way of example, OTs can be attached to patient anatomy (e.g., bones) near the surgical site, mounted on surgical instruments, (e g., pointers or measurement tools), and / or fixed to active equipment (e.g., robotic arms, saws, or drills). The distinct geometries of OTs enable the system to identify what each OM represents within the surgical field.

[0011] Stereoscopic cameras are foundational to spatial tracking in surgical navigation systems. These cameras capture images from slightly different angles, similar to human binocular vision. By detecting the reflective OMs in the surgical field, the cameras calculate the 3D positions of all visible optical trackers (OTs).

[0012] Typically, the navigation system compares the relative positions of the detected OMs to the predefined geometric pattern of each OT. Once an OT is identified, it is assigned to its corresponding surgical tool, anatomical structure, or piece of equipment. The identified OTs are then registered in a common coordinate system, or space, providing a comprehensive 3D representation of the surgical field. The system can now track the position and orientation of all OTs in real time.

[0013] Once the positions of the OTs are known in the common coordinate system, they are transformed into a Surgical Coordinate System (SCS), which provides a fixed reference frame for the procedure. The SCS could be equivalent to camera space, CT space, tracker space, or based on another tracked location, such as the base of a robot. By applying a 4x4 rigid transformation matrix 2 (see Figure 1), any OT can be positioned within the SCS, ensuring consistency in tracking across the system. To accurately transform the object into the SCS, a sequence of transformations may be applied. The process involves transformations between, for example, an object’s local space, OT space, camera space, and a final SCS, which is a chosen common space for all relevant items for navigation.VIS-007-W01

[0014] By applying these transformations, the system ensures precise positioning of objects within the SCS, providing consistent and accurate tracking across the surgical environment. It should be noted that each transformation matrix introduces error into the equation. Because of this, the more transformations that are required to move an object into the SCS, the greater the potential for error. For example, in the equation presented above, three transformations are required.

[0015] For surgical navigation systems to be effective, they must track rigidly attached OTs in real-time, updating positions multiple times per second. This continuous updating enables precise monitoring of tool positions, implant alignment, and anatomical movement, ensuring accurate and timely adjustments throughout the surgery.

[0016] Locating anatomy poses unique challenges because the position of the Optical Tracker (OT) relative to the anatomy is not always well defined. When an OT is attached to an instrument, its location relative to the rest of the instrument is precisely engineered and known. However, with anatomical structures, the initial placement of an OT follows general guidelines but is somewhat random within those parameters.

[0017] Another reason this anatomical registration can be difficult is the attachment method. For instruments, the OT can be attached rigidly with screws, ensuring stability and precision. However, for anatomical structures, the attachment must be created during surgery. For TKA, this typically involves drilling two pins bi-cortically (through both sides of the bone) and then attaching the OT to the pins. These pins are subject to flex due to the bone's characteristics, and the bone quality may not be ideal, further complicating the attachment. The OT must always be “visible” to the stereo camera and not be obscured by surgeons, OR staff, instruments, drapes or other equipment to function properly. Additionally, the OT may be bumped or deflected during the procedure by the surgeon or other equipment, leading to inaccuracies in tracking.

[0018] OTs are securely attached to bony landmarks using these pins or clamps. For instance, in procedures such as Total Knee Arthroplasty (TKA), OTs might be placed on the femur and tibia. These OTs serve as proxies for the position of the underlying anatomy, allowing for continuous tracking throughout the procedure.VIS-007-W01

[0019] Before the OTs attached to anatomy can be used to locate the anatomical structure within the SCS, the anatomy must first be registered. This registration process typically involves identifying key anatomical landmarks, such as specific points on the bone, and matching them with pre-operative imaging (CT or MRI scans). The system then uses the positions of the attached OTs to align the anatomy with the pre-operative 3D model.

[0020] Surgeons often use a pointer or probe (equipped with its own OT) to touch specific anatomical points, the tip-points of which are then recorded by the system. These points help to map (register) the patient’s anatomy with preoperative imaging or a model, allowing for accurate tracking during surgery. However, because the pointer introduces another OT and another transformation matrix into the sequence of transformations required to locate anatomy in the SCS, potential for error is increased.

[0021] Before surgery, detailed imaging of the target anatomy is often obtained using CT (Computed Tomography) or MRI (Magnetic Resonance Imaging). These scans create a 3D model of the anatomy, which serves as a reference during the procedure. This model is crucial for planning surgical interventions and aligning the patient’s real-time anatomy with the preoperative plan.

[0022] However, some systems and robotic surgical systems do not rely on pre-operative imaging. In these cases, a standard 3D model of bones or other anatomy is used during the planning process. These standard models have landmarks identified within them, which can be used to properly locate the standard model. To make the model more visually realistic in surgical planning applications, the standard models can be scaled, deformed, and / or sheared using an “affine” transformation matrix if needed. By way of example, comparing the landmark data in the CT and the landmark data obtained using a pointer, this affine transformation matrix can be computed. For instance, if an OT attached to a pointer is used to locate the medial and lateral epicondyles of the femur, the real distance between these two points could be calculated and used to scale the standard model. In this way, an appropriately sized model can be used to visualize the placement of various implants in proper scale, though it may not be anatomically accurate for that patient. Other transformation matrices noted herein that are for registration or tracking are rigid transformations and do not include scaling, deformation or shearing effects.VIS-007-W01

[0023] In both cases, whether pre-operative images are available or not, the process is essentially similar and will be treated as such for this disclosure. If pre-operative data, CT data, or similar data is referenced, it is possible that a scaled, deformed, and / or sheared standard model could be used in its place.

[0024] Example Workflow of Optical Tracking Systems for TKA (Prior Art)

[0025] All Total Knee Arthroplasty (TKA) surgeries are different. The patients are different, and the surgeons who perform the procedures have varying training, experience, and methodologies that shape their approach to surgery and their definition of an ideal TKA. There are also many different surgical navigation systems, robotic assistants, and implant systems to choose from. Because of these differences, a standard TKA workflow cannot be universally defined. The procedure that follows is an example of a typical prior art system. By way of example this workflow is represented in Figure 2.

[0026] Prior to surgery, detailed imaging of the target anatomy is often acquired using CT (Computed Tomography) or MRI (Magnetic Resonance Imaging). These scans are segmented to create a 3D model of the relevant structures, such as the femur and tibia in TKA. These 3D models are essential for planning the procedure and aligning the patient's real-time anatomy with the surgical plan. Important anatomical planes and landmarks are also labeled within the model to assist the surgeon during the operation.

[0027] In some systems, pre-operative imaging is not required. This is referred to as "imageless" mode. In this approach, a standard 3D model of the bones is used instead of patientspecific scans. Anatomical landmarks, such as the medial and lateral epicondyles of the femur, are identified during surgery using OTs and a pointer. The distance between these landmarks is used to scale the standard model to match the patient’s unique anatomy, allowing the surgeon to visualize implant placement with appropriate proportions, even in the absence of patient-specific imaging. Simple scaling or uniform scaling may be used but it is likely that more advanced scaling is used where the scaling factor in various directions is different. There may also be some twisting, deformation and / or shearing required to make the landmarks line up best.VIS-007-W01

[0028] For this workflow, we assume a patient-specific CT scan has been segmented to provide a detailed 3D model of the femur and tibia, though the imageless mode can also be utilized if necessary.

[0029] Before surgery begins, the system requires calibration. Stereoscopic cameras are positioned to ensure full visibility of the surgical field, and OTs may be attached to the surgical instruments, the robotic arm, and the robotic base. Calibration ensures that the system aligns correctly with the surgical setup, accounting for distances, camera angles, and the positioning of all equipment.

[0030] Robotic equipment requires additional steps for calibration to ensure that the robot functions correctly. This process includes aligning the robot's physical movements with the virtual model of the surgery to maintain accuracy and ensure that the robot performs as expected during the procedure.

[0031] After setup and calibration, the patient is positioned on the operating table, typically in a supine position with the knee exposed for TKA procedures. The surgical site is prepped and draped in a sterile fashion. Surgical exposure follows, with an incision made to access the knee joint and prepare it for the robotic system to perform the procedure.

[0032] Patient registration starts when OTs are attached to the patient’s anatomy, typically using two pins driven bi-cortically. For femoral registration, the camera system tracks both the OT attached to the femur and the OT attached to a pointer. The system’s user interface guides the surgeon through the process of identifying specific anatomical points on the femur. During this process, the surgeon collects approximately 50 or more points on the surface of the femur, though the number may vary depending on the system.

[0033] These points generate a sparse point cloud representing the surface of the femur. The point cloud is registered to pre-operative imaging (CT, MRI scans, or scaled standard models) using algorithms like RANSAC (Random Sample Consensus) and ICP (Iterative Closest Point). Once registration is complete, the system knows how the pre-operative imaging data must be transformed to align properly in camera space (and vice versa). With trackers also captured in camera space, and their registration to their respective OT virtual spaces, the real-timeVIS-007-W01 anatomical model can then be transformed and maintained within the chosen Surgical Coordinate System (SCS), enabling accurate tracking of the patient’s anatomy throughout the procedure.

[0034] Following patient registration, the surgeon identifies key anatomical landmarks necessary for establishing the mechanical axis of the leg, for example including the hip center (e.g. identified by rotating the leg while the optical tracking system calculates the center of rotation of the femoral head), knee center (e.g., identified either from pre-operative imaging or by locating the medial and lateral epicondyles of the femur), and / or ankle center (e.g., identified using the optical tracking system on the medial and lateral malleoli (ankle bones) to calculate the center of the ankle joint). These landmarks are crucial for ensuring proper implant alignment. The mechanical axis is one common methodology, but other alignment methods, like kinematic alignment, may use different anatomical landmarks depending on the procedure.

[0035] At this stage, a virtual surgical plan is developed (e.g., by the surgeon based on their experience and surgical philosophy) to guide the robotic system and surgeon throughout the procedure. If pre-operative planning was done, it serves as an initial basis but must be verified and perhaps adjusted in real time as the surgery progresses.

[0036] The virtual plan typically involves the following steps, which may be applied in a different order or iteratively, depending on the system:

[0037] 1. Implant Size Selection. The system assists the surgeon in selecting the optimal implant size based on the patient’s anatomy. Proper implant sizing ensures both the long-term success of the surgery and post-operative joint function.

[0038] 2. Implant Positioning. The system helps the surgeon determine the ideal implant position to achieve accurate joint alignment. The goal is to ensure proper leg alignment and balance between the medial and lateral sides of the knee. The alignment is measured in terms of varus (inward tilt) and valgus (outward tilt), with the system assisting the surgeon in making appropriate adjustments.

[0039] 3. Gap Balancing. By way of example, some surgeons may assess joint laxity by measuring gaps on the medial and lateral sides of the knee at various flexion angles, using this information to guide implant positioning in a process called “gap balancing.” Gap balancingVIS-007-W01 strives for equal tension in the soft tissues (primarily ligaments) around the knee joint throughout its range of motion. The system dynamically tracks the relative positions of the bones and calculates the space between them to guide the surgeon in balancing the medial and lateral gaps during the procedure. However, it should be noted that surgical techniques and philosophies can vary significantly, and some surgeons omit this step entirely if they do not subscribe to a gap balancing philosophy.

[0040] 4. Osteotomy Planning. Based on the selected implant size and position, the system defines the necessary osteotomies (bone cuts). These cuts must be precise to ensure proper implant placement. By way of example, the necessary bone cuts are determined not just by general planning but specifically by the selected implant size and its final planned position, which are derived from the geometry and design specifics of the chosen implant system.

[0041] 5. Rob otic- Assisted Bone Resection. During bone resection, the robotic system assists the surgeon in making precise cuts according to the surgical plan. By way of example, the robotic system assists the surgeon by maintaining the saw blade within the correct cutting plane, while still allowing the surgeon to control when the saw is activated and to move the saw freely within that defined plane as needed during the procedure. The system continuously tracks the patient's anatomy and provides feedback, ensuring the cuts are executed accurately.

[0042] 6. Intraoperative Verification and Adjustment. After each bone cut, the accuracy of the cut is verified. By way of example, the surgeon typically performs this cut verification by using an instrument specifically designed for this purpose, which is equipped with its own OT, to manually check the accuracy of the cut. If deviations are detected between the actual cuts and the surgical plan, the surgeon may need to adjust the plan to correct for any differences.

[0043] 7. Trialing and Final Implantation. After bone cuts are verified, trial implants are placed to assess the fit, alignment, and joint movement. If necessary, adjustments are made before the final implant is inserted.

[0044] 8. Post-Operative Evaluation and Closure. The surgeon performs a final assessment to ensure proper alignment and stability of the new joint. Once confirmed, the surgical site is closed, and the patient is prepared for recovery.VIS-007-W01

[0045] Despite their widespread use, stereoscopic camera and optical tracking systems present several challenges that affect their efficiency and accuracy in surgical applications.

[0046] For example, one challenge of stereoscopic camera and optical tracking systems is the complex setup and calibration of the optical tracking system. The setup of optical tracking systems is time-intensive and requires precise positioning of cameras and optical trackers. Even slight misalignment during calibration can lead to errors in tracking the patient's anatomy or surgical instruments, reducing surgical precision. Additionally, setup time adds to the length of the procedure, potentially increasing patient exposure to anesthesia and overall operative time.

[0047] Another challenge of stereoscopic camera and optical tracking systems is their reliance on optical trackers. The physical nature of optical trackers introduces several potential points of failure. For example, tracker placement can affect surgical exposure, requiring larger incisions or altered approaches that may increase invasiveness. Additionally, trackers are prone to physical damage, shifting, or detachment during surgery. Further, if any optical marker on the tracker becomes dirty during surgery, it may affect its registration to the virtual OT space, reducing accuracy. Certainly, if a tracker becomes dislodged or its line of sight is sufficiently obstructed, the system cannot accurately track the anatomy or instruments, leading to error. The specialized optical trackers, cameras, and calibration equipment are expensive, limiting their use to well-funded hospitals and surgical centers. The additional equipment also increases the logistical complexity of the operating room. These systems include a lot of equipment and specific procedures must be followed to ensure their proper use and functioning. Thus, the training and learning curve associated with these systems is substantial. While these systems can save time in some areas, they take time to set up and use properly. Depending on the surgeon this could be a net savings but can also result in a net increase in surgical time.

[0048] Additionally, every time an optical tracker is used, an additional transformation matrix is required to localize it and the additional transformations may introduce additional errors. Such transformations lead to errors depending on the distance the trackers have with the anatomy in question. In many cases trackers are more than 6 inches away from relevant anatomy, so any angular error is magnified by that distance. It is an advantage to navigate based on anatomy to remove this distance error as much as possible.VIS-007-W01

[0049] As mentioned, with current optical tracking navigation systems the line of sight between the cameras and optical trackers must remain unobstructed for the system to function effectively. However, during surgery, this line of sight can be blocked by surgical staff or equipment, causing the system to lose track of the optical markers. Limited field of view (FOV) of stereoscopic cameras can cause tracking loss if anatomy or instruments move outside the tracked area, necessitating frequent repositioning. These interruptions necessitate recalibration or repositioning, disrupting the surgical workflow and potentially adding stress and time to the procedure. For example, intraoperative re-registration is required after a tracker becomes dislodged or bumped such that it shifts relative to anatomy. This re-registration can significantly prolong surgery, impacting workflow and anesthesia time.

[0050] Additionally, there are limits to the speed of the system. If OTs are moving too fast, the system can lose registrations. When this happens, the system must notify the surgeon, and the work must slow down.

[0051] High cost and limited adoption is another challenge of current optical tracking navigation systems. The substantial resources required to acquire, maintain, and operate these systems have limited their widespread adoption. Beyond the financial investment, the time and effort needed for training surgical teams, the complexity of setup, and the extended procedural time also present significant barriers. These systems demand highly trained personnel to ensure proper use, and the intricate setup process can increase the overall time required for surgeries. Additionally, the ongoing need for system and / or tool calibration and troubleshooting during procedures adds to the operational complexity, making it challenging for smaller hospitals and clinics to adopt these systems. As a result, accessibility remains limited, particularly in resource- constrained environments.

[0052] While stereoscopic cameras and optical trackers have become valuable tools for improving surgical precision, they come with several limitations. The reliance on physical optical trackers, complex setup, time-intensive calibration, and susceptibility to mechanical failures all point to the need for a more robust and flexible solution. Additionally, the combination of high financial costs, lengthy training requirements, and the complexity of operating these systems has made broader adoption across surgical specialties challenging.VIS-007-W01

[0053] In joint replacement surgeries such as TKA, achieving balanced joint laxity is essential for optimal implant function and longevity. Joint laxity refers to the flexibility and stability of the joint under applied forces, such as those experienced during routine movements. A key part of the intraoperative procedurejoint laxity assessment involves applying controlled stresses to the joint and measuring the resulting gaps between the articulating surfaces. These gap measurements allow surgeons to understand the tension in surrounding soft tissues and ensure that the artificial joint will perform in a stable, balanced manner postoperatively.

[0054] Gap balancing is one methodology used by surgeons in joint replacement, especially in TKA, as it can directly impact implant stability and function. In TKA, the goal of the gap balancing technique is to create equal and stable gaps on both the medial (inner) and lateral (outer) sides of the joint when the knee is in different positions, such as full extension (0°) and the patient’s full flexion range. Achieving this balance helps prevent uneven loading on the joint, improper alignment, and excessive wear on one side of the implant, which could otherwise lead to complications such as limited range of motion, instability, or even implant failure.

[0055] For instance, in TKA, surgeons typically assess joint gaps by applying varus (medial) and valgus (lateral) stresses to the knee at specific points in the procedure. For example, applying a lateral force at the knee will cause the knee to angulate in a varus direction. Applying a medial force at the knee will cause the knee to angulate in a valgus direction. By observing how the medial and lateral gaps adjust to these forces, surgeons can identify any imbalances and make real-time adjustments to the surgical plan before any bones are cut. Properly balanced gaps ensure that soft tissues are neither too tight nor too loose, allowing for smoother motion and distributing forces more evenly across the joint. Balancing the gaps can be accomplished by different means and may require a change to the surgical plan and / or altering the planned position for the implant. In some instances, failure to balance gaps accurately can lead to complications such as pain, instability, and abnormal gait, as well as early implant wear, particularly if one side of the knee experiences excessive pressure over time.

[0056] For many years, surgeons have relied on various manual instruments to measure joint gaps during procedures. These tools provide basic quantitative information on joint laxity by giving an instantaneous reading of the gap size under controlled stress. However, manualVIS-007-W01 measurement devices have proven challenging to work with for several reasons. First, many manual devices are difficult to calibrate accurately, and achieving reliable readings depends on ensuring that the medial and lateral joint surfaces are carefully prepared. Variability in surface preparation can lead to inconsistent or inaccurate measurements. Additionally, these tools must be positioned precisely to capture accurate measurements. Any deviation from the intended position can skew the readings and provide an inaccurate assessment of joint laxity. Finally, manual instruments provide only a single, instantaneous reading rather than a continuous measurement. As ligaments stretch and adjust to applied forces, these readings may not capture the full extent of the joint’s response, particularly the peak gap sizes that represent the joint’s maximum laxity. Without peak values, surgeons may underestimate the j oint’ s true laxity, potentially leading to suboptimal soft tissue balancing. As a result of these limitations, manual measurement tools are often viewed as less reliable for achieving the precision needed for modern joint replacement procedures.

[0057] In recent years, optical tracking systems have become the best available approach for achieving accurate, real-time, quantitative measurements of joint laxity. These systems employ Optical Trackers (OTs) attached to the femur and tibia to continuously monitor their relative positions. As the joint is manipulated, multiple cameras capture the trackers position in real-time data enabling calculation of joint angles and gaps, including critical values such as flexion / extension, varus / valgus, and intemal / external rotational angles. Unlike manual tools, optical tracking systems provide a continuous flow of measurements, allowing surgeons to capture peak gaps during dynamic movements. This ensures that the maximum laxity of the joint is accurately represented, providing a comprehensive understanding of how the joint will behave under typical stresses.

[0058] Despite the advantages of optical tracking systems, they are not without challenges. The setup process is complex, involving the secure attachment of OTs to each bone with fixation pins or clamps. Once positioned, these trackers require a continuous line of sight with the camera system, which may be obstructed by surgical instruments or staff during complex procedures. If the trackers are accidentally dislodged or moved relative to the anatomy, recalibration is necessary, adding time and potential sources of error to the workflow. The procedural complexity of optical tracking systems makes them less practical in certain surgicalVIS-007-W01 environments, and their dependency on equipment and calibration increases the burden on the surgical team. Not to mention, optical tracking systems are invasive for the patient.

[0059] To perform accurate and repeatable joint laxity assessments, it is essential to establish standardized anatomical coordinate systems within CT bone datasets. These coordinate systems consist of origins, planes, and mutually orthogonal axes, forming a structured framework for measuring joint angles and gaps. There are two common methods for defining these coordinate systems within CT datasets: (1) Origin-Based Transformation (Assumes DICOM RAS Coordinate System), and (2) Full Anatomical Definition (No Assumption of RAS Coordinate System).

[0060] By way of example, the Origin-Based Transformation approach assumes that the original CT data is in the DICOM RAS (Right-Anterior-Superior) coordinate system, and it re- centers the existing axes based on anatomical landmarks to create new origins for each bone. By maintaining alignment with the DICOM RAS orientation, this method supports workflows that require compatibility with standard imaging protocols. Essentially, this method has three main steps, including identifying anatomical origins, defining new axes parallel to D1C0M RAS axes, and defining anatomical planes based on the newly defined axes.

[0061] To identify anatomical origin of a tibia (for example), first the tibial plateau is located. In the CT dataset, segmented to include just the tibia, the tibial plateau is the upper surface of the tibia where the femoral condyles rest. This surface typically appears as a relatively flat, broad area at the top of the tibial bone in the knee region. Next, the medial and lateral edges are identified by finding the highest points on each side within the plateau, corresponding to the medial tibial condyle (inner side) and lateral tibial condyle (outer side). Finally, the central point between the two highest points on the medial and lateral edges is determined. This midpoint of the tibial plateau serves as the anatomical origin for the tibia. To ensure accuracy, one may confirm that this midpoint aligns visually with the center of the tibial plateau’s width.

[0062] To identify anatomical origin of a femur (for example), first the medial and lateral epicondyles are identified in the CT dataset. The medial and lateral epicondyles are the bony prominences at the widest points on either side of the distal femur. Next, the central point between the medial and lateral epicondyles is determined. This midpoint serves as the femoralVIS-007-W01 origin, providing a central and anatomically consistent reference point aligned with the femur’s articulating surface.

[0063] New axes parallel to DICOM RAS axes are typically defined as follows. The medial- lateral (ML) axis (X-axis) is aligned parallel to the DICOM X-axis, running right (positive) to left (negative) from the new origin. The anterior-posterior (AP) axis (Y-axis) is aligned parallel to the DICOM Y-axis, running posterior (negative) to anterior (positive) from the new origin. The proximal-distal (PD) axis (Z-axis) is aligned parallel to the DICOM Z-axis, running inferior (negative) to superior (positive) from the new origin.

[0064] Anatomical planes based on the new axes are then defined as follows. The sagittal plane is formed by the ML (X) and PD (Z) axes, dividing the bone into left and right sections. The coronal plane is formed by the ML (X) and AP (Y) axes, separating anterior and posterior sections. The transverse plane is formed by the AP (Y) and PD (Z) axes, dividing proximal and distal sections.

[0065] By way of example, the Full Anatomical Definition method does not rely on a preexisting coordinate system and instead defines both the origin and axes based on anatomical landmarks for each bone.

[0066] By way of example, femoral origin is defined in the same way as described above. Specifically, first the medial and lateral epicondyles are identified in the CT dataset. The medial and lateral epicondyles are the bony prominences at the widest points on either side of the distal femur. Next, the central point between the medial and lateral epicondyles is determined. This midpoint serves as the femoral origin, providing a central and anatomically consistent reference point aligned with the femur’s articulating surface.

[0067] Next the femoral axes are defined using anatomical landmarks. For example, the medial-lateral (ML) axis (X-axis) is defined by identifying the medial and lateral epicondyles on the distal femur. The ML axis (X-axis) is the line connecting these two points, with the positive direction running from the medial epicondyle to the lateral epicondyle. The proximal-distal (PD) axis (Z-axis) is defined as the line perpendicular to the ML axis (passing through the femoral origin) and parallel to the anterior surface of the femoral metaphysis. The positive direction ofVIS-007-W01 the PD axis runs from distal (closer to the knee) to proximal (upward along the femoral shaft). Ensuring this axis is perpendicular to the ML axis and aligned with the anterior surface provides anatomical consistency. The anterior-posterior (AP) axis (Y-axis) is defined as the line perpendicular to both the ML and PD axes. Using the right-hand rule, this alignment will place the positive direction of the AP axis running from posterior to anterior (back to front) through the femur.

[0068] The femoral anatomical planes are defined as follows. The sagittal plane is formed by the ML (X) and PD (Z) axes, dividing the bone into left and right sections. The coronal plane is formed by the ML (X) and AP (Y) axes, separating anterior and posterior sections. The transverse plane is formed by the AP (Y) and PD (Z) axes, dividing proximal and distal sections.

[0069] By way of example, the tibial origin is identified by locating the medial and lateral edges of the tibial plateau and establishing the origin at the midpoint between these edges. This serves as the central reference point for the tibial coordinate system.

[0070] The tibial axes can be defined using anatomical landmarks. For example, the medial- lateral (ML) axis (X-axis) extends between the medial and lateral edges of the tibial plateau, with the positive direction running from medial to lateral, as defined for the femur. The proximal- distal (PD) axis (Z-axis) is defined as the perpendicular to the ML axis (extending through the tibial origin) and aligned with the proximal tibial shaft direction. The positive direction runs from distal to proximal. The anterior-posterior (AP) axis (Y-axis) is defined as the line perpendicular to both the ML and PD axes, with the positive direction running posterior to anterior.

[0071] The tibial anatomical planes are defined in the same way as the femoral anatomical planes. The sagittal plane is formed by the ML (X) and PD (Z) axes, dividing the bone into left and right sections. The coronal plane is formed by the ML (X) and AP (Y) axes, separating anterior and posterior sections. The transverse plane is formed by the AP (Y) and PD (Z) axes, dividing proximal and distal sections.VIS-007-W01

[0072] By embedding these standardized anatomical coordinate systems within CT data, the resulting datasets provide an essential reference frame for accurate intraoperative measurement, supporting consistency in joint laxity assessments.

[0073] With the anatomical coordinate systems in place for each bone, three primary angles of knee rotation — flexion / extension (F / E), varus / valgus (V / V), and internal / external (EE) rotation — can be calculated. These angles are fundamental to understanding the knee’s alignment and stability, especially during procedures like TKA. By way of example, the F / E angle is measured in the femoral sagittal plane, defined by the femoral AP (X) and PD (Z) axes, and determined by projecting the tibial PD axis onto the femoral sagittal plane and calculating the angle between the femoral PD axis and the projected tibial PD axis. A positive angle indicates flexion, while an angle close to 0° indicates extension. The V / V angle is measured in the femoral coronal plane, defined by the femoral ML (Y) and PD (Z) axes, and determined by projecting the tibial PD axis onto the femoral coronal plane and measuring the angle between the femoral PD axis and the projected tibial PD axis. A positive angle indicates varus, and a negative angle indicates valgus. The I / E rotation angle is measured in the femoral transverse plane, defined by the femoral ML (Y) and AP (X) axes and determined by projecting the tibial AP axis onto the femoral transverse plane and measuring the angle between the femoral AP axis and the projected tibial AP axis. A positive value indicates internal rotation, and a negative value indicates external rotation.

[0074] By embedding these anatomical definitions and coordinate systems directly into the CT dataset, surgical navigation systems have access to a standardized reference, allowing for precise, real-time calculations of these angles. This enables accurate assessments of joint alignment and stability, enhancing intraoperative decision-making and supporting optimal implant positioning in TKA and other joint replacement procedures. The exact construction and definition of the coordinate systems used to compute gaps can vary based on surgeon training and preference and is not critical if the method supports mathematical computation of the instantaneous gaps. This is also true of the gap measurement itself, which can be defined in various ways but if the measure is consistent and the relative motion from a varus to a valgus stress can be computed and the surgeon understands the nuances of the definition, the various methods are equally valid.VIS-007-W01

[0075] In orthopedic procedures, particularly in lower limb surgeries such as TKA, accurate knowledge of joint alignment and rotational centers is critical for achieving optimal outcomes. In some instances, topological datasets can be used to calculate a center of rotation or other constrained position or motion based in an indirect measurement of unobservable anatomy as it is moved through a motion path. The hip center of rotation, in particular, serves as a useful anatomical reference. Its accurate localization enables surgeons to make well-informed adjustments to knee alignment, implant positioning, and joint balance. A precise understanding of the hip center directly impacts lower limb alignment, which influences gait, joint loading, and overall implant longevity post-surgery.

[0076] Traditionally, the hip center of rotation is established preoperatively through imaging, such as computed tomography (CT) or magnetic resonance imaging (MRI). However, during surgery, direct measurement of this point is challenging because it typically lies outside the immediate field of view (e.g., at least in part because the surgical exposure is at the knee and the hip is buried deep within tissue), requiring either indirect estimates or complex tracking setups. Intraoperative methods that rely on optical tracking systems can capture alignment information, but they often require additional equipment, extended setup time, require line-of-sight maintenance, and include patient-invasive pins, all of which complicate the surgical workflow.

[0077] In image-guided surgical navigation, precise anatomical reference points are critical to ensuring alignment and optimal outcomes, particularly in procedures like TKA where accurate lower limb alignment is necessary. Traditionally, these reference points are determined either through direct measurement or using indirect tracking methods like optical trackers. However, these systems can be complex to set up, require a clear line of sight, and are prone to errors if the trackers shift during surgery.

[0078] In modern surgical procedures, particularly in orthopedic surgeries such as TKA, precise alignment and verification of anatomical structures and planned bone resections are essential for successful outcomes. Traditionally, verification during surgery relies heavily on OTs attached to anatomical structures or specialized instruments. In current practices, a critical verification process, referred to as a checkpoint, can help to ensure that the attached optical trackers have not shifted relative to the anatomy they represent. For example, in TKAVIS-007-W01 procedures, a checkpoint screw can be installed in a fixed location on the femur, typically in the metaphysis, where it will not be affected by bone resections. The location of this screw is periodically verified using a pointer equipped with an optical tracker. The transformation matrix between a point on the femoral OT and this checkpoint location remains consistent throughout the procedure if the OT’s position relative to the bone is stable. If any change is detected in the checkpoint position beyond an acceptable tolerance, it indicates potential tracker movement, and the surgeon must stop to correct it. This process can lead to procedural delays and the need for surgical steps to be retraced.

[0079] Another significant verification need involves confirming the accuracy of planned bone cuts. In TKA, for instance, up to five planar cuts are made on the femur to accommodate the implant placement. These cuts must closely match the planned orientation and position; otherwise, the surgeon faces challenges in adjusting to discrepancies after the bone has already been resected. To verify these cuts, specialized instruments with planar surfaces and attached OTs are commonly used. These instruments are positioned on the cut surfaces, and their locations are compared against the preoperative surgical plan. Any variance can require additional adaptation from the surgeon, adding complexity to the procedure.

[0080] While effective, these optical tracker-dependent methods have significant drawbacks. They rely on additional hardware and setup time, create potential points of failure if trackers shift or instruments are misaligned, are invasive for the patient, and require recalibration when inconsistencies are detected.SUMMARY

[0081] This disclosure builds upon the innovations presented in U.S. Patent Application No. 18 / 505,973 (U.S. Pub. 2024 / 0102795) (“the ‘973 app”), filed Nov. 9, 2023 and entitled “GENERATION OF ONE OR MORE EDGES OF LUMINOSITY TO FORM THREE- DIMENSIONAL MODELS OF SCENES”. More specifically, the present disclosure introduces advanced clinical applications for the previously described 3D scanning system. The ‘973 app unveiled a groundbreaking 3D scanner whose capabilities were not previously available in surgical settings. The present disclosure expands on the utility of that technology, particularlyVIS-007-W01 emphasizing its application in orthopedic surgeries such as Total Knee Arthroplasty (TKA) and Robotic Assisted TKA (RA-TKA).

[0082] The enhancements detailed herein expand upon the exemplary clinical uses disclosed in the ‘973 app, with the goal of illustrating how the technology can be highly beneficial and tailored in certain arthroplasty procedures. For example, by leveraging advanced scanning methodologies, the system reduces the need for optical trackers, streamlining setup and improving surgical precision. While this disclosure can be applied broadly across multiple surgical domains, particular emphasis is given to its use in orthopedic procedures such as TKA. The subsequent sections will outline the significant advancements this technology brings to the clinical environment, setting a new standard for precision and efficiency in surgical navigation and instrumentation. Moreover, although described herein in this context as a specific example, it should be understood that the system and methods disclosed herein may be used in any scenario (surgical, medical, or otherwise) to generate a reconstructed 3D model of an object, register it to a reference object in a different coordinate space, and track movement of the object in real space with respect to the reconstructed 3D model of the object within the coordinate space.

[0083] The present disclosure describes a 3D scanner specifically designed for surgical applications, capable of collecting visible topological data from the surgical field in real time. This represents a new class of data that was previously unavailable during surgery. By capturing continuous, high-rate 3D topological data during the procedure, this scanner can replace traditional workflows that rely on optical trackers — without the associated costs, setup time, and limitations. Although described within the context of surgical applications as a specific example, the system and methods disclosed herein may be used in a wide variety of non-surgical applications to register and track an object within a coordinate space.

[0084] This disclosure expands on how this topological data can be applied in a surgical setting to perform various procedures. Although the current disclosure emphasizes applications in TKA, the underlying principles can be adapted for other surgeries with minimal modifications by those skilled in the art. While the 3D scanner described herein is the preferred method for collecting this data, the use of the scanner is not a requirement for this innovation, as otherVIS-007-W01 technology capable of capturing 3D surface data during surgery may be utilized to achieve similar results.

[0085] When a continuously updated 3D topological data of sufficient accuracy and rate of the surgical site is available, the data can be used for Continuous Anatomical Auto Tracking (CAAT). This innovation eliminates the need for introduced physical optical markers and reduces the complexity of setup, offering a more streamlined, cost-effective, and accurate approach to surgical navigation. CAAT allows for real-time registration and tracking of anatomical structures, instruments and equipment without using OMs or OTs or other fiducials, marking a significant advancement in surgical technology. CAAT promises greater precision, flexibility, and efficiency in the operating room, benefiting a wide range of surgical procedures.

[0086] Additionally, this disclosure addresses a critical challenge in CT image-based surgical navigation: the misalignment between pre-operative CT data, which accurately captures bone structures, and intraoperative topological scans, which predominantly capture the cartilage- covered bone surfaces exposed during surgery. Cartilage, with its variable thickness and low visibility in CT scans, introduces alignment errors when CT data is registered directly to topological scans, especially in procedures like TKA.

[0087] By way of example, this disclosure introduces a novel approach to overcome cartilage- to-bone surface mismatch, providing an immediate solution that aligns the cartilage and bone surfaces without reintroducing optical trackers. Specifically, this disclosure addresses the challenges of cartilage and bone misalignment in surgical navigation, providing a method to accurately register CT data within the surgical coordinate system, enhancing alignment accuracy and improved outcomes in image-guided surgery. By accurately accounting for the cartilage thickness in the registration process, this method enables the navigation system to achieve reliable alignment with the bone structure, maintaining the high-precision, non-invasive advantages of topological scanning.

[0088] This disclosure provides a method to correct for cartilage-bone misalignment by segmenting and registering bone-only data. Initially, optical intraoperative scans are segmented to isolate captured surface bone separate from the cartilage, creating datasets that align with the bone-only data in pre-operative CT scans. One may also use regions of known hard tissueVIS-007-W01(particularly thin cartilage), such as the edges of the condyles, and the sulcus between the condyles. Using a combination of coarse alignment (e.g., RANSAC) and fine alignment (e.g., ICP), a transformation matrix is generated to accurately co-locate the CT dataset with the optical data set in a shared coordinate space. Once established, this matrix allows the entire set of related intraoperative data — both bone and cartilage — to be precisely aligned in the common space without further manual adjustments.

[0089] By way of example, the method disclosed herein sets the foundation for achieving accurate surgical navigation in the presence of cartilage-bone mismatch, while maintaining the streamlined, automated benefits of modern topological scanners. This single method provides a practical and effective solution, improving the quality and reliability of image-based surgical navigation.

[0090] In some embodiments, this disclosure describes systems and methods for real-time assessment of anatomical structures during surgical procedures. By leveraging topological data collection techniques, the disclosure enables accurate measurement of joint laxity and relative gap changes between bones or other anatomical structures without requiring optical tracking systems. The approach captures continuous, three-dimensional surface scans of relevant anatomical structures, such as bones, enabling precise calculation of joint gaps under various applied stresses without requiring attached optical tracking devices. It is particularly applicable to orthopedic surgery, joint replacement, and other procedures requiring precise alignment and stability assessment, though its principles can be adapted to a variety of surgical and diagnostic applications.

[0091] In some embodiments, a method disclosed herein establishes a baseline “zero-stress” measurement and calculates relative changes in gap size under controlled stresses, such as varus or valgus forces. By determining the change in varus / valgus angles and using a known contact distance between key anatomical points, the system and method described herein calculates lateral and medial gaps with accuracy. The system’s real-time tracking continuously monitors the relative positions of the anatomical structures, capturing peak gap values to provide a comprehensive assessment of joint laxity.VIS-007-W01

[0092] Applicable to a range of surgical fields, the method described herein offers a versatile solution for measuring relative distances and angles between any two anatomical structures where topological data can be collected and registered to preoperative imaging. Examples include orthopedic procedures, such as knee and shoulder replacements, as well as spinal and hip surgeries.

[0093] In some embodiments, the method described herein provides a way to calculate the hip center of rotation dynamically and with high confidence, using only topological data captured intraoperatively. By leveraging the femur’s constrained rotational movement about the hip, this method enables precise, real-time calculation of the hip center without requiring an expanded scan field or external trackers. This approach not only simplifies the workflow but also enhances intraoperative decision-making by offering accurate anatomical reference points that were previously only estimable with additional equipment.

[0094] This disclosure involves real-time 3D topological scanning to calculate anatomical reference points, such as the hip center of rotation, during procedures. This technology combines aspects of computer vision, structured light 3D scanning, and least-squares fitting algorithms to enhance surgical alignment and implant positioning. Applications include TKA and other joint surgeries where precise alignment and intraoperative reference points are critical to patient outcomes.

[0095] The method described herein is for accurately determining the hip center of rotation in real-time during surgery using a 3D topological scanner (such as the scanner described herein and / or in the ‘973 app). By way of example, as the surgeon slowly moves the knee, the scanner captures a series of femoral origin points within its field of view. These points, transformed in camera space, approximate a spherical surface due to the femur’s constraint by the hip joint. The center of this sphere corresponds to the hip center of rotation, an essential anatomical reference for surgical alignment.

[0096] In some embodiments, to estimate this center, a least-squares sphere-fitting method may be applied to the collected data points, minimizing errors to generate the best-fit sphere. The method further validates the precision of this calculation through a confidence ellipsoid, generated by plotting the residual errors of each point, providing a probabilistic boundary thatVIS-007-W01 ensures the estimated center’s accuracy to a defined confidence level, typically 95%. This approach enables surgeons to achieve optimal implant alignment without additional tracking devices, streamlining the surgical workflow and improving patient outcomes.

[0097] In some embodiments, this disclosure introduces a method for identifying specific anatomical points that are beyond the scanner’s field of view without performing calculations on visible anatomy or using optical trackers. Rather than relying upon line-of-sight-dependent trackers, this approach leverages a pointer instrument with a distinct “finial” design (or other navigation feature). This finial, which may be part of the handle or positioned above the handle, remains scannable even as the pointer is extended to contact points beyond the field of view. By registering the finial’s position in 3D topological scans, the system can infer the location of anatomical points using the known geometry of the pointer.

[0098] While this method applies broadly to various orthopedic procedures, TKA and locating the ankle center are used as examples here to explain the principles. In TKA, for instance, determining the center of the ankle joint is crucial for alignment. Using the pointer to locate both the medial and lateral malleoli (for example), the system infers the ankle center by registering the finial’s position, providing an anatomical reference without requiring additional tracking devices.

[0099] Procedures like TKA require precise landmark identification, and this method addresses this need while streamlining the surgical setup by eliminating traditional tracking devices. However, this method can be broadly applied to any surgical procedure that requires precise anatomical point localization.

[0100] In TKA, accurate localization of the ankle relative to the tibia is critical for establishing mechanical axis alignment. Current navigation systems such as stereo tracking platforms rely on external cameras and arrays of markers to track instruments. The approach described herein leverages the optical 3D scanner already directed at the exposed tibia to simultaneously localize both the tibia and a specialized pointer. The pointer extends distally to touch the ankle, while proximally it carries a prominent structure optimized for robust localization, which for example may be a cluster of fiducial spheres, or otherwise be a unique finial that uniquely describes alignment and orientation in six degrees of freedom, such as aVIS-007-W01 contiguous shape that is highly visible when illuminated by the wavelengths used by the scanner. Because the scanner captures both the tibial surface and the fiducials in the same frame, the position and orientation of the pointer tip relative to the tibia is established with high accuracy. This avoids additional hardware or reorientation of the scanner and provides a workflow- integrated method for ankle localization during TKA.

[0101] In some embodiments, this disclosure provides a method and device for localizing the ankle during TKA procedures by extending the capabilities of an optical 3D scanning system that is concurrently being used to register the tibia. By way of example, a rigid pointer is designed to contact the ankle at its distal tip, while its proximal end is equipped with a cluster of fiducial spheres (or finial or other navigation feature). The cluster may be optimized for the scanner’s strongest localization performance, positioned near the exposed tibia so that both the fiducials and tibial anatomy are scanned within the same field of view. The rigid body relationship between the fiducial cluster and the pointer tip my be determined through calibration, thereby allowing the ankle contact point to be precisely resolved.

[0102] In some embodiments, calibration of the pointer tip relative to the fiducials could be achieved through a dedicated attachment that accepts the pointer tip in a defined seating geometry. By scanning this attachment with the optical 3D system, the tip position is constrained with high certainty and registered into the same coordinate system as the fiducials. During surgery, once the pointer tip is placed on the ankle, the scanner captures both the tibial anatomy and fiducial cluster. Because the transformation between fiducials and pointer tip is known, the ankle position can be localized relative to the tibia.

[0103] By way of example, the system provides several advantages over conventional external camera tracking systems:

[0104] Improved Accuracy - The fiducials are positioned close to the tibia, where the scanner already has a direct and optimized view of the tibial surface.

[0105] Simplified Workflow - The method eliminates the need for line-of-sight external tracking equipment and leverages the same scanner already in use for tibial digitization.VIS-007-W01

[0106] Robustness at Distance - The distal ankle can be localized with surgical reliability without repositioning hardware, for example eliminating the need to redirect the scanner to view the ankle anatomy.

[0107] Calibration Flexibility - To calibrate the pointer, the location of the tip needs to be determined by the CAAT system in some manner. For example, in some embodiments, a dedicated tip-localizing attachment enables precise calibration of the pointer tip relative to the fiducial cluster. In some embodiments, the 3D scanner may have a divot configured to receive the tip. In some embodiments, the tip may have fiducials attached that are easily visible to the scanner when the scanner is used in a mode that has an extended field of view. In some embodiments, markings on the pointer shaft may indicate the location of the tip when out of view. In some embodiments, the CAAT system may be configured to stitch scan datasets of the pointer to register the tip.

[0108] The pointer and method disclosed herein therefore integrates tibial surface registration and ankle localization into a single, scanner-based workflow. It reduces equipment complexity and improves efficiency relative to existing systems.

[0109] In some embodiments, this disclosure provides a verification method that replaces traditional checkpoint and osteotomy verification methods by leveraging intraoperative topological data captured through a 3D scanner. By scanning the anatomical site in real time, this disclosure provides a method for precise, imageless verification of anatomical landmarks and resected surfaces without requiring additional trackers or specialized instruments. These verifications can be performed continuously throughout the procedure or as single, on-demand scans at specific stages, depending on the needs of the surgeon. This approach not only reduces setup complexity but also streamlines workflow by eliminating procedural delays associated with OT recalibration, offering a robust, tracker-free solution for accurate and reliable intraoperative verification.

[0110] In some embodiments, this disclosure introduces a method for real-time verification of anatomical structures and bone resections during surgery, leveraging topological data to replace conventional optical tracker-dependent systems. In traditional practices, checkpoints are used to verify OT stability, while OT-based instruments with planar surfaces are employed to confirmVIS-007-W01 resection planes. This disclosure eliminates these requirements by providing continuous registration and tracking through topological data, eliminating the need for physical trackers and checkpoints.

[0111] By way of example, the method operates in two stages: first, topological data is collected to verify anatomical consistency, replacing checkpoint methods; second, post-resection scans are compared to a modified, idealized model of the anatomy to assess resection accuracy. These stages streamline intraoperative workflows by minimizing setup complexity and reducing interruptions caused by tracker recalibrations or alignment adjustments.

[0112] As additional description to the embodiments described below, the present disclosure describes the following embodiments.

[0113] Embodiment 1 is a system for measuring gaps between a first bone and a second bone within an affected j oint having a medial side and a lateral side in an operating room environment during a surgical procedure, comprising a structured light three-dimensional scanner, including: a light source: a liquid crystal matrix; and an image capture device; a computer system including a memory and at least one processor; and computer readable media embodied in a non-transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a first three-dimensional data representation of a topological surface of the first bone using a first dataset, the first dataset derived from one or more images of the topological surface of the first bone captured by the image capture device without optical trackers or applied fiducials; register the first three-dimensional data representation to a second dataset such that the first three-dimensional data representation and the second dataset are aligned within a common coordinate system, the registration occurring without optical trackers or applied fiducials; track the first bone relative to the common coordinate system without optical trackers or applied fiducials; construct a second three- dimensional data representation of a topological surface of a second bone using a third dataset, the third dataset derived from one or more images of the topological surface of the second bone captured by the image capture device without optical trackers or applied fiducials; register the second three-dimensional data representation to a fourth dataset such that the second three- dimensional data representation and the fourth dataset are aligned within the common coordinateVIS-007-W01 system without optical trackers or applied fiducials; track the second bone relative to the common coordinate system without optical trackers or applied fiducials; and calculate, with the affected joint under applied stress, at least one of a medial gap value, a lateral gap value, and a change in varus-valgus angle.

[0114] Embodiment 2 is the system of embodiment 1, wherein the light source is a linear light source.

[0115] Embodiment 3 is the system of embodiments 1 or 2, wherein the first three- dimensional data representation is a point cloud or three-dimensional data convertible into a point cloud.

[0116] Embodiment 4 is the system of any of embodiments 1 through 3, wherein the captured images of the topological surface of the first bone are structured light patterned images that encode the surface of the first bone.

[0117] Embodiment 5 is the system of any of embodiments 1 through 4, wherein the first three-dimensional data representation of the topological surface of the first bone is constructed by: decoding light patterns contained in the one or more captured images of the topological surface of the first bone by analyzing brightness history to create camera pixel correspondence to light angle projection; using triangulation to calculate a three-dimensional coordinate for each pixel in the one or more captured images of the topological surface of the first bone based on the decoded light patterns; and combining the triangulated three-dimensional coordinates of each pixel into the first three-dimensional point cloud representation of the surface of the first bone in physical space.

[0118] Embodiment 6 is the system of any of embodiments 1 through 5, wherein the second dataset is derived from a preoperative computed tomography scan or magnetic resonance imaging scan of the first bone.

[0119] Embodiment 7 is the system of any of embodiments 1 through 6, wherein the second dataset is derived from one or more subsequent images captured by the image capture device without said optical trackers or applied fiducials.VIS-007-W01

[0120] Embodiment 8 is the system of any of embodiments 1 through 7, wherein the first bone is a femur, the second bone is a tibia, and the affected joint is a knee.

[0121] Embodiment 9 is the system of any of embodiments 1 through 8, wherein: the computer system further comprises a user interface and a display device, and the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display, on the display device, the tracked first three-dimensional data representation of the first bone and the tracked second three-dimensional data representation of the second bone.

[0122] Embodiment 10 is the system of any of embodiments 1 through 9, wherein the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display, on the display device, a real-time determination of at least one of a flexion-extension angle, varus-valgus angle, and interior-exterior rotation angle based on the tracked first three-dimensional data representation of the first bone and the tracked second three-dimensional data representation of the second bone.

[0123] Embodiment 11 is the system of any of embodiments 1 through 10, wherein tracking the first bone relative to the common coordinate system comprises continuously updating the first bone’s position relative to the common coordinate system by repeatedly acquiring surface data, registering the repeatedly acquired surface data in the common coordinate system, and updating transformation matrices.

[0124] Embodiment 12 is the system of any of embodiments 1 through 11, wherein the computer readable media includes a further set of instructions, that, when executed by a processor, cause the computer system to: establish a zero-gap varus-valgus angle and zero gap condition of the first bone and second bone while the affected joint is positioned in an extension orientation with no stress applied to the affected joint; and record the established zero-gap varus- valgus angle and zero gap condition as a baseline measurement.

[0125] Embodiment 13 is the system of any of embodiments 1 through 12, wherein the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to calculate the lateral gap value by: measuring a testVIS-007-W01 varus-valgus angle of the affected joint with varus stress applied to the affected joint, thereby opening the lateral gap between the first bone and second bone on the lateral side and creating a stable contact between the first bone and second bone on the medial side; comparing the measured test varus-valgus angle of the affected joint to the baseline measurement varus-valgus angle to calculate the change in varus-valgus angle of the affected joint; recording peak lateral gap measurements by continuously tracking the first bone and second bone in real time; determining a known articular contact distance between contact points on the first bone and second bone; calculating the lateral gap value based on the determined articular contact distance and the change in varus-valgus angle of the affected joint.

[0126] Embodiment 14 is the system of any of embodiments 1 through 13, wherein the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to calculate the medial gap value by: measuring a test varus-valgus angle of the affected joint with valgus stress applied to the affected joint, thereby opening the medial gap between the first bone and second bone on the medial side and creating a stable contact between the first bone and second bone on the lateral side; comparing the measured test varus-valgus angle of the affected joint to the baseline measurement varus-valgus angle to calculate the change in varus-valgus angle of the affected joint; recording peak lateral gap measurements by continuously tracking the first bone and second bone in real time; determining a known articular contact distance between contact points on the first bone and second bone; calculating the medial gap value based on the determined articular contact distance and the change in varus-valgus angle of the affected joint.

[0127] Embodiment 15 is a system for calculating hip center of rotation during a surgical procedure in an operating room environment, comprising: a structured light three-dimensional scanner, including: a light source: a liquid crystal matrix; and an image capture device; a computer system including a memory and at least one processor; and computer readable media embodied in a non-transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: use a three-dimensional data representation of a topological surface of a femur constructed from a first dataset, the first dataset derived from a pre-operative imaging event; obtain a plurality of second datasets from the image capture device during a plurality of image capture events, each second dataset of theVIS-007-W01 plurality of second datasets comprising topological scan data of the femur derived from one or more images intraoperatively captured by the image capture device during an image capture event of the plurality of image capture events without optical trackers or applied fiducials; register the three-dimensional data representation to each second dataset of the plurality of second datasets in a series of registrations to find a transformation matrix such that the three- dimensional data representation and the plurality of second datasets are aligned within a common coordinate system, the registrations occurring without said optical trackers or applied fiducials; track the femur relative to the common coordinate system without said optical trackers or applied fiducials; use the found transformation matrix to compute a coordinate location of a chosen anatomic point on the three-dimensional data representation in a common coordinate system stationary with respect to the hip center of rotation for each registration of the series of registrations such that a plurality of coordinate locations is generated for sphere fitting; and calculate a size and location of a best-fit sphere to the plurality of generated coordinate locations using a sphere-fitting algorithm; establish and check the quality of the best-fit sphere; and report the center of the best-fit sphere as the hip center of rotation.

[0128] Embodiment 16 is the system of embodiment 15, wherein the sphere-fitting algorithm is a least squares fitting method.

[0129] Embodiment 17 is the system of embodiments 15 or 16, wherein the light source is a linear light source.

[0130] Embodiment 18 is the system of any of embodiments 15 through 17, wherein the three-dimensional data representation is a point cloud or three-dimensional data convertible into a point cloud.

[0131] Embodiment 19 is the system of any of embodiments 15 through 18, wherein the captured images are structured light patterned images that encode the surface of the physical object.

[0132] Embodiment 20 is the system of any of embodiments 15 through 19, wherein the first dataset is derived from a preoperative computed tomography scan or magnetic resonance imaging scan.VIS-007-W01

[0133] Embodiment 21 is the system of any of embodiments 15 through 20, wherein: the computer system further comprises a user interface and a display device, and the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display, on a display device, the tracked three-dimensional data representation of the femur.

[0134] Embodiment 22 is the system of any of embodiments 15 through 21, wherein tracking the femur relative to the common coordinate system comprises continuously updating the femur’s position relative to the common coordinate system by repeatedly acquiring surface data, registering the repeatedly acquired surface data in the common coordinate system, and updating transformation matrices.

[0135] Embodiment 23 is a surgical pointer tool, comprising: a rigid body comprising a proximal end comprising a handle and a fiducial element, the fiducial element having a surface topology configured for optimized localization by a surgical navigation system using a 3D scanner; a distal end including a pointer tip, and an elongated shaft extending between the proximal end and the distal end; wherein the fiducial element has a distinct orientation relative to the pointer tip such that the orientation of the fiducial element reliably indicates the position and orientation of the pointer tip.

[0136] Embodiment 24 is the surgical pointer tool of embodiment 23, wherein the fiducial element comprises a finial.

[0137] Embodiment 25 is the surgical pointer tool of embodiments 23 or 24, wherein the fiducial element comprises a plurality of spheres.

[0138] Embodiment 26 is the surgical pointer tool of any of embodiments 23 through 25, wherein the fiducial element comprises a unique geometric shape.

[0139] Embodiment 27 is the surgical pointer tool of any of embodiments 23 through 26, wherein the elongated shaft includes a curved portion.

[0140] Embodiment 28 is the surgical pointer tool of any of embodiments 23 through 27, wherein the curved portion is located near the distal end.VIS-007-W01

[0141] Embodiment 29 is the surgical pointer tool of any of embodiments 23 through 28, wherein the elongated shaft has a linear configuration.

[0142] Embodiment 30 is the surgical pointer tool of any of embodiments 23 through 29, wherein the pointer tip comprises a blunt tip.

[0143] Embodiment 31 is the surgical pointer tool of any of embodiments 23 through 30, wherein the rigid body is composed of biocompatible material.

[0144] Embodiment 32 is a system for instrument-based anatomical point location using topology registration, comprising: a structured light three-dimensional scanner, including: a light source: a liquid crystal matrix; and an image capture device having a field of view; a surgical pointer tool comprising a rigid body having a proximal end comprising a handle and a fiducial element, a distal end including a pointer tip, and an elongated shaft extending between the proximal end and the distal end; a computer system including a memory and at least one processor; and computer readable media embodied in a non-transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a three-dimensional data representation of a topological surface of a physical object using a first dataset, the first dataset derived from one or more images captured by the image capture device without optical trackers or applied fiducials; register the three- dimensional data representation to a second dataset such that the three-dimensional data representation and the second dataset are aligned within a common coordinate system, the registration occurring without said optical trackers or applied fiducials; and track the physical object relative to the common coordinate system without said optical trackers or applied fiducials; determine a location of the pointer tip of the surgical pointer tool positioned outside of the field of view of the image capture device based on the position and orientation of the fiducial element positioned within the field of view of the image capture device.

[0145] Embodiment 33 is the system of embodiment 32, wherein the light source is a linear light source.

[0146] Embodiment 34 is the system of embodiments 32 or 33, wherein the fiducial element has a surface topology configured for optimized localization the structured light three-VIS-007-W01 dimensional scanner.

[0147] Embodiment 35 is the system of any of embodiments 32 through 34, wherein the three-dimensional data representation is a point cloud or three-dimensional data convertible into a point cloud.

[0148] Embodiment 36 is the system of any of embodiments 32 through 35, wherein the captured images are structured light patterned images that encode the surface of the physical object.

[0149] Embodiment 37 is the system of any of embodiments 32 through 36, wherein the physical object is an anatomical structure in a surgical procedure.

[0150] Embodiment 38 is the system of any of embodiments 32 through 37, wherein the three-dimensional data representation of the topological surface of the physical object is constructed by: decoding light patterns contained in the one or more captured images by analyzing brightness history to create camera pixel correspondence to light angle projection; using triangulation to calculate a three-dimensional coordinate for each pixel in the one or more captured images based on the decoded light patterns; and combining the triangulated three- dimensional coordinates of each pixel into the three-dimensional point cloud representation of the surface of the physical object in physical space.

[0151] Embodiment 39 is the system of any of embodiments 32 through 38, wherein the three-dimensional data representation of the physical object comprises a first three-dimensional data representation of a first physical object, and the computer readable media includes a further set of instructions, that, when executed by a processor, cause the computer system to: construct a second digital three-dimensional data representation of a topological surface of a second physical object using a third dataset, the third dataset derived from one or more images of surface topology of the second physical object captured by the image capture device using infrared light and without the use of optical trackers or applied fiducials; register the second three-dimensional data representation to a fourth dataset such that the second three-dimensional data representation and the fourth dataset are aligned with one another and the with the first three-dimensional data representation and second dataset within the common coordinate system without the use ofVIS-007-W01 optical trackers or applied fiducials; and continuously track the second physical object relative to the common coordinate system without the use of optical trackers or applied fiducials.

[0152] Embodiment 40 is the system of any of embodiments 32 through 39, wherein the first physical object is a first anatomical structure and the second physical object is a second anatomical structure.

[0153] Embodiment 41 is the system of any of embodiments 32 through 40, wherein the first physical object is an anatomical structure and the second physical object is a surgical instrument.

[0154] Embodiment 42 is the system of any of embodiments 32 through 41, wherein the surgical instrument is the pointer tool.

[0155] Embodiment 43 is the system of any of embodiments 32 through 42, wherein the second dataset is derived from a preoperative computed tomography scan or magnetic resonance imaging scan.

[0156] Embodiment 44 is the system of any of embodiments 32 through 43, wherein: the computer system further comprises a user interface and a display device, and the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display, on a display device, the tracked three-dimensional data representation of the physical object.

[0157] Embodiment 45 is the system of any of embodiments 32 through 44, wherein tracking the physical object relative to the common coordinate system comprises continuously updating the physical object’s position relative to the common coordinate system by repeatedly acquiring surface data, registering the repeatedly acquired surface data in the common coordinate system, and updating transformation matrices.

[0158] Embodiment 46 is the system of any of embodiments 32 through 45, wherein the surgical pointer tool is positioned such that the pointer tip is touching a target anatomy location.

[0159] Embodiment 47 is the system of any of embodiments 32 through 46, wherein the fiducial element comprises a finial.VIS-007-W01

[0160] Embodiment 48 is the system of any of embodiments 32 through 47, wherein the fiducial element comprises a plurality of spheres.

[0161] Embodiment 49 is the system of any of embodiments 32 through 48, wherein the fiducial element comprises a unique geometric shape.

[0162] Embodiment 50 is the system of any of embodiments 32 through 49, wherein the elongated shaft includes a curved portion.

[0163] Embodiment 51 is the system of any of embodiments 32 through 50, wherein the curved portion is located near the distal end.

[0164] Embodiment 52 is the system of any of embodiments 32 through 51, wherein the elongated shaft has a linear configuration.

[0165] Embodiment 53 is the system of any of embodiments 32 through 52, wherein the pointer tip comprises a blunt tip.

[0166] Embodiment 54 is the system of any of embodiments 32 through 53, wherein the rigid body is composed of biocompatible material.

[0167] Embodiment 55 is the system of any of embodiments 32 through 54, wherein the computer readable media includes a further set of instructions, that, when executed by a processor, cause the computer system to calibrate the location of the pointer tip of the surgical pointer tool with respect to the fiducial element.

[0168] Embodiment 56 is the system of any of embodiments 32 through 55, further comprising a calibration attachment configured to engage the pointer tip during a calibration event.

[0169] Embodiment 57 is the system of any of embodiments 32 through 56, wherein the calibration attachment is configured to engage the pointer tip in a defined seating geometry.

[0170] Embodiment 58 is the system of any of embodiments 32 through 57, wherein the computer readable media includes a further set of instructions, that, when executed by aVIS-007-W01 processor, cause the computer system to: determine a location of an inaccessible anatomical point positioned between two accessible anatomical points, by: determining a first location of the pointer tip when the pointer tip is touching a first portion of patient anatomy; determining a second location of the pointer tip when the pointer tip is touching a second portion of patient anatomy; and calculating a midpoint between the first and second determined locations.

[0171] Embodiment 59 is a system for continuous registration and tracking of objects in an operating room environment, comprising: a structured light three-dimensional scanner, including: a light source: a liquid crystal matrix; and an image capture device; a computer system including a memory and at least one processor; and computer readable media embodied in a non- transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a three-dimensional data representation of a topological surface of a physical object using a first dataset, the first dataset derived from one or more images captured by the image capture device and without optical trackers or applied fiducials; register the three-dimensional data representation to a second dataset such that the three-dimensional data representation and the second dataset are aligned within a common coordinate system, the registration occurring without said optical trackers or applied fiducials; and track the physical object relative to the common coordinate system without said optical trackers or applied fiducials; wherein tracking the physical object relative to the common coordinate system comprises: continuously updating the physical object’s position relative to the common coordinate system by repeatedly acquiring surface data through subsequent image capture events by the image capture device; registering the repeatedly acquired surface data in the common coordinate system; updating transformation matrices; and continuously updating the three-dimensional data representation based on the repeatedly acquired surface data.

[0172] Embodiment 60 is the system of embodiment 59, wherein the light source is a linear light source.

[0173] Embodiment 61 is the system of embodiments 59 or 60, wherein the three-dimensional data representation is a point cloud or three-dimensional data convertible into a point cloud.

[0174] Embodiment 62 is the system of any of embodiments 59 through 61, wherein the captured images are structured light patterned images that encode the surface of the physicalVIS-007-W01 object.

[0175] Embodiment 63 is the system of any of embodiments 59 through 62, wherein the physical object is an anatomical structure in a surgical procedure.[00176J Embodiment 64 is the system of any of embodiments 59 through 63, wherein the three-dimensional data representation of the topological surface of the physical object is constructed by: decoding light patterns contained in the one or more captured images by analyzing brightness history to create camera pixel correspondence to light angle projection; using triangulation to calculate a three-dimensional coordinate for each pixel in the one or more captured images based on the decoded light patterns; and combining the triangulated three- dimensional coordinates of each pixel into the three-dimensional point cloud representation of the surface of the physical object in physical space.

[0177] Embodiment 65 is the system of any of embodiments 59 through 64, wherein the second dataset is derived from a preoperative computed tomography scan or magnetic resonance imaging scan.

[0178] Embodiment 66 is the system of any of embodiments 59 through 65, wherein: the computer system further comprises a user interface and a display device, and the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display, on a display device, the tracked three-dimensional data representation of the physical object.BRIEF DESCRIPTION OF THE DRAWINGS

[0179] Many advantages of the present disclosure will be apparent to those skilled in the art with a reading of this specification in conjunction with the attached drawings, wherein like reference numerals are applied to like elements and wherein:

[0180] Fig. 1 illustrates an example of a prior art rigid transformation matrix for transferring a 3D data set from one coordinate system to another;

[0181] Fig. 2 is a flowchart depicting an example of a typical prior art workflow for a robotically assisted total knee arthroplasty;VIS-007-W01

[0182] Fig. 3 is a block diagram illustrating essential features of a continuous anatomical automatic tracking (CAAT) system of the present disclosure, according to some embodiments;

[0183] Fig. 4 is a perspective view of an example of a surgical environment utilizing the CAAT system of Fig. 3, according to some embodiments;

[0184] Fig. 5 is a perspective view of an example of a 3D scanner forming part of the CAAT system of the Fig. 3, according to some embodiments;

[0185] Fig. 6 is a perspective view of the 3D scanner of Fig. 5 with the cover removed, according to some e embodiments;

[0186] Fig. 7 is a plan view of the 3D scanner of Fig. 6, according to some embodiments;

[0187] Fig. 8 is another perspective view of the 3D scanner of Fig. 6, according to some embodiments;

[0188] Fig. 9 is an perspective view of an example of a rigid scaffolding forming part of the 3D scanner of Fig. 5, according to some embodiments;

[0189] Fig. 10 is an exploded view of an example of a light source assembly forming part of the 3D scanner of Fig. 5, according to some embodiments;

[0190] Figs. 11-13 are perspective, bottom plan, and side plan views, respectively, of an example of a pattern generator forming part of the 3D scanner of Fig. 5, according to some embodiments;

[0191] Fig. 14 is a partial exploded view of an example of a thermal management system forming part of the 3D scanner of Fig. 5, according to some embodiments;

[0192] Fig. 15 is a block diagram depicting an example of a data acquisition process using the CAAT system of Fig. 3, according to some embodiments;

[0193] Fig. 16 is an example of reconstructed point cloud of an exposed knee, simulated with sawbones and using a visible light source generated using the CAAT system of Fig. 3, according to some embodiments;VIS-007-W01

[0194] Figs. 17A-B depicts a typical prior art stereoscopic camera field of view (on the left) and what the cameras actually capture (on the right), according to some embodiments;

[0195] Figs. 18A-B depicts an image and 3D point cloud from a 3D scanner forming part of the CAAT system of Fig. 3, using infrared light instead of visible light, according to some embodiments;

[0196] Fig. 19 depicts a 3D point cloud from a 3D scanner forming part of the CAAT system of Fig. 3, using infrared light instead of visible light and shown at an oblique angle, which illustrates in particular the 3D nature of the regeneration, according to some embodiments;

[0197] Fig. 20 depicts the 3D point cloud of Fig. 60, with some anatomical structures segmented for easy identification, according to some embodiments;

[0198] Fig. 21 is a block diagram depicting an example of a method of registration of anatomical structures and / or surgical instruments using the CAAT system of Fig. 3, according to some embodiments;

[0199] Fig. 22 illustrates three data sets registered into the same coordinate space using the CAAT system of Fig. 3, particularly CT data from the femur and tibia registered to a 3D topological scan, according to some embodiments;

[0200] Fig. 23 is a block diagram depicting an example of a method of calculating a hip center of rotation using the CAAT system of Fig. 3, according to some embodiments;

[0201] Fig. 24 is a block diagram depicting an example of a method of assessing gaps between femoral condyles (e.g., medial and lateral) and a tibial plateau using the CAAT system of Fig. 3, according to some embodiments;

[0202] Fig. 25 is a block diagram illustrating several steps of an example method of calculating joint gaps based on changes in alignment between the femur and tibia under applied stress using the CAAT system of Fig. 3, according to some embodiments;VIS-007-W01

[0203] Fig. 26 is a block diagram depicting an example of a user interface of the CAAT system of Fig. 3, illustrating in particular a baseline orientation of a knee joint, according to some embodiments;

[0204] Fig. 27 is a block diagram depicting an example of a user interface of the CAAT system of Fig. 3, illustrating in particular a knee joint under applied varus stress, according to some embodiments;

[0205] Fig. 28 is a block diagram depicting an example of a user interface of the CAAT system of Fig. 3, illustrating in particular a knee joint under applied valgus stress, according to some embodiments;

[0206] Fig. 29 is a block diagram depicting an example of a method of tracking movable anatomy and a movable surgical instrument configured to affect the movable anatomy using the CAAT system of Fig. 3, according to some embodiments;

[0207] Fig. 30A-B is a block diagram depicting an example of a method of complex anatomy tracking with robot-controlled instrument in the context of a Total Knee Arthroplasty (TKA) using the CAAT system of Fig. 3, according to some embodiments;

[0208] Fig. 31 is a block diagram depicting an example robotically assisted TKA workflow with pre-operative imaging using the CAAT system of Fig. 3, according to some embodiments;

[0209] Fig. 32 is a block diagram depicting an example robotically assisted TKA workflow without pre-operative imaging using the CAAT system of Fig. 3, according to some embodiments;

[0210] Fig. 33 is a perspective view of an example of a pointer instrument configured for use with the CAAT system of Fig. 3, according to some embodiments;

[0211] Fig. 34 is a block diagram depicting several steps in an example method for identifying specific anatomical points that are beyond the scanner’s field of view without performing calculations on visible anatomy or using optical trackers using the CAAT system of Fig. 3, according to some embodiments;VIS-007-W01

[0212] Fig. 35 is a block diagram depicting several steps of an example method for real-time verification of the positioning and orientation of an anatomical structures using the CAAT system of Fig. 3, according to some embodiments;

[0213] Fig. 36 is a block diagram illustrating several steps of an example method for real-time verification of osteotomies and bone resections using the CAAT system of Fig. 3, according to some embodiments;

[0214] Figs. 37 and 38 are block diagrams illustrating example computer systems with which any of the devices or systems described herein may be implemented, according to some embodimentsDETAILED DESCRIPTION OF A PREFERRED EMBODIMENT

[0215] Illustrative embodiments of the disclosure are described below. In the interest of clarity, not all features of an actual implementation are described in this specification. It will of course be appreciated that in the development of any such actual embodiment, numerous implementation-specific decisions must be made to achieve the developers’ specific goals, such as compliance with system-related and business-related constraints, which will vary from one implementation to another. Moreover, it will be appreciated that such a development effort might be complex and time-consuming but would nevertheless be a routine undertaking for those of ordinary skill in the art having the benefit of this disclosure. The surgical navigation system and related methods disclosed herein boasts a variety of inventive features and components that warrant patent protection, both individually and in combination.

[0216] By way of example, this disclosure introduces a novel system and method for registering and tracking anatomical structures during surgical procedures by leveraging continuously updated 3D topological data of the surgical field. Unlike traditional systems that rely on physical optical markers (OMs) and Optical Tracker Constellations (OTCs), the system and method disclosed herein utilizes the data captured by a 3D scanner to perform Continuous Anatomic Auto Tracking (CAAT). Hereinafter, the system disclosed herein may be referred to as “CAAT system 10”. While shown and described herein within the specific context of a surgical application (specifically a TKA procedure), the CAAT system 10 including the 3DVIS-007-W01 scanner 12 may be used in any scenario (surgical, medical, or otherwise) to generate a reconstructed 3D model of an object.

[0217] Referring to Fig. 3, and by way of example, the CAAT system 10 disclosed herein may feature a 3D scanner 12, hub 14, user interface 16, and distributed computation 18. By way of example, the CAAT system 10 also includes a software program 20 embodied in a non- transitory computer readable storage medium (e.g., the hub 14) and comprising a set of instructions configured to enable the computer to perform the various tasks described herein. In some embodiments, one or more peripheral computing devices 21 may be associated with the CAAT system 10, for example to integrate use and control of certain medical and surgical equipment that may be used in robotics-driven procedures. Such peripheral computing device(s) 21 may be configured to be controlled by the hub 14 and / or user interface 16.

[0218] In some embodiments, the 3D scanner 12 continuously captures topological data of one or more objects within a field of view at a high repetition rate, providing detailed surface information of anything within the field of view (e.g., anatomical structures, surgical equipment, surgical tools, surgical staff, etc.,) and ensuring uninterrupted tracking during dynamic surgical procedures. While the 3D scanner 12 disclosed herein is optimized for this application, the CAAT system 10 is compatible with any 3D scanner capable of optically generating topological information from the surgical site. In some embodiments, the 3D scanner 12 may be enclosed in a sterile drape to maintain sterility, allowing the 3D scanner 12 to function within the sterile surgical environment.

[0219] In some embodiments, the hub 14 is a dedicated computer or computer system responsible for receiving raw image data from the 3D scanner 12. Equipped with specialized hardware and software, the hub 14 processes this data in real-time, converting the data into a usable format for tracking anatomical structures and surgical instruments. In some embodiments, the CAAT system 10 performs automatic registration of the data to a Surgical Coordinate System (SCS), eliminating the need for manual optical trackers or registration steps. In some embodiments, the hub 14 may comprise multiple computer systems that communicate with one another so that data can be processed serially or in parallel on separate hardware.VIS-007-W01

[0220] In some embodiments, the hub 14 transmits transformation matrices for each tracked object to a separate or second computer system, where the data is further processed based on the surgical workflow. This information is presented to the surgeon in an intuitive, user-friendly interface 16, providing real-time visualization of anatomy and surgical tools. In some embodiments, the interface guides the surgeon through key actions such as implant positioning, bone cuts, and other critical tasks.

[0221] In some embodiments, while the hub 14 plays a central role in data processing, the CAAT system 10 architecture allows for flexibility in where computations occur. For example, in some embodiments, some processing tasks, such as initial data filtering or compression and / or reconstruction of a 3D point cloud, may be performed by a processor within the 3D scanner 12 itself to reduce data transmission loads. Additionally, certain computations may be handled by other computers within the system, either in series or parallel with the hub 14, depending on the complexity of the procedure and the system's overall configuration. In some cases, specific computational tasks, such as visualization rendering or more advanced data analysis, may be managed directly within the user interface (UI) 16 system. This distributed computation 20 approach enables scalable performance and ensures that real-time requirements are met across a range of surgical applications.

[0222] 3D Scanner

[0223] Fig. 4 illustrates an example of a surgical environment 22 which may be a sterile field within an operating room at a hospital or surgical center, in which a patient 24 is positioned on an operating table 26 for a surgical procedure such as a Total Knee Arthroplasty (TKA). By way of example, a surgical instrument 28 (e.g., drill, oscillating saw, etc.) may be provided at the end of a first robotic arm 30, which itself is attached to a first cart 32. A 3D scanner 12 forming part of the CAAT system 10 is provided at the distal end (or “end effector”) of a second robotic arm 34 which is provided on a second cart 36. It should be noted that this surgical environment 22 is provided by way of example only to illustrate one possible setup and positioning of the 3D camera in relation to a patient during a surgical procedure. In some embodiments, the second robotic arm 34 to which the 3D scanner 12 is attached may be attached to a wall, ceiling, immobile table, or other structure.VIS-007-W01

[0224] Figs. 5-8 illustrate an example of a 3D scanner 12 configured for use with the CAAT system 10 of the present disclosure, according to some embodiments. In some embodiments, the 3D scanner 12 comprises an optical system 42, a light source assembly 44, a thermal management system (TMS) 46, a cover 48, and a rigid scaffolding 50. In some embodiments, the optical system 42 includes a camera, lens, and filter, and is configured for continuously capturing data including high repetition images of patterns projected onto the surgical field (from which topological data is derived and used in reconstructing a 3D point cloud) and communicating the captured data to the hub 14. In some embodiments, the light source assembly 44 is configured to project a structured light pattern onto the target surface, which the CAAT system 10 may then analyze as described below to generate a 3D model of the surface area with the optical system’s field of view, as described herein. In some embodiments, the TMS 46 is configured to remove heat from the 3D scanner 12. More specifically the TMS 46 removes heat generated by the light source and the optical system 42 by using airflow to transfer the heat through a heat sink and ultimately out of the 3D scanner 12 through ventilation tubes 178, as described in detail below. The TMS 46 shown and described herein is representative of one example of removing heat from the 3D scanner 12. However, the 3D scanner 12 may be modified to incorporate other methods of removing heat in addition to or instead of the TMS 46 described herein.

[0225] In some embodiments, the optical system 42 has a field of view 68 with a central axis Li, and the light source assembly 44 has an illumination field 80 with a central axis £2. In some embodiments, the optical system 42 and light source assembly 44 may be positioned at an angle 0i relative to one another defined as the offset angle between the central axis Li of the optical system 42 and the central axis L2 of the light source assembly 44, as shown by way of example in Fig. 7. This arrangement produces an overlap area (or “working volume”) 69 in which the field of view 68 overlaps with the illumination field 80. Objects within the overlap area 69 may be scanned by the 3D scanner 12. Objects that are not in the overlap area 69 will not be scanned by the 3D scanner 12. In some embodiments, the angle 0i may also be defined as the apex angle in a triangulation between the optical system 42, the light source assembly 44, and the target site. By way of example, the apex location in the triangulation represents the optimal point within the overlap area 69 in which the optical system 42 produces the best imaging results. In someVIS-007-W01 embodiments, the optical system 42 and light source assembly 44 may be positioned such that the angle 0i may be within a range of 10° to 30°. In some embodiments, the optical system 42 and light source assembly 44 may be positioned such that the angle 0i may be within a range of 10° to 20°. In some embodiments, the optical system 42 and light source assembly 44 may be positioned such that the angle 0i = 16.5°. In some embodiments, the 3D scanner 12 may be configured such that the angle 0i may be adjustable during use, for example to adjust the working volume 69 of the 3D scanner 12 or if the positioning of the 3D scanner 12 relative to the target site needs to be moved. In some embodiments, the 3D scanner 12 may be configured such that the focus is operable by a motor. In some embodiments, the 3D scanner 12 may be configured such that the focus is automatically adjustable. In some embodiments, the 3D scanner 12 may be configured such that the working volume 69 may be adjusted by changing the lateral distance between the optical system 42 and the light source assembly 44. In some embodiments, the 3D scanner 12 may be configured such that the focus may be adjusted by changing angle 0i between the optical system 42 and the light source assembly 44. In some embodiments, the 3D scanner 12 may be configured to include an adjustable and / or motorized iris to enable a user to fine tune the level of light entering the camera.

[0226] Fig. 9 illustrates an example of an optical system 42 forming part of the CAAT system 10 of the present disclosure, according to some embodiments. In some embodiments, the optical system 42 may comprise a camera 60, lens 62, filter 64, and a data connection 66. In some embodiments, the camera 60 may be any compatible camera available in the market currently or in the future. In some embodiments, the camera 60 may have a silicon sensor configured to collect the light. In some embodiments, the sensor may be selected to have a high sensitivity to a specific range of wavelengths of light, including but not limited to (and by way of example only) a range of 750-850 nm, or 750-800 nm, or 800-850 nm. In some embodiments, the camera 60 has a field of view 68 defined as the area in front of the camera lens 60 and / or filter 64 that is captured by the camera. Points of interest within the field of view 68 may be captured by the camera 60 and transmitted as data to the hub 14. In some embodiments, the data may be sent directly to the hub 14 by way of the data connection 66. In some embodiments, the data may be sent indirectly to the hub 14 by way of an intermediate component, which may process the data before sending the data to the hub 14. Points of interest that are outside of the field of view 68VIS-007-W01 will not be captured by the camera 60. In some embodiments, the lens 62 extends from the camera 60 and may be configured to have an adjustable and / or motorized field of view range. Alternatively, the lens 62 may have a fixed field of view and changing the lens 62 would then change the field of view.

[0227] In some embodiments, the filter 64 may be a high pass wavelength filter or spectral filter. In some embodiments, the filter 64 may be selected and / or configured to block light wavelengths below a predetermined threshold value from passing through the filter 64. In some embodiments, the predetermined threshold value may be 800 nm such that wavelength values below 800 nm are blocked and wavelength values above 800 nm (e.g., including infrared light) are able to pass through the filter 64. In some embodiments, the predetermined threshold value may be 750 nm. In some embodiments, the filter 64 may comprise a band pass filter configured to block light wavelengths below a minimum predetermined threshold value and above a maximum predetermined value from passing through the filter 64 such that light having wavelength values within a specified range may pass through the filter 64. In some embodiments, the specified range may be 800-850 nm. In some embodiments, the specified range may be 810-840 nm. By way of example, such specified ranges may block visible light (e.g., light wavelengths below 800 nm) and higher light wavelengths (e.g., light wavelengths above 850 nm) from passing through the filter 64. This helps to maximize the dynamic range of the sensor to capture the pattern projected. For example, in a surgical environment, a NIR wavelength range would not be visible to the surgeon or other staff and the ambient lighting within the operating room would be blocked by the filter, allowing only light reflecting off the target area to be allowed into the camera.

[0228] Fig. 10 illustrates an example of a light source assembly 44 forming part of the CAAT system 10 of the present disclosure, according to some embodiments. In some embodiments the light source assembly 44 includes a linear light source 70, a light shield 72, a light box 74, a transmissive LCD (or pattern generator) 76, and a pattern generator cover 78. In some embodiments, light generated by the light source 70 passes through the light shield 72, light box 74, and pattern generator 76 on its way to the surgical target site. In some embodiments, the light source assembly 44 has an illumination field 80 defined as the maximum possible area in front of the light source assembly 44 that may be illuminated by light emanating from the linearVIS-007-W01 light source 70. In some embodiments, the linear light source 70 comprises an elongated emission component configured to emit light through the light shield 72 and light box 74. In some embodiments, the emitted light may have a wavelength value within a specified range. In some embodiments, the specified range may be 800-850 nm. In some embodiments, the specified range may be 810-840 nm. In some embodiments, the linear light source 70 comprises a plurality of LED emitters mounted to a chip-on-board (COB) printed circuit board (hereinafter “COB 82”) in a linear orientation. In some embodiments, the linear light source 70 may comprise one or more groups of LED emitters, with each of the one or more groups of LED emitters independently powered with voltage and current. In some embodiments, the linear light source 70 may comprise 10 groups of 21 LED emitters.

[0229] Figs. 11-13 illustrate an example of a pattern generator 76 forming part of the 3D scanner 12, according to some embodiments. In some embodiments, the pattern generator 76 comprises a transmissive LCD screen with individually controllable opacity comprising a liquid crystal (LC) layer 140 positioned between first and second polarizing layers 142, 144[00230J In some embodiments, the first polarizing layer 142 ensures light entering the LC layer 140 is polarized. The LC layer’s twist (controlled by voltage) determines whether light’s polarization matches the second polarizer, and how much light is allowed through the LC layer 140. The second polarizing layer 144 acts as a gate that only passes light correctly rotated by the LC layer 140. This arrangement enables each energized line 150 within the LCD layer 140 to precisely modulate light intensity, forming a distinct and sharp pattern of edges. In some embodiments, the first and second polarizing layers 142, 144 may be near infrared (NIR) polarizers (e.g., which allow near infrared light through but block most visible light) instead of more common optically transmissive polarizers. In some embodiments polarizers may also allow for transmission of some visible light (though not as well as the usual polarizers), allowing for dual functionality in the event that the 3D scanner 12 is configured to have both NIR and visible lights present on a single unit.

[0231] Fig. 14 illustrates an example of a thermal management system (TMS) 46 forming part of the 3D scanner 12, according to some embodiments. By way of example, the TMS 46 is provided to remove heat from the major heat sources in the 3D scanner 12, namely the linearVIS-007-W01 light source 70 and the camera 60 to prevent overheating of the light source 70 and the camera 60. In some embodiments, the TMS 46 may include a heat transfer assembly 170, a heat sink 172, an air box 174, a vent box 176, and one or more ventilation tubes 178. In some embodiments, heat is transferred from the linear light source 70 and the camera 60 to the heat sink 172 by the heat transfer assembly 170. Meanwhile, airflow is forcibly directed into one of the ventilation tubes 178 and / or forcibly pulled out of the other of the ventilation tubes 178 by fans that are at distal ends of the ventilation tubes which may be located outside of the sterile field (e.g., external of the sterile drape). In some embodiments, the forcible pushing and forcible pulling operate in concert so that the airflow is in one direction through the TMS 46. In some embodiments, only pulling fans may be used. In some embodiments, only pushing fans may be used. Heat that is pulled from the linear light source 70 and the camera 60 is directed into the heat sink 172, wherein it is transferred to the air that is directed through the heat sinkl72. This heated air is then directed out of the TMS 456 through a ventilation tube 178.

[0232] Scanning and Data Acquisition Process[00233J As shown in Fig. 4, by way of example, the 3D scanner 12 may be positioned to continuously capture the surgical field. In some embodiments, as the surgeon manipulates instruments and interacts with the patient’s anatomy, the 3D scanner 12 generates a point cloud of the visible surfaces within the overlap area 69 (e g., the surfaces that are within both the illumination field 80 of the light source assembly 44 and the field of view 68 of the optical system 42). In some embodiments, assuming data is collected at a rate exceeding 20 Hz, the data stream may be continuously updated in real-time. While faster rates are preferred to reduce system latency, rates above 20 Hz are generally sufficient for functional real-time performance. In some embodiments, the point cloud data is then used to create transforms that can be used to update the position and orientation of 3D reference geometry of the anatomy and any tracked instruments within the Surgical Coordinate System (SCS).

[0234] Fig. 15 is a block diagram depicting a data acquisition process 250 using a 3D scanner 12 described herein. While the 3D scanner 12 disclosed herein introduces several novel features, the general principles of structured light scanning are well established and are described here forVIS-007-W01 clarity. Furthermore, this data acquisition process 250 may be used with other scanners, such as a shadow caster of the type disclosed in in the ‘973 app.

[0235] In some embodiments, a first step 252 in the data acquisition process 250 is to generate and project a structured light pattern onto a surface to be scanned. For example, in some embodiments, the 3D scanner 12 may illuminate the target surface with a predefined light pattern (e.g., commonly grids, stripes, or dots) projected onto the surfaces being scanned. By way of example, this may be accomplished by configuring the pattern generator 76 described above to produce the desired structured light pattern. In some embodiments, the 3D scanner 12 described herein may be configured to use infrared light, ensuring that the operating room lighting does not interfere with the scanning process. In some embodiments, a series of light patterns is projected to encode all the data necessary for reconstructing the 3D point cloud. Multiple pattern series can be used, depending on the specific requirements of data encoding. In some embodiments, variations in intensity, color, or phase may be applied to the pattern to enhance data accuracy or accelerate acquisition.[00236J In some embodiments, a second step 254 in the data acquisition process 250 is image capture. For example, in some embodiments, cameras or sensors within the 3D scanner 12 (e.g., optical system 42 described above) capture the reflected structured light as it interacts with the target surface. In some embodiments, an image may be taken for each pattern projected in the series. By way of example, the captured images reflect deformations in the light pattern caused by the surface's contours and depth. In some embodiments, these images may be collected by one or multiple cameras, with the relative positions and orientations of the cameras being known to the CAAT system 10. In some embodiments, the first step 252 of generating and projecting structured light pattem(s) onto a surface to be scanned and the second step 254 of image capture may be repeated in an iterative loop until all the required images have been acquired.

[0237] In some embodiments, once all required images have been obtained by way of the previous steps, a third step 256 in the data acquisition process 250 is reconstruction of the 3D surface, for example within the hub 14. In some embodiments, reconstruction of the 3D surface may occur on board the 3D scanner 12 in a main control board rather than in a separate hub 14, in which case the following sub-steps also occur within the main control board of the 3D scannerVIS-007-W0112. In some embodiments, the reconstruction step 256 includes sub-steps of pattern decoding, triangulation and 3D point cloud generation. In some embodiments, after image capture, the images are sent to the hub 14 for processing. In some embodiments, a first sub-step 258 of the reconstruction step 256 is pattern decoding. By way of example, data can be transferred from the 3D scanner 12 to the hub 14 continuously as images are captured, or in bulk once an entire pattern series is complete. In some embodiments, the reconstruction step 254 cannot be completed until all captured images are transferred to the hub 14, but some computation can be completed on each image as it arrives in the hub 14. In some embodiments, the CAAT system 10 is configured to decode the captured light patterns by analyzing their brightness history to create camera pixel correspondences to light angle projection. This, together with known pixel observation ray, enable via triangulation in-depth information, allowing the CAAT system 10 to reconstruct the 3D surface as a 3D point cloud.

[0238] In some embodiments, a second sub-step 260 of the reconstruction step 256 is triangulation. By way of example, the CAAT system 10 may use triangulation to calculate the 3D coordinates of each point where the light interacts with the surface. With the known positions of the linear light source 70, camera 60, and pattern generator 76, the CAAT system 10 determines a 3D point for each pixel in the captured images.

[0239] In some embodiments, a third sub-step 262 of the reconstruction step 256 is generation of a 3D point cloud 268 representing the scanned target surface. By way of example, the triangulated points from each frame may then be combined into a 3D point cloud 268, representing the surface in 3D space, with each point assigned x, y, and z coordinates. An example of a 3D point 268 cloud is shown in Fig. 16. In some embodiments, depending on system capabilities, additional data such as color or intensity may be included to enhance detail and visual fidelity.

[0240] In some embodiments, a fourth step 264 in the data acquisition process 250 is data processing and filtering. For example, in some embodiments, the raw point cloud data may undergo post-processing to reduce noise, fill gaps, and improve accuracy. In some embodiments, if multiple scans are taken from different angles, the data may be aligned, merged, or "stitched" together to create a cohesive 3D model.VIS-007-W01

[0241] In some embodiments, a fifth step 266 in the data acquisition process 250 is to output the data such that the final point cloud 268 is made available for further processes, such as segmentation and registration, which can be used to guide the surgical workflow.

[0242] Differences between optical tracker generated data and 3D scanned topology data

[0243] It is important to distinguish between the data generated by Optical Markers (OMs) and Optical Trackers (OTs) (e.g., as used in prior art system) versus the data produced through 3D scanning of a surgical site, as disclosed herein.

[0244] By way of example, in Figs. 17A-B, the image on the left (Fig. 17A) depicts a typical TKA surgery, showing externally applied optical trackers 270 attached to the robot 272, femur 274, and tibia 276. Each externally applied optical tracker 270 comprises four optical markers 278 in a predetermined arrangement. This image represents the typical field of view for stereoscopic systems. On the right (Fig. 17B), the image shows what one of the stereoscopic cameras captures after segmenting the image to isolate the OMs 278 (which by way of example appear as black markers on a white background). Clearly visible are three OTs 270, each consisting of four OMs 278. When two such images, taken from slightly different angles, are combined via triangulation, the OTs 270 can be located in space, inferring the positions of the bones and robotic instruments registered to their respective trackers.

[0245] In contrast, the data collected by a 3D topological scanner such as 3D scanner 12 presents a very different picture. By way of example, Figs. 18A-B illustrates images from the 3D scanner 12. By way of example, the image on the left (Fig. 18A) is a scanner image 280 captured by the 3D scanner 12 using infrared light, and the image on the right (Fig. 18B is a 3D point cloud 268 generated from the whole pattern series of collected image data.

[0246] For example, one key difference is the field of view. For a stereoscopic system to capture all the necessary OTs 270 during surgery, the field of view must be quite large, often requiring the cameras to be positioned several meters away, typically at an angle from the side and slightly above the surgical area. From this vantage point, the bones are often not visible to the naked eye, as they may be obstructed by surgeons, instruments, other tissues, etc. By contrast, the 3D scanner 12 provides a much narrower field of view, captured from a closer, top-VIS-007-W01 down perspective, for example as shown in Fig. 4. Moreover, the OTs are some distance away from the anatomy and errors localizing the trackers are amplified by that distance, so that the error is greater at the anatomy. In contrast, when the CAAT system 10 registers and tracks the anatomy itself, that distance does not exist, thereby potentially resulting in reduced relative localization errors when compared to tracker-based systems.

[0247] Another key difference is that the data density is markedly different between the two systems. By collecting points one by one, the stereoscopic system provides sparse data, providing around 50 possible data points on the surface of a femur, for example. This must be collected manually by using a pointer to touch a point and then a foot pedal may be activated to capture the point. This is time consuming and can be error prone if the pointer is moving. Also, on the articular surface of the femur and tibia, if the surgeon is trying to register to preoperative CT bone, then the pointer must be sharp and penetrate the cartilage fully without penetrating the bone. Conversely, the 3D scanner 12 of the present disclosure captures a much more robust data set without penetrating patient anatomy. For example, the image shown in Fig. 18B has approximately 4 megapixels, with about 300,000 individual points on the femur and 200,000 on the tibia. Additional points correspond to surrounding soft tissues. In some embodiments, a depth fdter may be applied to the data to remove much of the background (such as the blue surgical drape).

[0248] By way of example, to further illustrate the 3D nature of the data, Fig. 19 presents the scan data (e.g., point cloud 268) from an oblique angle, giving a clearer sense of its depth. By way of example, for users familiar with knee anatomy, identifying key anatomical structures is relatively straightforward. However, for clarity, Fig. 20 highlights several major anatomical landmarks, by way of example, including the distal end of the femur 274, the anterior cruciate ligament (ACL) 282, the proximal end of the tibia 276, the patellar tendon 284, and a side view of the patella 286. Other points correspond to skin, muscle, fascia, and fat, though they are not specifically labeled. Note that these images use simulated tissue models commonly used for research, not real tissue.

[0249] By way of example, in real tissue, the visual characteristics will vary due to differences in how materials interact with light. These images are representative, but real tissueVIS-007-W01 tend to introduce more noise, particularly in tissues that are partially transparent to infrared light. In some embodiments, this increased noise can be mitigated by applying more aggressive filtering algorithms, for example. In some embodiments, to help define a more reliable shape at one point in a procedure, a wavelength may be used that has a shorter optical penetration length. That wavelength may overlap with other sources of light in the operating room, but may accommodated for the sake of an initial accurate view for reference with subsequent scans taken at wavelengths that introduce more noise, but that do not compete with other sources of light in the operating room. In addition, it may also be a relatively narrow-band source (e.g. 5 to 10 nm bandwidth) that has a relative brightness at that wavelength much greater than that of the room light, such that spectrally filtering for that wavelength results in high-contrast scans with minimal competition with the ambient light.

[0250] Registration Using Topological Data

[0251] Once topological data is available in the form of a 3D point cloud 268, a wide range of capabilities become possible. In some embodiments, multiple 3D point clouds may be aligned and / or combined within a single coordinate space for a more accurate and efficient surgery. By way of example, this aligning and / or combining may occur by way of a process called “registration”, which aligns two or more 3D point cloud datasets, combining them into a common coordinate space. This registration process allows different datasets — such as preoperative CT scans and intraoperative 3D scans — to be aligned and even combined, enabling more precise surgical planning and execution. By way of example, Fig. 21 illustrates an example registration process 290 that may be performed by the CAAT system 10. In some embodiments, the registration process 290 comprises several steps including but not necessarily limited to preprocessing the data for accuracy 292, segmentation 294, registration of anatomical structures 296, registration of instruments and equipment 298, and making precise measurements 300. Although presented as a block diagram, many of these “steps” may be performed in any order, in series or parallel, and / or independently of one another. For example, anatomic registration and instrument registration may occur in any order, independent of one another, or one may occur without the other occurring.VIS-007-W01

[0252] In some embodiments, a first step 292 in the registration process 290 using topological point cloud data is pre-processing the raw point data cloud to ensure accuracy and reliability. In some embodiments, this pre-processing step 292 may include several sub-steps including noise reduction 301, smoothing 302, upsampling 303 the 3D point cloud, and / or downsampling 304 the 3D point cloud. By way of example, noise in the data can be introduced from various sources, such as light interference or reflections from surgical instruments. In some embodiments, noise reduction techniques 302 like statistical outlier removal and radius-based filtering (for example) are used to clean the dataset, ensuring that only relevant points remain. In some embodiments, smoothing algorithms 303, such as Gaussian filtering or bilateral filtering, may be applied to create a more continuous and realistic surface representation. For example, this step may help to reduce local variations in point locations that are not significant for registration, ensuring a more refined dataset without losing critical anatomical details. In some embodiments, the 3D point cloud may optionally be upsampled 304 by increasing the data points in the point cloud. In some embodiments, the 3D point cloud may optionally be downsampled 304 by decreasing the data points in the point cloud. In some embodiments, further data processing techniques such as outlier removal 305, normal estimation 306, and / or feature extraction 307 may be used.

[0253] In some embodiments, a second step 294 in a registration process 290 using topological point cloud data is segmentation. Segmentation enables the CAAT system 10 to handle multiple objects simultaneously during registration while reducing the computational load. By way of example only, once the point cloud data has been pre-processed, the segmentation step 294 can be applied to isolate different anatomical structures or surgical tools. Segmentation is an optional step but provides significant benefits for both data handling and computational efficiency. One such benefit is that segmentation allows for the identification of different objects within the dataset. For example, all points belonging to the femur can be isolated from the main dataset, while points belonging to the tibia can be similarly separated. Any remaining data can be segmented or discarded if unnecessary. Another benefit of segmentation is related to speed and latency. For example, registering large point clouds can be computationally intensive. By segmenting and reducing the dataset to only the objects of interest, computational requirements are lowered, improving processing speed and reducingVIS-007-W01 overall system latency. In some embodiments, the segmentation step 294 may occur before the pre-processing step 292 to perform segmentation using raw, unfiltered data.

[0254] In some embodiments, various methods of segmentation 294 may be used, including but not limited to manual segmentation 308, traditional algorithms 309, and artificial intelligence (Al) 310 powered segmentation. By way of example, manual segmentation 308 may be useful in complex cases, as operators can manually label objects in the point cloud to isolate key structures. In some embodiments, traditional algorithmic techniques 309 such as edge detection or region-growing may be used to identify structures based on their shape or intensity. In some embodiments, Al-powered segmentation 310 may be used so that pre-trained deep learning models can automatically segment point cloud data in real-time, offering high accuracy and minimal manual intervention. In some embodiments, the Al-powered segmentation 310 may operate within a 2D image space (e.g., a 2D image or 2D depth-map representation) or a 3D space (e.g., point cloud space).

[0255] In some embodiments, a third step 296 in a registration process 290 using topological data is anatomic registration. In most cases, registration involves aligning two datasets, such as a preoperative CT dataset and a 3D topological scan from the surgical site. By way of example only, the alignment may be performed using algorithms such as RANSAC (Random Sample Consensus) 312 for initial, coarse registration and TCP (Iterative Closest Point) 314 for fine- tuning. These algorithms compare corresponding points between datasets and compute a transformation matrix that minimizes alignment errors. For example, RANSAC 312 selects key features in the point cloud, filtering out outliers, and computes an initial transformation matrix by focusing on valid, inlier points, and ICP 314 iteratively refines the resulting transformation matrix by minimizing the distance between corresponding points in the two datasets, leading to precise alignment. The resulting transformation is a 4x4 rigid matrix that translates, rotates, and scales one dataset to align with the other, for example similar to the transformation matrix shown in Fig. 1. In the context of a TKA procedure, for instance, this could align a CT scan with a realtime topological scan of the femur 274 and tibia 276

[0256] In some embodiments, the anatomic registration step 296 can be repeated for multiple objects, such as the femur 274 and tibia 276, allowing simultaneous registration of more than oneVIS-007-W01 object assuming the computational horsepower is available. Fig. 22 illustrates the registration of CT data from the femur 274 and tibia 276, along with a 3D topological scan or point cloud 268, into a common coordinate space 316. By using registered datasets, the CAAT system 10 provides surgeons with accurate measurements that aid in making critical decisions during surgery. By way of example only, Fig. 22 shows the registration of three datasets — CT data from the femur and tibia, and a 3D topological scan — into a common coordinate space for this purpose.

[0257] The registration step 296 is a critical step in aligning two 3D point cloud datasets, such as preoperative CT data with a real-time topological scan in a surgical setting. This method achieves the same results as registration done with Optical Markers (OMs) 278 and Optical Trackers (OTs) 270, but without the need for external tools anchored to bones or an additional transformation to account for the indirect determination of a transformation matrix via the trackers. Additionally, the topological dataset is far denser and more detailed than what a surgeon could collect using a pointer with OT.[00258J In some embodiments, a fourth step 298 in a registration process 290 using topological data is instrument and equipment registration. By way of example, this step is optional and may occur before, during, or independently of anatomic registration in the previous step 296. In addition to anatomical structures, instruments and surgical equipment can be registered using topographical data. By way of example, each instrument has a unique shape that can be used for registration. For instance, a bone saw has a distinct geometry compared to a scalpel, making identification possible in the topological data. In some embodiments, the same registration methods used for anatomy can be applied to instruments, but some challenges exist. For example, instruments with reflective surfaces may not be well-captured in the 3D scan. In some embodiments, treating surfaces to scatter light can reduce specular reflections and improve scan quality. Another challenge is that some materials absorb light, causing poor visibility in the scan. As with specular reflections, in some embodiments treating surfaces to scatter light can mitigate this issue. By way of example, another challenge with instrument and equipment registration is occlusions, in that instruments and / or equipment may be partially obscured by the surgeon's hand. In some embodiments, by extending the handle or otherwise introducing distinct geometry, the CAAT system 10 can still register the instrument even when partially blocked. InVIS-007-W01 some embodiments, distinct geometry may include (but is not limited to) portions of the tracker that are not optical markers, patterns printed on the instruments (e.g., particularly on their most distal end) and / or attachment of a fiducial to the instrument enabling identification and understanding of the instrument position and / or orientation at the surgical site itself.

[0259] In cases where an anatomical point is not directly visible (e.g., the center of the ankle), a pointer instrument can be used. In some embodiments, for example, registering an end of the pointer to the scanned data, the anatomical point can be inferred accurately. In some embodiments, this could be done by registering a proximal end (e.g., that would extend into camera view) of the pointer in the common coordinate system and, based at least in part on the known attributes or the pointer, the relative positioning of the distal tip (e.g., that would be positioned near the target out-of-view anatomical point) would be known or inferred. In this way, moving the scanner in order to change its field of view to see anatomy at the ankle (for example) may be avoided.

[0260] In some embodiments, a fifth step 300 in a registration process 290 using topological data is measurements. For example, once the objects are registered within the same coordinate space, precise measurements can be made, providing crucial information during surgery. These measurements may include, but are not limited to distance measurements, angle measurements, and / or gap measurements. By way of example, measurements made after registrations are relative measurements between two registered objects within the coordinate space or a single registered object and the scanner reference frame in the coordinate space. In some embodiments, the CAAT system 10 can measure the distance between two points on anatomical structures or instruments. For example, measuring the width of the knee’s sulcus is as simple as selecting two points (either manually, or with the points identified by Al or by registration with a model, or with CT), with the system computing the distance in real-time. In some embodiments, more complex measurements, like assessing the varus or valgus angle of the knee, can be performed by identifying the sagittal planes of the femur and tibia in the registered datasets. Once identified, the system calculates the angle between the planes, providing vital information for implant alignment. In some embodiments, if two objects are registered, measurements between the two objects can be computed. For example, in TKA surgeries, the medial and lateral gaps between the femoral condyles and tibial plateau can be measured to assess ligament balance.VIS-007-W01Surgeons often aim to balance these gaps to ensure even tension in the knee, which is critical for the success of the implant.

[0261] Also, by measuring known anatomic features of the knee, careful fitting of a parametric model of the knee may be possible directly for implant selection or even real-time manufacture.

[0262] Tracking with Topological Data

[0263] By way of example, tracking and registration are closely related, but they serve distinct purposes in a surgical setting. As previously described, registration is the process of aligning two or more 3D point cloud datasets, bringing them into a single coordinate system. This is typically a one-time event or a relatively infrequent operation. In contrast, tracking involves continuously registering these datasets over time, monitoring changes in position or orientation, and keeping objects aligned within the same coordinate system. Tracking updates the spatial alignment of registered objects multiple times per second, which allows for dynamic monitoring of anatomical structures and surgical tools as they move throughout the procedure.

[0264] Real-time tracking is essentially registration performed in a continuous loop, where the system rapidly processes new data to reflect movements or changes as they occur. The faster this loop is executed, the more accurately the system can represent the current state of the surgical environment. In some embodiments, and by way of example only, registration uses a dense point cloud, while tracking may use “sparse” scans, which are not the highest pixel density scans, but are much faster to take. The goal of real-time tracking is to create a seamless experience where movements of anatomical structures, instruments, or other objects are captured and reflected without noticeable delay.

[0265] Defining what constitutes "real-time" is challenging because it varies depending on the context and application. In the world of cinema, for example, a standard film is displayed at 24 frames per second (fps). At this rate, the human brain perceives the sequence of still images as continuous motion, which most viewers consider to be real-time. Computer monitors typically operate at higher refresh rates, such as 60Hz or 120Hz, offering smoother visuals. GamingVIS-007-W01 monitors push this further, with refresh rates of up to 240Hz, though this is generally only advantageous in specific, fast-paced applications.

[0266] From a physiological standpoint, studies suggest that the human brain begins to interpolate between frames at speeds around 24 fps, making discrete images appear as a fluid, continuous motion. For the purposes of this disclosure, we define real-time tracking as performing at least 20 registrations per second (rps). This threshold provides a sufficiently smooth experience, making the data appear continuous to the surgeon. Faster registration speeds are always preferable, as they can further reduce latency, but they come with significant tradeoffs in terms of the system’s requirements. Higher speeds increase the demands on light intensity, the speed of image sensors, and computational power to process the data quickly enough to maintain real-time performance.

[0267] In some embodiments, speed may be increased by bypassing the coarse or global registration step (e.g., RANSAC as described above) during tracking, and instead loop through the fine registration (e.g., ICP) step. This is because typically the tracked object does not move very much or very far in the short time between scan registrations, it is already close enough for successful ICP operation. In some embodiments, the exclusion of a coarse registration step may provide a significant increase in tracking speed. In some embodiments, tracking may be lost during ICP-only operation (e.g., the tracked object moves faster or farther than ICP can successfully register). In such cases, by way of example only, the system 10 may revert to a global registration step which may be RANSAC as described above, or in some embodiments an Al-driven analysis of captured images (e.g., from the 3D scanner 12) may be used to quickly determine the location of the tracked object.

[0268] In some embodiments, the CAAT system 10 may increase tracking speed by using a previously registered object position to constrain the considered data points of where the object may move to next for fine registration. In some embodiments, this would reduce the points for consideration during fine registration (e.g., ICP), and fewer points to analyze generally leads to increased processing speed. By way of example, in some embodiments, this data may be constrained based on velocity and / or acceleration. In some embodiments, an object can’t go faster than a maximum determined velocity in any direction, which limits or constrains theVIS-007-W01 potential data points needed for fine registration. In some embodiments, the data points could be constrained base on likely acceleration of the object. In some embodiments, object velocity may be used to extrapolate an initial guess location for the fine registration.

[0269] In some embodiments, the CAAT system 10 may constrain the considered data points of where the object may move to next for fine registration by limiting points of consideration in the acquired topological scan(s) that are a certain distance away from the previously registered object points. In some embodiments, this may include points located within an axis-aligned bounding box in every direction by an amount that is limited by practical physics, for example a set amount (e.g., a guess without first measuring velocity). In some embodiments, the CAAT system 10 may be configured to select a direction for the bounding box growth based on its previous velocity.

[0270] In some embodiments, the CAAT system 10 may constrain the considered data points of where the object may move to next for fine registration by considering only points within a radius from each of the relevant points (and optionally in a particular direction).

[0271] Single object tracking (Hip Center Example)

[0272] The simplest case of tracking is the real-time monitoring of a single object, where its position is continuously updated in the Surgical Coordinate System (SCS). This type of tracking is critical in various surgical procedures where accurate positioning of key anatomical landmarks or surgical tools is required. In this example, the scanner is kept stationary, or its motion is itself tracked relative to the operating room via some other navigation device.

[0273] A practical example of single object tracking in a TKA procedure is the calculation of the hip center of rotation. In some TKA methodologies, such as the Mechanical Alignment approach, determining the precise location of the hip center is essential for establishing the mechanical axis of the leg, which guides the alignment of implants. By way of example, Fig. 23 presents a block diagram depicting steps of a method 320 of determining the hip center of rotation using the CAAT system 10 described herein. As an initial pre-operative (“pre-op”) step 321, a 3D model of the femur may be generated from a pre-op scan, for example by way of a CT scan, MRI, and the like, and uploaded to or otherwise made available to the CAAT system 10.VIS-007-W01By way of example, the method 320 for calculating hip center of rotation described herein uses dynamic, real-time topological data to calculate the hip center of rotation from a series of transformed femoral origin (or “chosen anatomic”) points. By way of example, the chosen anatomic points may be any given point on the pre-op 3D model of the femur that is registered into camera space. In some embodiments, for example, during surgery, as the surgeon rotates the knee within the scanner’s field of view, the scanner captures multiple scans in real time. Each scan records the femoral origin, which is transformed in camera space based on the femur’s position. Since the femur is constrained by the hip joint, these transformed origin points lie on a spherical surface, with the center of this sphere representing the hip’s center of rotation.

[0274] In some embodiments, the first eight steps of the method may be repeated in a (rapidly occurring) loop until sufficient data has been accumulated to accurately determine and report a hip center of rotation and femur length. By way of example, a first step 322 in the method 320 of determining the hip center of rotation is using the 3D scanner 12 described herein to acquire topological scan data of the femur 274. For example, as the surgeon moves the patient’s knee within the field of view 80 of the 3D scanner 12, multiple scans of the femur may be taken in real time. By way of example, this continuous tracking captures multiple high-density data sets within a few seconds, providing numerous transformed femoral points. In some embodiments, a second step 324 of the method 320 of determining the hip center of rotation is registering the 3D model of the femur 274 to the acquired topological scan data. In some embodiments, a third step 326 in the method 320 of determining the hip center of rotation is producing a transformation matrix for each time point. This transformation matrix 2 maps the position of the femur into camera space, allowing the system to track the femur's movements in real-time as the surgeon manipulates the leg.

[0275] In some embodiments, a fourth step 328 of the method 320 of determining the hip center of rotation is to use the generated transformation matrix to compute the location of a chosen anatomic point on the pre-op 3D model for determining the length of the femur in camera space. In some embodiments, a fifth step 330 of the method 320 of determining the hip center of rotation is to send coordinates of the tracked chosen anatomic point to the hub 14 (for example) for sphere fitting.VIS-007-W01

[0276] As previously mentioned, the first eight steps of the method may be repeated in a (rapidly occurring) loop until sufficient data has been accumulated to accurately determine and report a hip center of rotation and femur length. The first five steps identified above describe the CAAT system 10 producing a data point comprising calculated (x, y, z) coordinates for the chosen anatomic point location on the pre-op 3D model upon registration to each scan, which in some embodiments may be happening at a high frequency (e.g., at least 20 times per second in some embodiments). In some embodiments, by way of example only, the chosen anatomic point may be selected manually or automatically with a registration to a model, or with artificial intelligence. Thus, a sixth step 332 of the method 320 of determining the hip center of rotation is to add a newly determined coordinates data point to the dataset containing the previously calculated coordinates. In some embodiments, a seventh step 334 of the method 320 of determining the hip center of rotation is to calculate the size and location of a sphere to which the accumulated data points were best fit, for example using least squares fitting to find the bestfitting sphere, which minimizes the error between the observed data points — the transformed chosen anatomic points captured in real time — and the theoretical spherical surface they approximate. In some embodiments, the best-fitting sphere may be determined using other methods including but not limited to RANSAC. The center of this sphere represents the hip center of rotation, while the radius of the sphere aligns with the physical distance dictated by the femoral constraints. The least squares sphere-fitting method minimizes residuals between the observed points and the estimated spherical surface, resulting in a highly accurate center calculation.

[0277] By way of example, the mathematical basis for the least squares sphere-fitting method is the general equation of a sphere:(x - x0)2+ (y - y0)2+ (z - z0)2- r2where (xo, yo, zo) are the coordinates of the sphere’s center, representing the hip center of rotation, r is the radius of the sphere, corresponding to the distance from the chosen anatomic point to the hip center, and each transformed chosen anatomic (e.g., femoral) point (xj, yi, Zi) should ideally satisfy this equation if it lies perfectly on the sphere’s surface.VIS-007-W01

[0278] In real-world data, due to noise and minor inaccuracies, the chosen anatomic points do not lie precisely on the sphere. Thus, the residual error for each data point (xi, yi, Zi) may be defined as:

[0279] The goal is to minimize the sum of the squares of these residuals across all data points. The objective function to minimize is: + (yi - To)2+ Oi - A,)2) - r)where n represents the total number of data points.

[0280] In some embodiments, the calculation may be optimized using the Levenberg- Marquardt Algorithm. By way of example, the optimization process involves adjusting the parameters (xo, yo, zo) and r to achieve the minimum sum of squared residuals. Given the nonlinear nature of the problem, the Levenberg-Marquardt algorithm (e.g., a standard for nonlinear least squares optimization) may be used. This algorithm iteratively updates the center coordinates (xo, yo, zo) and r by considering the gradient of the objective function until convergence, minimizing the residuals.

[0281] By way of example, the Levenberg-Marquardt algorithm process may include: (i) initializing (xo, yo, zo) and r with initial guesses, possibly derived from preliminary data or approximate mean positions; (ii) iteratively updating the center coordinates and radius based on the error gradient, adapting step sizes according to the algorithm’s damping factor to balance convergence speed and accuracy; and (iii) converging to the best-fit parameters, continuing iterations until changes fall below a pre-defined tolerance level, signifying minimized residuals and an optimized fit.

[0282] In some embodiments, an eighth step 336 of the method 320 of determining the hip center of rotation is to compare residual errors of the fit of the sphere to the predetermined acceptance criteria to assess whether the generated best fit sphere is a good fit (e.g., decision boxVIS-007-W01336). Tn some embodiments, once the optimization completes, the quality of the fit may be evaluated by calculating the residual etfor each point using the equation:

[0283] These residuals quantify how closely each point aligns with the estimated spherical surface. For example, small residuals indicate accurate fit, with residuals near zero suggesting the points closely follow the spherical surface, indicating a good fit and that the estimated hip center accurately represents the true center.

[0284] In some embodiments, confidence analysis may be applied to verify the accuracy of the calculated hip center. By way of example, this may be accomplished through a statistical confidence ellipsoid, calculated based on the residual errors of each fitted point. This ellipsoid offers a measure of confidence, typically set to 95%, indicating that the true center likely falls within this ellipsoid, ensuring a reliable reference point for surgical adjustments.

[0285] For example, to assess the accuracy of the calculated center of rotation, a confidence ellipsoid can be constructed around the estimated center (xo, yo, zo). In some embodiments, this confidence ellipsoid defines a probabilistic boundary, delineating the region within which the true center is likely to reside at a specified confidence level, such as 95%. In some embodiments, the confidence ellipsoid is derived from the residual errors, which represent the difference between each observed data point and the surface of the fitted sphere. In some embodiments, the covariance matrix of these residual errors is calculated to reflect the spread of deviations from the fitted center in the x, y, and z dimensions. By way of example, to construct the confidence ellipsoid, this covariance matrix may be scaled by a factor derived from a chi-square distribution (for a 95% confidence level with 3 degrees of freedom, this factor is approximately 7.815), adjusting the size of the ellipsoid to define a region within which there is a 95% probability that the true center lies. The resulting semi-axes of the ellipsoid reflect the precision along each axis (x, y, z). When these semi-axes are small (e.g., less than 10 mm), it can be concluded with 95% confidence that the true center lies within 1.5 mm of the estimated center. In some embodiments, a small ellipsoid denotes high confidence in the fit, while a larger ellipsoid may suggest the need for further data points or refinement of the fitting process.VIS-007-W01

[0286] If the sphere is not a good fit, then the workflow reverts to the first step 322 of acquiring topological scan data of the femur 274. In some embodiments, if the generated best fit sphere is determined to be a good fit, then a final step 338 in the method 320 of determining the hip center of rotation is to report the hip center and femur origin length (e.g., based upon the position of the sphere origin and the radius of femur).

[0287] While some noise is inevitably introduced due to factors such as movement artifacts in the pattern images or reconstructed object, or slight variations in scan accuracy, the CAAT system 10 can compensate for these by averaging or applying noise-reduction techniques. Once the hip center is identified in camera space, it can easily be transformed into any other coordinate system (e.g., SCS), ensuring that this critical anatomical point is available for accurate surgical planning and alignment, including but not limited to intraoperative real-time modification of the surgical plan.

[0288] Through this approach, the hip center of rotation can be calculated in real time during surgery, significantly enhancing the surgeon’s ability to achieve optimal alignment and implant positioning without additional tracking equipment. This innovation supports streamlined intraoperative workflows while providing the precision needed for high-quality surgical outcomes. A successful application of this mathematical approach allows surgeons to confidently use the estimated hip center of rotation as an anatomical reference during surgery, improving alignment and outcomes without additional tracking equipment.

[0289] This example highlights the value of single object tracking in real-time during surgery, where critical anatomical landmarks must be continuously monitored and accurately calculated to guide the procedure.

[0290] Two Object Tracking and Measuring (Gap Balancing Example)

[0291] A more complex use case for tracking occurs when two or more objects are tracked simultaneously. While a previous section described how distances can be calculated between two objects through registration, this concept can be extended into a continuous tracking mode, where multiple registrations are performed over time to monitor the relative positions of multipleVIS-007-W01 objects. This is especially valuable in real-time scenarios, where dynamic forces are applied, and accurate measurements must be taken as the objects move in relation to each other.

[0292] An example of this in a TKA procedure is when the surgeon uses a gap balancing methodology to assess the gaps between the medial and lateral condyles of the femur and the tibial plateau. By way of example, Fig. 24 is a flowchart depicting an example method 350 of assessing gaps between two objects, for example femoral condyles (e.g., medial and lateral) and a tibial plateau. In this method 350, the goal is to ensure that the gaps between the femur and tibia are balanced when the knee is in both extension and flexion, which helps maintain proper knee alignment and functionality after the procedure.

[0293] Measuring these gaps presents a challenge because the surgeon does not perform this measurement passively. Instead, they manually apply forces to the knee in both a varus (outward) and valgus (inward) direction. This creates a dynamic scenario where the knee joint, the entire leg, and the surgeon’s hands are moving as the force is applied. The critical challenge is that the surgeon is not looking for a single static gap value but rather the peak gap under maximum load. Because significant force is applied during this process, the positions of both the femur and tibia are constantly shifting, and these movements happen quickly. The gap measurement needs to be recorded at high speed, tracking the movement of both the femur and tibia in real time to capture the moments where the force is greatest, and the gap is widest.

[0294] In some embodiments, a first step 352 in the method 350 of assessing gaps between femoral condyles and a tibial plateau is to acquire topological scan data of the femur and tibia and generate reconstructed 3D point clouds of each, for example using techniques described above. In some embodiments, a second step 354 of the method 350 of assessing gaps between femoral condyles and a tibial plateau is to independently register the femur and tibia in the CAAT system 10 as described above, for example using the acquired and reconstructed scan data in addition to reference data 356 of the femur and tibia, for example from preoperative imaging (e.g., CT imaging) or data collected intraoperatively. In some embodiments, a third step 358 in the method of 350 assessing gaps between femoral condyles and a tibial plateau is to use transformation matrices to convert reference data to the surgical coordinate system or to another virtual space. In some embodiments, a fourth step in the method 350 of assessing gaps betweenVIS-007-W01 femoral condyles and a tibial plateau is to calculate the distances between key points on the femur and tibia in real time. As the surgeon applies varus and valgus forces, the system tracks the resulting movements and dynamically measures the gaps between the medial and lateral condyles and the tibial plateau. In some embodiments, this includes performing a first sub step 360 of computing the medial gap (e.g., the minimum distance between the femur and tibia on the medial side) and a second sub step 362 of computing the lateral gap (e.g., the minimum distance between the femur and tibia on the lateral side). In some embodiments, a fifth step 364 in the method 350 of assessing gaps between femoral condyles and a tibial plateau is to consolidate the data and present it to the surgeon in a usable format. In some embodiments, the next step 366 is to determine whether there is enough data to proceed with the procedure. If the answer is no, then the workflow reverts to the first step 352 of acquiring and reconstructing topological scans of the femur and tibia, and the workflow continues to cycle through the steps until enough data has been obtained to assess the gaps between femoral condyles and a tibial plateau, at which point the tracking may end (box 368).

[0295] In some embodiments, by performing multiple registrations rapidly at a rate defined as real-time (e.g., 20+ registrations per second), the CAAT system 10 can record the gap values continuously as the knee moves under load. In some embodiments, recorded measurements may be analyzed to determine the peak gap value at the moment when the maximum force is applied in each direction. Because these forces are only applied for a brief period, rapid and continuous tracking is essential to ensure the correct measurements are captured at the precise moment. In some embodiments, the rapidly occurring multiple registrations may be referred to as a “tracking mode.”

[0296] By way of example, Fig. 25 is a block diagram illustrating several steps of a method 700 of calculating joint gaps based on changes in alignment between the femur and tibia under applied stress. In some embodiments, this method may be used to compute the medial gap 360 and lateral gap 362 as part of the method 350 of assessing gaps between femoral condyles and a tibial plateau described above. For example, in some embodiments, after completing bone registration, the CAAT system 10 may be configured to calculate joint gaps based on changes in alignment between the femur and tibia under applied stress. This approach provides real-time gap measurements during surgery without the need for external optical trackers. By way ofVIS-007-W01 example, the method 700 is described herein with respect to calculating lateral gap measurement when varus stress is applied to the affected knee. In some embodiments, the method 700 may calculate medial gap measurement in the same manner when valgus stress is applied to the affected knee.

[0297] By way of example, a first step 702 of the method 700 of calculating joint gaps based on changes in alignment between the femur and tibia under applied stress is positioning the affected knee in extension, as shown by way of example only in Fig. 26. With the knee in full extension, the femur 274 and tibia 276 may be positioned and stabilized the to allow initial angle measurements. In some embodiments, the CAAT system 10 may be configured to include (e.g., by way of user interface 16 which may include a display component) the flexion / extension (F / E) angle display 720, the varus / valgus (V / V) angle display 722, and the internal / external (int / ext) rotation angle display 724 to help the surgeon achieve neutral alignment (e.g., approximately zero degrees in F / E and int / ext rotation).

[0298] In some embodiments, a second step 704 of the method 700 of calculating joint gaps based on changes in alignment between the femur and tibia under applied stress is establishing the zero-stress V / V angle 722 and zero gap condition. In some embodiments, the V / V angle 722 in this no-stress position is recorded as the baseline measurement. For example, at this baseline position, it can be assumed that no gap exists on either the medial or lateral side of the joint, with both contact points aligned in the absence of applied stress. This alignment relies on the topological scan for data collection, eliminating the need for optical trackers.

[0299] In some embodiments, a third step 706 of the method 700 of calculating joint gaps based on changes in alignment between the femur and tibia under applied stress is applying stress to the affected joint. In some embodiments, for example to ultimately calculate the lateral gap 726, sub-step 706a is to apply positive varus stress 728 to the affected joint to open the lateral side and create stable contact 730 on the medial side, for example as shown in Fig. 27. Similarly in some embodiments, for example to ultimately calculate the medial gap 732, sub-step 706b is to apply positive valgus stress 734 to the affected joint to open the medial side and create stable contact 736 on the lateral side, for example as shown in Fig. 28. By way of example, the CAAT system 10 may be configured to continuously display the F / E 720, V / V 722, and int / extVIS-007-W01724 rotation angles based on ongoing topological scans, providing real-time feedback so the surgeon can correct minor misalignments in F / E or int / ext rotation if needed.

[0300] In some embodiments, a fourth step 708 of the method 700 of calculating joint gaps based on changes in alignment between the femur and tibia under applied stress is topological data collection through structured light scanning. For example, in some embodiments, once the knee is stabilized, a series of scans may be collected to produce high-resolution data reflecting the joint’s response to the applied stress. This scanning relies solely on topological data, removing the need for recalibration or line-of-sight maintenance typical of optical tracking.

[0301] In some embodiments, a fifth step 710 of the method 700 of calculating joint gaps based on changes in alignment between the femur and tibia under applied stress is measuring change in the V / V angle 722. By way of example, the change in the V / V angle (A0) may be calculated by comparing the currently measured angle to the established baseline under no-stress condition. By way of example, for a varus stress application 728, the angle change AO quantifies the lateral opening or gap 726 of the joint while the medial side maintains contact under varus stress. Similarly, for a valgus stress application 734, the angle change AO quantifies the medial opening or gap 732 of the joint while the lateral side maintains contact under valgus stress.

[0302] In some embodiments, a sixth step 712 of the method 700 of calculating joint gaps based on changes in alignment between the femur and tibia under applied stress is continuous tracking for peak gap measurement. By way of example, achieving an accurate joint laxity assessment requires capturing the peak values for both the medial gap 732 and lateral gap 726. Since consistent force application is challenging over time, only continuous tracking of the femur 274 and tibia 276 in real time can reliably identify the peak gap values. By continuously tracking both bones during force application, the CAAT system 10 ensures that peak gap values are recorded for both sides of the joint, providing a full view of joint laxity based on maximum displacement under stress.

[0303] In some embodiments, a seventh step 714 of the method 700 of calculating joint gaps based on changes in alignment between the femur and tibia under applied stress is determining the known articular contact distance (D). By way of example, this may be accomplished by measuring or estimating the distance between medial and lateral contact points (e.g., between theVIS-007-W01 femoral condyles and tibial plateau), serving as the effective lever arm for calculating the lateral and medial gaps 726, 732. In some embodiments, the articular contact distance (D) may be estimated by the CAAT system 10 based on topological data collected by one or more scans acquired by the 3D scanner 12, which may provide the articular contact distance (D) based on cartilage contact. In some embodiments, the articular contact distance (D) may be estimated from the pre-operative CT-scan tibia and femur data registered to topological data collected by one or more scans acquired by the 3D scanner 12, which may have closest-approaching bones. By way of example, if the registered femur and tibia do not meet in the tracking of the CT-based bone, a presumption may be that cartilage is between them. Once this contact distance is determined, no further distance measurements are required.

[0304] In some embodiments, an eighth step 716 the method of calculating joint gaps based on changes in alignment between the femur and tibia under applied stress is applying a trigonometric calculation to calculate the lateral and medial gaps 726, 732. By way of example, using the known articular contact distance and V / V angle change AO, the lateral gap 726 (GL) may be calculated as:GL = D x tan(AO) where D is the articular contact distance, and AO is the change in the V / V angle from baseline when the affected j oint is under applied varus stress. Similarly, by way of example, the medial gap 732 (GM) may be calculated as:GM = D x tan(AO) where D is the articular contact distance, and AO is the change in the V / V angle from baseline when the affected j oint is under applied valgus stress.

[0305] By way of example, the calculated lateral gap value GL represents the increase in space on the lateral side of the joint from the baseline zero-gap condition under applied varus stress. Similarly, the calculated medial gap value G represents the increase in space on the medial side of the joint from the baseline zero-gap condition under applied valgus stress. By way of example, these calculations quantify medial and lateral joint laxity without the need for tracker-based measurements. In some embodiments, this calculation is ultimately anVIS-007-W01 approximation because the closest point varies with angle due to the bone surfaces being not planar. Because the CAAT system 10 is scanning the cartilage surface, it may be advantageous to directly determine the distances between the cartilage surfaces.

[0306] In some embodiments, because the CAAT system 10 determines the surfaces directly, the medial and lateral gaps maybe be determined numerically, for example from a registered CT scan, by calculating closest-points for each bone (or cartilage from registering optical scan data to the new bone positions).

[0307] Example

[0308] Suppose an example environment in which (i) the no-stress baseline V / V angle 0baseiine=0°, (ii) under varus stress, the V / V angle changes to 0 stressed 5 , giving a positive angle change (e.g., A0 = ©stressed - 9baseiine = 5°), and (iii) the distance D between medial and lateral contact points is measured at 60 mm. Using the calculation disclosed above, the gap can be calculated asG=60mm * tan(5°) = 5.2mmThis 5.2 mm lateral gap reflects the knee’s lateral joint laxity under varus stress, calculated exclusively from topological data without optical tracking. Using a gap balancing technique, a surgeon would then measure the gap medially and then modify the surgical plan by adjusting the implant position and orientation to achieve and equal gap on both sides of the knee.

[0309] The method of calculating joint gaps based on changes in alignment between the femur and tibia under applied stress described herein offers a relatively simple and efficient approach to calculating relative joint gaps by leveraging changes in the varus / valgus (V / V) angle and a known contact distance between the medial and lateral points of the joint surfaces. However, alternative methods for calculating relative gaps may also be used to meet specific needs or anatomical considerations in similar surgical applications. Some examples include cross-sectional area analysis and 3D surface distance mapping.

[0310] Regarding cross-sectional area analysis, in some embodiments, instead of using a single angle change and contact distance, a cross-sectional approach could be taken, where serialVIS-007-W01 cross-sections of the joint are analyzed to calculate gap changes across a region. By capturing multiple measurements in parallel planes, the system can calculate an average or profile of gap changes across the lateral or medial side, providing a more detailed spatial representation of joint laxity.

[0311] In some embodiments, the 3D surface distance mapping method involves point-to- point or point-to-surface distance mapping across the entire 3D surface of the femur and tibia. In this approach, the system would calculate the average distance or minimum distance between the entire surfaces, providing an integrated measurement of the gap. This method can give a comprehensive picture of joint laxity by capturing the entire interaction surface rather than relying on a single angle or lever-arm distance.

[0312] These alternative methods are more computationally intensive but could provide greater detail in cases where understanding spatial variations in joint laxity across a broader area is beneficial.

[0313] While the presented example focuses on measuring relative joint gaps between the femur and tibia during a TKA procedure, the principles behind this method are broadly adaptable. This approach can be applied to measure relative gaps or angular changes between any two anatomical structures where topological scanning and registration to preoperative imaging data (e.g., CT, MRI, or an optical reference scan) are feasible. By capturing 3D surface data and aligning it with preoperative imaging, this approach can measure gaps, angular relationships, or relative displacements in other anatomical contexts. For example, in shoulder arthroplasty or repair, shoulder j oint laxity, or relative gaps between the humeral head and glenoid cavity, could be calculated using similar principles. Changes in the shoulder's rotation or flexion angles under applied forces could be used to assess joint stability or soft tissue tension. In spinal procedures, relative measurements could track the gap or height between vertebrae by assessing changes under varying loads, angles of flexion, or introduction of implants. This approach could support intervertebral disk replacement or fusion procedures where maintaining precise intervertebral spacing is critical. As another example, during a hip replacement procedure the relative gap between the acetabulum and femoral head could be measured to assessVIS-007-W01 joint stability and implant alignment. This measurement could help optimize soft tissue tension and balance around the joint.

[0314] These examples show that the method 700 is adaptable to measure relative geometric relationships across various anatomical structures, as long as 3D topological data can be captured and registered to preoperative imaging or optical reference scans if the optical reference scan captures the relevant anatomy of the separations. This versatility broadens the method’s utility, making it applicable for multiple orthopedic and surgical contexts where joint alignment, laxity, or spacing measurements are essential for procedural success. Thus, the method 700 disclosed herein provides flexible, accurate, and non-invasive measurement capabilities for assessing relative gaps or distances between specific points of anatomical structures, enhancing intraoperative guidance and potentially improving patient outcomes across multiple surgical fields.

[0315] There are several benefits to real-time gap measurement. For example, the ability to track the femur and tibia in real time allows for dynamic gap measurements under varying loads, providing more accurate and clinically relevant data than static measurements. Additionally, real-time tracking ensures that the CAAT system 10 captures the peak gap values, which are critical in determining the balance between the medial and lateral sides of the knee.Furthermore, since the leg and knee joint are moving throughout the measurement process, continuous tracking allows the system to account for these movements and still produce accurate results.

[0316] Tracking Anatomy with a Robot-Assisted Surgical Instrument

[0317] A more complex use case for tracking arises when a surgical instrument is attached as the end effector of a robot. In this scenario, the position and orientation of the instrument must be continuously updated to accommodate any anatomy movement. An example of this in TKA is when an oscillating saw is attached to a robotic system, and the robot assists the surgeon by positioning the saw in the correct plane for a planned osteotomy. By way of example, Fig. 4 illustrates an operating environment 22 in which a surgical instrument 28 is positioned on the end of a robotic arm 30.VIS-007-W01

[0318] By way of example, the robot’s function is to prevent the saw blade from drifting outside of the predefined cut plane, maintaining specific translational and angular tolerances. This process is straightforward if the knee is rigidly fixed. However, in practice, the knee may move slightly due to various factors such as imperfect manual stabilization or surgeon manipulation. As the knee moves, the robot must continuously adjust to follow the anatomy and ensure the osteotomy remains accurate. Traditionally, this can be achieved using Optical Markers (OMs) and Optical Trackers (OTs) (e.g., unless they get bumped), but it can also be accomplished using real-time topological data collected from the 3D scanner 12 without the use of OMs and / or OTs. For example, Fig. 29 is a block diagram illustrating an example method 370 of tracking movable anatomy and a movable surgical instrument configured to affect the movable anatomy, according to some embodiments of the disclosure. In some embodiments, this method 370 is essentially an extension of the multi-object tracking described previously, but with the additional complexity of having to communicate the anatomy’s movement to the robot so that the position of the end effector can be updated in real time.

[0319] In this example scenario, multiple objects are tracked simultaneously. Thus, in some embodiments, a first step 372 in the method 370 is to acquire topological scan data (e.g., using the 3D scanner 12) of the target anatomy (e.g., a femur) and the surgical instrument 28 and generate reconstructed 3D point clouds of each, for example using techniques described above. In some embodiments, a second step 374 of the method 370 of tracking movable anatomy and a movable surgical instrument configured to affect the movable anatomy is to independently register the target anatomy and the surgical instrument 28 on the end of the robot arm 30 in the CAAT system 10 as described above, for example using the acquired and reconstructed scan data in addition to reference data 376 of the target anatomy and surgical instrument. In some embodiments, the reference data 376 of the target anatomy may be acquired from preoperative imaging (e.g., CT imaging) or data collected intraoperatively. In some embodiments, the reference data 376 of the surgical instrument 28 may be acquired from preoperative CAD data, preoperative scans, intraoperative scans, and / or a fiducial marker if the instrument does not scan well. In some embodiments, the origin point (or other point of interest) of the instrument 28 is identified in the data set or calibrated at the time of surgery. In some embodiments, for example, a fiducial may be registered relative to the tool in order to calibrate its position relative to the end effector.VIS-007-W01

[0320] In some embodiments, a third step 378 in the method 370 is to use transformation matrices to convert the target anatomy position and orientation into robot space. By way of example, the robot's base remains fixed, and a point on the base is defined as the origin of the surgical coordinate system (or “SCS”), which in this example may also be referred to as "robot space." In some embodiments, all other objects (3D scanner 12, instrument (e.g., saw) 28, femur 274, etc.) may be transformed into the robot space to ensure alignment. By way of example, by tracking the femur, its position can be continuously transformed into the robot space. In some embodiments, an optional step 380 of the method 370 may be performed to smooth the input data to reduce the effects of noise before the data is sent to the robot, for example using a moving average and / or Kalman filter. In some embodiments, because the robot base might inadvertently move relative to the patient, the common coordinate system may preferentially be the patient anatomy itself (either femur or tibia, for example). This is advantageous and more direct than the common space being trackers that have been attached rigidly to bone, for example.

[0321] In some embodiments, a fourth step 382 in the method 370 is to send the target anatomy position and orientation to the robot. By way of example, the instrument 28 (e.g., saw blade) can be tracked either directly using the 3D scanner 12 as disclosed herein or by relying on data from the robot's internal control system, which already knows the position of its end effector (and hence the saw blade) in robot space. However, for the CAAT system 10 to navigate with the scanner 12, the corresponding position in scanner space would need to be known. Thus, the robot space will need to have been registered to the scanner space, which may be done by preliminarily scanning and registering the instrument 28 in a few positions. After that the CAAT system 10 can transform the robot-reported position of the end effector to the space of the scanner. In that way the CAAT system 10 determines where the instrument 28 is in scanner space, even if the instrument 28 is not within the field of view of the scanner.

[0322] In some embodiments, a practical result of the method 370 is that the position of the target anatomy is continuously tracked as it moves during surgery, for example using the CAAT system 10 and the 3D scanner 12. In some embodiments, the updated position of the target anatomy is continuously communicated to the robot 30, which as a fifth step 384 in the method 370, may intraoperatively plan the optimal kinematic motion path to maintain relative position to the target. Stated differently, the robot may adjust the instrument 28 (e.g., saw blade) movement in real time to maintain alignment with the target anatomy and the predefined cut plane. By wayVIS-007-W01 of example, the robot effectively compensates for any slight anatomical shifts, ensuring the accuracy of the osteotomy.

[0323] This real-time adjustment can be performed within predefined translational and angular tolerances, providing the surgeon with a more precise and reliable way of cutting within a moving surgical field. Depending on the design of the robot and its positional control system, it may be necessary to update the registration data at a rate faster than 20 registrations per second (rps) to achieve a finely tuned control system.

[0324] In some embodiments, a sixth step 386 in the method 370 is to execute the motion path of the instrument with minimal latency between the time the 3D topological data is captured to when an updated transformation matrix is sent to the robot. In such a use case, latency becomes critical. Any delay in updating the robot’s position relative to the anatomy can lead to the saw blade falling out of alignment with the femur, reducing the accuracy of the cut. Minimizing latency between the time the 3D topological data is captured and when the updated transformation matrix 2 is sent to the robot is essential for maintaining real-time performance. For instance, if the system operates at 30 registrations per second (rps) and the latency is 50 milliseconds (ms), then the robot will get data on its new position 50 ms after the bone being tracked was there. Additionally, there may be additional latency introduced by the robot’s control system (e.g., the robot needs to calculate an optimal path, engage motors, accelerate, and finally reach the intended point). If, for example, it takes the robot 20 ms to perform these tasks, the total lag would be 70 ms. Thus, it is essential to reduce both sources of lag in a real-time tracking system to ensure that the robot can respond in real time to anatomical movements.

[0325] In some embodiments, a seventh step 388 in the method 370 is to determine whether tracking of the target anatomy and surgical instrument needs to continue. If the determination is affirmative, then the workflow cycles back to step 372 of acquiring and reconstructing topological scan data of the target anatomy. If the answer is no (e.g., the osteotomy has been completed), then the system may end the tracking (e.g., box 389).

[0326] By increasing the registration rate and reducing computational latency, the system can deliver accurate, real-time adjustments to the robot, ensuring that the saw blade remains in theVIS-007-W01 correct plane even as the target anatomy moves. This approach maintains the accuracy of the osteotomy and helps prevent unintended deviations, improving surgical outcomes.

[0327] Anatomy Tracking and Robot Following with a Mounted Scanner[00328J In previous examples, the 3D scanner 12 was positioned at a fixed location. However, mounting the scanner on the robotic arm 34 (e.g., as shown by way of example in Fig. 4) allows the 3D scanner’s 12 position and field of view 68 to change dynamically as the robot 34 moves. This configuration offers significant advantages during surgical procedures where the anatomy or surgical site frequently shifts.

[0329] For instance, in a Total Knee Arthroplasty (TKA) procedure, the knee is often moved by the surgeon, transitioning from full extension to 90 degrees of flexion, and sometimes into full flexion. While a fixed 3D scanner 12 can capture the surgical site in one position, maintaining a clear view may become challenging as the anatomy moves. If the knee or other anatomical structures shift out of the 3D scanner 12’ s field of view 68, manual repositioning of the scanner 12 with respect to the affected knee would be required, interrupting the workflow.

[0330] By mounting the 3D scanner 12 on the robot’s arm 34 to move along with the end effector (or by itself on a robot arm 34 independent of an end effector, as shown by way of example in Fig. 4), the system can continuously track bones like the femur or tibia and automatically adjust the scanner’s position to maintain a fixed perspective on the surgical site. As the knee moves through different angles, the robot dynamically updates the scanner's position, keeping the anatomy within the field of view. This automation reduces the need for manual intervention, saving time and effort. In some embodiments, the robot may be provided with one or more degrees of freedom to enable movement of the 3D scanner 12. By way of example, the robot shown in Fig. 4 includes seven (7) degrees of freedom, however robots with fewer degrees of freedom (e g., 1, 2, 3, etc.) or more degrees of freedom (e.g., 8, 9, 10, etc.) may be used as may be needed or available.

[0331] In addition to maintaining a fixed view, the robot-mounted scanner synchronizes its movements with the anatomy in real-time. This optimizes scanner positioning throughout the procedure, allowing the surgeon to focus on the operation without worrying about viewpointVIS-007-W01 adjustments. In some embodiments, multiple viewpoints can be pre-programmed or customized for different stages of the surgery, with the surgeon easily switching between them via the user interface (UI), further enhancing procedural efficiency.

[0332] Anatomy Tracking with Robot-Controlled Instrument (Oscillating Saw Example)

[0333] In this use case, both a 3D scanner 12 and an oscillating saw 28 are mounted to a robot arm. By way of example, the 3D scanner 12 tracks the anatomy while the robot positions the saw 28 to make precise cuts along a predefined cut plane, which may be modified intraoperatively based on the surgeon’s real-time observations. In some embodiments, if the 3D scanner 12 position changes as the robot moves, it may be necessary to apply one or more transformations to ensure that the anatomical data, 3D scanner 12, robot base, and saw 28 remain aligned in the same coordinate space. In some embodiments, the common coordinate system may be referenced to the patient, the 3D scanner 12, the robot base, or to another object.

[0334] By way of example only, Figs. 30A-B depicts a block diagram illustrating a method 390 of complex anatomy tracking with robot-controlled instrument in the context of a TKA, where the robot assists with cutting the femur.

[0335] In some embodiments, a first step 392 of the example method 390 of complex anatomy tracking with robot-controlled instrument is to mount the scanner 12 and the saw or saw blade 28 to a robot end effector 28, which by way of example may be an articulating arm having multiple degrees of freedom. In some embodiments, a next step 393 in the method 390 is to move (or initiate the robot to move) the robot arm to a known position in which the scanner 12 faces the robot base. In some embodiments, a next step 394 in the method 390 is to use the scanner 12 to acquire a dataset comprising structured light image data of the robot base (“robot base image dataset”). In some embodiments, a next step 395 in the method 390 is to upload a dataset comprising the robot base CAD geometry to the CAAT system 10 (“robot base CAD dataset”). In some embodiments, a next step 396 in the method 390 is to register the robot base image dataset to the uploaded robot base CAD dataset. In some embodiments, a next step 397 in the method 390 is to compute a transformation matrix T(B / S) (transformation matrix of the robot base dataset to the common coordinate system) at this fixed position to relate the robot base to the common coordinate system. In some embodiments, a next step 398 in the method 390 mayVIS-007-W01 be to track or continuously monitor robot position sensors and update the transformation matrix T(B / S) in real time.

[0336] By way of example, the above steps establish a robot base position in the common coordinate system. As previously mentioned, all other objects (3D scanner 12, instrument (e.g., saw) 28, femur 274, etc.) are transformed into the common coordinate system to ensure alignment (e.g., not necessarily in that order, and not necessarily after establishing the robot base position in the common coordinate system as described). By way of example, the robot's base remains fixed, and in some embodiments a point in the common coordinate system may be referenced to the base. In some embodiments, the scanner 12 may be used for correcting errors resulting from base-frame drift. For example, after robot-patient registration, in the absence of any motion of a tracker attached to the patient, the end-effector is known relative to the position of the base based on its own localizers. However, the end-effector position may become inaccurate relative to the patient if the robot base is not perfectly rigid with respect to the patient, even if the patient does not move in any way. So, even though the robot-reported position is used once the system 10 is calibrated, the scanner 12 has the additional capability of checking the actual relative location of the end-effector with respect to the patient. That way, if the base flexes, creating a millimetric deviation at the end-effector (for example), the CAAT system 10 can capture that and correct the relative end-effector location relative to the patient.

[0337] In some embodiments, the following steps of the method 390 of complex anatomy tracking with robot-controlled instrument may be followed to relate the position of the saw 28 or saw blade on the robot end effector to the common coordinate system. In some embodiments, this may be accomplished by expressing the position of the robot end effector in real time as a transformation matrix T(SB / S) (transformation matrix of the saw blade dataset to the common coordinate system). By way of example, the position of the robot's end effector may be known in real-time through the robot control system. This transformation matrix T(SB / S) is used to relate the end effector’s (and therefore the sawblade’s) position to the common coordinate system.

[0338] In some embodiments, a next step 399 in the method 390 is to use the 3D scanner 12 to acquire a structured light image dataset of the saw blade (“saw data set”) if the saw blade isVIS-007-W01 positioned within the camera’s field of view or a saw blade calibration tool if the saw blade is not within the camera’s field of view. In some embodiments, a next step 400 in the method 390 is to upload a saw blade or saw blade calibration tool CAD geometry dataset to CAAT system 10. In some embodiments, a next step 401 in the method 390 is to register the saw dataset to the uploaded CAD geometry dataset. In some embodiments, a next step 402 in the method 390 is to compute a transformation matrix T(SB / S) (transformation matrix of the saw dataset to the scanner space) at the fixed position. In some embodiments, a next step 404 in the method 390 is to track or continuously monitor robot position sensors and update the transformation matrix T(SB / S) in real time.

[0339] In some embodiments, the following steps of the method 390 of complex anatomy tracking with robot-controlled instrument may be followed to register and track the femur position in real-time within the common coordinate system. In some embodiments, the CAAT system 10 may accomplish this by producing the transformation matrix T(F / S) (transformation matrix of the femur to the common coordinate system).[00340J In some embodiments, a next step 406 in the method 390 is to manually move the robot to position where the scanner faces the femur. In some embodiments, a next step 408 in the method 390 is to use the 3D scanner 12 to acquire a structured light image dataset of the femur (“femur dataset”), which will include a subset data representative of femoral cartilage. In some embodiments, a next step 410 in the method 390 is to determine whether a reference dataset (e.g., preoperative CT scan) of the femur is available. If yes, then in some embodiments, a next step 412 in the method 390 is to upload a segmented reference dataset of the femur to the CAAT system 10. In some embodiments, a next step 414 in the method 390 is to register the femur / femoral cartilage scan dataset to the reference (e.g., CT) dataset, for example using one or more methods described herein. If there is no available reference dataset, then in some embodiments, a next step 416 in the method 390 is to create a femur / femur cartilage model dataset from the femur dataset obtained by intraoperative scanning. In some embodiments, a next step 418 in the method 390 is to register the created model dataset to the femur / femoral cartilage scan dataset, for example using one or more methods described herein. In some embodiments, a next step 420 in the method 390 is to compute a transformation matrix T(F / S) to transform this registration into the common coordinate system. In some embodiments, a nextVIS-007-W01 step 422 in the method 390 is to continuously repeat the previous steps (e.g., scan femur / femoral cartilage, register the scan dataset to the reference dataset, and recompute the transformation matrix T(F / S) in real time to track or monitor the position of the femur in the common coordinate system.

[0341] In some embodiments, the registrations described above may be performed in any order, since the goal of registering each of the robot, target anatomy (e.g., femur), surgical instrument (e.g., saw blade) and / or scanner into a common coordinate system is the same regardless of which occurs first. For example, in some embodiments, it may be advantageous to register the patient anatomy first, using the patient anatomy as the origin of the common coordinate system to minimize the need to re-register the patient anatomy in the event of movement.

[0342] In some embodiments, the following steps of the method 390 of complex anatomy tracking with robot-controlled instrument may be followed to register the surgical plan (e.g., cut plane) into the common coordinate space and perform the surgical cut. By way of example, the cut plane is initially determined during preoperative planning and adjusted during surgery based on patient-specific anatomy and knee kinematics. In some embodiments, the cut plane may be defined relative to the femur.

[0343] In some embodiments, a next step 424 in the method 390 is to load a preoperative CT dataset and / or femur image data into a surgical planning tool and determine proper cut planes in the common coordinate system. In some embodiments, a next step 426 in the method 390 is to continuously convert robot base, scanner, and saw blade into the common coordinate system by applying the correct respective transformation matrices and / or their inverses as needed. In some embodiments, a next step 428 in the method 390 is to continuously update the robot’s position to validate the position of the robot with respect to the anatomy, for example to ensure the saw blade remains in the cut plane at all times even with movement of the femur. In some embodiments, a next step 429 in the method 390 is to manually power and move the saw blade within the cut plane as needed to execute the cut as planned.

[0344] By way of example, all of these transformation matrices are computed in real-time, and the data is continuously fed into the robot's control system, to maintain the saw’s alignmentVIS-007-W01 within the predefined cut plane (for example) while adjusting its position based on real-time anatomical movement. This allows the surgeon to follow the operative plan, for example to manipulate the saw within the plane while the robot ensures that the blade remains within the precise boundaries of the cut plane.

[0345] It is important to note that the same principles of transforming anatomy and objects into a common coordinate space apply to all the previous use cases described but the equations were not explicitly given. Whether it’s tracking a single object like the femur, maintaining a stable field of view, or synchronizing a scanner and robot, the underlying process of sequentially applying transformation matrices remains a key part of the system. These transformations ensure that all components of the system — whether anatomy, surgical instruments, or the robot — are correctly aligned within the SCS, enabling precise, real-time interactions.

[0346] By way of example, Fig. 31 is a block diagram depicting an example novel TKA workflow 430 with pre-operative imaging using the CAAT system 10 of the present disclosure. By way of example, the workflow 430 may be divided into several portions, including a preoperative portion 432, OR setup portion 434, initial registration portion 436, surgical planning portion 438, Bone resection portion 440, and implantation portion 442.

[0347] Regarding the pre-operative portion 432, prior to surgery, detailed imaging 444 of the target anatomy is often acquired using CT (Computed Tomography) or MRI (Magnetic Resonance Imaging). In some embodiments, these scans may be segmented to create a 3D model of the relevant structures, such as the femur and tibia in TKA. These 3D models are essential for pre-operative planning 446 of the procedure and aligning the patient's real-time anatomy with the surgical plan. Important anatomical planes and landmarks are also labeled within the model to assist the surgeon during the operation.

[0348] For this workflow, we assume a patient-specific CT scan has been segmented to provide a detailed 3D model of the femur and tibia, though the imageless mode can also be utilized if necessary.

[0349] By way of example, the OR setup portion 434 may include patient positioning 448, 3D scanner setup 450, robot setup 452, and instrument setup 454. Before surgery begins, the CAATVIS-007-W01 system 10 requires calibration. In some embodiments, scanner setup 450 comprises one or more 3D scanners 12 being positioned to ensure full visibility of the surgical field. Calibration ensures that the CAAT system 10 aligns correctly with the surgical setup, accounting for distances, camera angles, and the positioning of all equipment.

[0350] Robotic equipment requires additional steps for calibration to ensure that the robot functions correctly. The robot setup 452 process includes aligning the robot's physical movements with the virtual model of the surgery to maintain accuracy and ensure that the robot performs as expected during the procedure.

[0351] In some embodiments, patient positioning 448 occurs after setup and calibration. By way of example, in a TKA procedure the patient is positioned on the operating table, typically in a supine position with the knee exposed for TKA procedures, as shown by way of example in Fig. 4. The surgical site is prepped and draped in a sterile fashion.

[0352] By way of example, the initial registration portion 436 may include surgical exposure 456, scanning 458, bone to cartilage registration 460, hip center calculation 462 via tracking, and ankle center calculation 464 using a pointer. In some embodiments, upon completion of the OR setup 434 portion, surgical exposure 456 may occur, with an incision made to access the knee joint and prepare it for the robotic system to perform the procedure.

[0353] In some embodiments, femoral registration may occur as described above, by using the 3D scanner to generate a 3D point cloud model registered to the SCS. This real-time anatomical model is maintained within the Surgical Coordinate System (SCS), enabling accurate tracking of the patient’s anatomy throughout the procedure. In some embodiments, the CAAT system 10 may perform bone to cartilage registration 460, including misalignment correction as described below.

[0354] Following patient registration, the surgeon may identify key anatomical landmarks necessary for establishing the mechanical axis of the leg, for example including the hip center 462 (e g. identified by rotating the leg while the CAAT system 10 calculates the center of rotation of the femoral head as described above), and / or ankle center 464 (e.g., identified using the optical tracking system on the medial and lateral malleoli (ankle bones) to calculate theVIS-007-W01 center of the ankle joint or the pointer instrument described below). These landmarks, or others, or measured curvatures, for example, are crucial for ensuring proper implant alignment. The mechanical axis is one common methodology, but other alignment methods, like kinematic alignment, may use different anatomical landmarks depending on the procedure.

[0355] At this stage, a virtual surgical plan 438 is developed to guide the robotic system and surgeon throughout the procedure. If pre-operative planning was done, it serves as an initial basis but must be verified and adjusted in real time as the surgery progresses.

[0356] By way of example, the virtual surgical plan 438 may involve the implant selection 466, initial implant positioning 468, gap balancing via tracking 470, positioning adjustments 472, and cut plane definitions 474. Regarding implant selection 466, in some embodiments the CAAT system 10 may assist the surgeon in selecting the optimal implant size based on the patient’s anatomy. Proper implant sizing ensures both the long-term success of the surgery and post-operative joint function. Regarding implant positioning, in some embodiments the CAAT system 10 may help the surgeon determine the ideal implant position to achieve accurate joint alignment. The goal is to ensure proper leg alignment and balance between the medial and lateral sides of the knee. The alignment is measured in terms of varus (inward tilt) and valgus (outward tilt), with the system assisting the surgeon in making appropriate adjustments. In some embodiments, gap balancing 470 ensures equal tension in the soft tissues (primarily ligaments) around the knee joint throughout its range of motion. By way of example the CAAT system 10 dynamically tracks the relative positions of the bones and calculates the space between them to guide the surgeon in balancing the medial and lateral gaps during the procedure, as described above. At this time, any positioning adjustments 472 should be made. Regarding defining the cut plane 474, in some embodiments based on the selected implant size and position, the system defines the necessary osteotomies (bone cuts). These cuts must be precise to ensure proper implant placement.

[0357] In some embodiments, during the bone resection 440 portion, the CAAT system 10 assists the surgeon in making precise cuts according to the surgical plan. The system continuously tracks the patient's anatomy and provides feedback, ensuring the cuts are executed accurately. After each bone cut, the system verifies the accuracy of the cuts, implant alignment,VIS-007-W01 and gap balancing. If deviations are detected, the system alerts the surgeon, allowing for realtime adjustments to be made. By way of example, this cut and verify process occurs for each osteotomy cut, including anterior 476, posterior 478, distal 480, posterior chamfer 482, anterior chamfer 484, and / or tibia 486.

[0358] In some embodiments, after bone cuts are verified, the workflow 430 proceeds with the implantation 442 portion. In some embodiments, trial implants 488 are placed to assess the fit, alignment, and joint movement. If necessary, adjustments are made before the final implant 490 is inserted. In some embodiments, the surgeon then performs a final assessment (e.g., including range of motion tests 492) to ensure proper alignment and stability of the new joint. Once confirmed, the surgical site is closed 494, and the patient is prepared for recovery.

[0359] By way of example, Fig. 32 is a block diagram depicting an example novel TKA workflow 630 using the CAAT system 10 of the present disclosure without pre-operative imaging. By way of example, the workflow 630 may be divided into several portions, including a pre-operative portion 632, OR setup portion 634, initial registration portion 636, surgical planning portion 638, Bone resection portion 640, and implantation portion 642.

[0360] Regarding the pre-operative portion 632, prior to surgery, detailed imaging 444 of the target anatomy is often acquired using CT (Computed Tomography) or MRI (MagneticResonance Imaging). In some embodiments, these scans may be segmented to create a 3D model of the relevant structures, such as the femur and tibia in TKA. These 3D models are essential for pre-operative planning 446 of the procedure and aligning the patient's real-time anatomy with the surgical plan. Important anatomical planes and landmarks are also labeled within the model to assist the surgeon during the operation.

[0361] Regarding the pre-operative portion 632, prior to surgery, a standard 3D model 646 of the bones may be used instead of patient-specific scans described in the previous workflow 430. Anatomical landmarks, such as the medial and lateral epicondyles of the femur, are identified during surgery using the 3D scanner 12 as described above. In some embodiments, the distance between these landmarks is used to scale the standard model to match the patient’s unique anatomy, allowing the surgeon to visualize implant placement with appropriate proportions, even in the absence of patient-specific imaging. These 3D models 646 may be used for pre-operativeVIS-007-W01 planning 644 of the procedure and aligning the patient's real-time anatomy with the surgical plan. Important anatomical planes and landmarks are also labeled within the model to assist the surgeon during the operation.

[0362] By way of example, the OR setup portion 634 may include patient positioning 648, 3D scanner setup 650, robot setup 652, and instrument setup 654. In some embodiments, scanner setup 650 comprises one or more 3D scanners 12 being positioned to ensure full visibility of the surgical field and registration of the one or more 3D scanners 12 in a common coordinate system. In some embodiments, robot setup 652 may include registration of the robot base in the common coordinate system. In some embodiments, instrument setup may include registration of a surgical instrument within the common coordinate system, including but not limited to a saw blade (for example) attached as a robot end effector. By way of example, registration ensures that the various components of the CAAT system 10 align correctly to one another within the surgical setup, accounting for distances, camera angles, and the positioning of all equipment.

[0363] In some embodiments, patient positioning 648 occurs after setup and calibration. By way of example, in a TKA procedure the patient is positioned on the operating table, typically in a supine position with the knee exposed for TKA procedures, as shown by way of example in Fig. 4. The surgical site is prepped and draped in a sterile fashion.

[0364] By way of example, the initial registration portion 636 may include surgical exposure 656, scanning and segmentation 658, articular surface matching (ASM) registration 660, and initial size and position setup 662. In some embodiments, upon completion of the OR setup 634 portion, surgical exposure 656 may occur, with an incision made to access the knee joint and prepare it for the robotic system to perform the procedure.

[0365] In some embodiments, femoral registration may occur as described above, by using the 3D scanner 12 to generate a 3D point cloud model registered to the common coordinate system. This real-time anatomical model is maintained within the common coordinate system, enabling accurate tracking of the patient’s anatomy throughout the procedure. By way of example, during ASM registration 660, the CAAT system 10 analyzes intraoperatively acquired topological data of the patient’s articular surfaces, and automatically compared them to digitalVIS-007-W01 models of available implants, assessing which implant design and size will best match the patient’s unique anatomy (e.g., initial size and position set 662).

[0366] At this stage, a virtual surgical plan 638 is developed to guide the robotic system and surgeon throughout the procedure. If pre-operative planning was done, it serves as an initial basis but must be verified and adjusted in real time as the surgery progresses.

[0367] By way of example, the virtual surgical plan 638 may involve gap balancing via tracking 670, positioning adjustments 672, and cut plane definitions 674. In some embodiments, gap balancing 670 ensures equal tension in the soft tissues (primarily ligaments) around the knee joint throughout its range of motion. By way of example the CAAT system 10 dynamically tracks the relative positions of the bones and calculates the space between them to guide the surgeon in balancing the medial and lateral gaps during the procedure, as described above. At this time, any positioning adjustments 672 should be made. Regarding defining the cut plane 674, in some embodiments based on the selected implant size and position, the system defines the necessary osteotomies (bone cuts). These cuts must be precise to ensure proper implant placement.

[0368] In some embodiments, during the bone resection 640 portion, the CAAT system 10 assists the surgeon in making precise cuts according to the surgical plan. The system continuously tracks the patient's anatomy and provides feedback, ensuring the cuts are executed accurately. After each bone cut, the system verifies the accuracy of the cuts, implant alignment, and gap balancing. If deviations are detected, the system alerts the surgeon, allowing for realtime adjustments to be made. By way of example, this cut and verify process occurs for each osteotomy cut, including anterior 676, posterior 678, distal 680, posterior chamfer 682, anterior chamfer 684, and / or tibia 686.

[0369] In some embodiments, after bone cuts are verified, the workflow 630 proceeds with the implantation 642 portion. In some embodiments, trial implants 688 are placed to assess the fit, alignment, and joint movement. If necessary, adjustments are made before the final implant 690 is inserted. In some embodiments, the surgeon then performs a final assessment (e.g., including range of motion tests 692) to ensure proper alignment and stability of the new joint. Once confirmed, the surgical site is closed 694, and the patient is prepared for recovery.VIS-007-W01

[0370] Anatomical Point Location Without Optical Trackers

[0371] By way of example, in cases where neither direct measurement (e.g., as disclosed herein in relation to gap measurement) nor inferred calculations (e.g., as disclosed herein in relation to calculating the hip center of rotation) are possible - such as when the anatomical landmark is entirely outside the scan field and no calculation model is feasible - another approach may be necessary. By way of example, this disclosure introduces a method to gather anatomical data beyond the scanner’s field of view without relying on scanned anatomic reference data or visible calculations. In some embodiments, this method may be used with a specialized pointer instrument with a unique handle geometry and / or finial feature, designed to stay visible to the scanner even as the pointer contacts anatomical landmarks outside the scan field. The system registers the finial’s position and, using the pointer’s known geometry, infers the precise location of the anatomical landmark.

[0372] For example, in TKA, identifying the center of the ankle is crucial for aligning the lower limb during surgery. In some embodiments, the surgeon can position a distal tip of pointer at the medial and lateral malleoli of the patient’s ankle 6, with the proximal end having a unique geometry and / or finial remaining within the scanner’s view (e.g., near the patient’s knee 4) (e.g., Fig. 33). By registering the finial to the common coordinate space (e.g., which may be the scanner space) and applying an anatomical offset based on the pointer’s geometry, the CAAT system 10 may calculate the ankle center as a reference for alignment. This process eliminates the need for complex optical tracking systems and does not rely on any calculations from visible anatomy, making it suitable for locating points completely beyond the scan field.

[0373] The differences among the various methods may be explained as follows. By way of example, the method 700 described above measures anatomical data directly within the scanner 12 field of view 80, such as joint laxity or gap measurements, by capturing high-resolution 3D surface data of the relevant structures. It requires that the anatomy of interest be within the scan field for real-time measurement, making it effective for intraoperative assessments but limited for points outside this range. By way of example, the method 320 disclosed above infers anatomical data beyond the scanner’s field of view through indirect calculations based on scanned reference points. This method applies when the anatomical landmark can be estimatedVIS-007-W01 mathematically from visible structures, such as calculating the hip center of rotation using a sphere-fitting algorithm on femoral head data visible within the scan field. By way of example, the method 800 disclosed herein gathers anatomical data for points beyond the scan field without requiring direct measurement or mathematical inference. In some embodiments, this method may use a pointer instrument with a visible finial to register the pointer’s position within the common coordinate system. The known geometry of the pointer enables determination of a precise location of anatomical points that lie entirely outside the scanner’s field of view, such as the ankle center, without relying on visible reference data or optical trackers.

[0374] In this way, the method 800 disclosed herein extends the capabilities of image-guided navigation by providing a solution for point localization in scenarios where neither direct measurement nor inferred calculations are viable. This innovation facilitates more versatile, accurate landmark identification, broadening the scope of intraoperative navigation options in complex orthopedic procedures.

[0375] Fig. 33 depicts an example of a pointer instrument 750 configured for use with the CAAT system 10 to identify specific anatomical points that are beyond the scanner’s field of view without performing calculations on visible anatomy or using optical trackers, according to some embodiments. In some embodiments, the pointer instrument 750 comprises a proximal end 752, a distal end 754, and an elongated shaft 756 extending between the proximal and distal ends 752, 754. In some embodiments, the proximal end 752 includes a handle 758. In some embodiments, the pointer instrument 750 includes a uniquely shaped finial 760 extending proximally from the handle 758. In some embodiments, the handle 758 may have a unique scannable surface geometry. In some embodiments, the pointer instrument 750 may be positioned such that the proximal end 752 including the handle 758 (e.g., with scannable surface geometry) and / or the finial 760 remains visible to the scanner 12 even as the distal end 754 of the pointer instrument 750 reaches points beyond the limits of the scanner 12 field of view. In some embodiments, the finial 760 may have a geometry that enables the CAAT system 10 to register its orientation and position accurately within the common coordinate space, ensuring that the inferred location of anatomical points is precise. In some embodiments, the finial 760 may be a single shape with a complex geometry. In some embodiments, the finial 760 may comprise aVIS-007-W01 plurality of shaped ends clustered together. In some embodiments, as shown by way of example only in Fig. 33, the finial 760 may comprise one or more spheres.

[0376] In some embodiments, the distal end 754 may include a distal tip 762 configured for placement on, in, or near target anatomy. In some embodiments, the distal tip 762 may be pointed to facilitate penetration of the distal tip 762 into target anatomy. In some embodiments, the distal tip 762 may be blunt to ensure that the distal tip 762 does not penetrate the target anatomy. In some embodiments, the distal end 754 may have a bend or angular offset 764 from a longitudinal axis of the elongated shaft 756 to improve positioning of the distal tip 762 against target anatomy (e.g., the ankle).

[0377] In some embodiments, the elongated shaft 756 may be rigid. In some embodiments, the elongated shaft 756 may be straight. In some embodiments, the elongated shaft 756 may be curved. For example, there may be scenarios when the location of a hidden posterior anatomy may be desired while scanning with an anterior view. In that case a pointer tool 750 having a curved shaft 756 configured to extend around the anatomy culminating in a distal tip 754 positioned at the target (hidden) anatomy, may also present for the anterior view a finial that enables registration of the hidden anatomy. In some embodiments, it may be possible to localize anatomic features hidden from view due to a limited exposure, for example, whereby the pointer distal tip 762 is within the surgical site but outside the field of view of the 3D scanner 12, while the proximal end is visible for localization. In this case, the pointer instrument 750 itself would be made of materials suitable for contact with the patient tissue, and sterilizable.

[0378] In some embodiments, the geometry of the handle 758 and / or finial 760 may indicate the spatial orientation of the distal tip 762.

[0379] By way of example, Fig. 34 is a block diagram depicting several steps in a method 800 for identifying specific anatomical points that are beyond the scanner’s field of view without performing calculations on visible anatomy or using optical trackers, according to some embodiments. In some embodiments, the method 800 may comprise the following steps.

[0380] In some embodiments, a first step 802 in the method 800 is preparing the 3D scanner system for topological registration. By way of example, before the procedure begins, the 3DVIS-007-W01 scanner system may be calibrated to capture and map visible anatomical surfaces within its field of view with high resolution, creating a precise 3D reference map. In some embodiments, this setup establishes a common coordinate system, which aligns with preoperative imaging or intraoperative reference models, providing a framework for accurate real-time tracking. In some embodiments, in the context of TKA, common coordinate system may be referenced to the 3D scanner 12, allowing the scanner 12 to serve as a stable reference frame for tracking the pointer and determining the ankle center. In some embodiments, as previously mentioned, the common coordinate system may be referenced to any object within the scanner’s field of view, including but not limited to the scanner 12, robot base, or a portion of the patient’s anatomy. In some embodiments, during this setup process the CAAT system 10 may be calibrated with precise specifications of the pointer instrument 750, including but not limited to length, orientation of the pointer tip 762 relative to a unique shape of the handle 758 or finial 760, and / or distance between the finial 760 and the distal tip 762 such that the CAAT system 10 can determine the precise location and orientation of the distal tip 762 when the finial 760 (or uniquely shaped handle 758) is positioned within the camera 12 field of view.

[0381] In some embodiments it may be necessary to calibrate the location of the distal tip 762 with respect to the finial 762, and not rely on precises specification. For example, the elongated shaft 756 may occasionally bend or get out of alignment over time and especially between cases. In some embodiments, the calibration procedure may entail 3D scanning along the length of the shaft 756 in a way that stitching of one more scan datasets may occur or be required, because it may be that the entirety of the pointer instrument 750 is not visible at once to the scanner 12. This could be facilitated by one or more regular markings on the pointer that are discernable at high resolution by the camera 12 and facilitate the dataset stitching. Practically, for the pointer instrument 750 to reach the ankle, only one dataset stitching may suffice. It is possible also that in addition to, or instead of, the markings, that unique 3D fiducial shapes be present on the pointer 750 that are visible to the scans being stitched, for purpose of facilitating the stitching. For this calibration procedure, the dataset stitching may occur using a subsection of the overall scanning volume that has been verified to be optimally accurate and precise, but that requires more stitching than would be required if the entire scanning volume were used. Otherwise, if the accuracy can maintain it, a single stitch could be used. To check the calibration, a known rotation of the pointer 750 could be applied and the calibration procedure performed again, toVIS-007-W01 make sure that the calibration procedure results in a relative position of the tip to the finial is consistent with the rotation, and not biased by systematic inaccuracies in the scanner volume.

[0382] In some embodiments, a second step 804 in the method 800 is positioning the pointer instrument 750 such that the distal tip 762 is placed on the medial and lateral malleoli. By way of example, with topological registration active, the surgeon places the pointer’s tip 762 sequentially on the medial and lateral malleoli of the ankle - two key bony landmarks necessary for calculating the ankle center. Throughout this process, the finial 760 remains within the 3D scanner 12 field of view, enabling continuous registration of the pointer 750 position even though the actual landmarks are outside the scan field.

[0383] In some embodiments, a third step 806 in the method 800 is real-time scanning and finial registration. By way of example, the 3D scanner 12 registers the unique shape and position of the finial 760 in real-time, continuously capturing its exact location within the common coordinate system. By tracking the finial 760, the CAAT system 10 accurately infers the pointer’s 750 full spatial orientation and position relative to each malleolus, using the pointer’s known geometry to extrapolate the location of the tip 762.

[0384] In some embodiments, a fourth step 808 in the method 800 is calculating the ankle center. By way of example, after marking both the medial and lateral malleoli, the CAAT system 10 uses these two points to calculate the approximate midpoint. Since the ankle center does not lie precisely at this midpoint due to anatomical asymmetry, the CAAT system 10 may apply a slight medial and superior offset based on anatomical data, refining the calculation to determine the true ankle center location. This inferred point, accurately calculated within the scanner’s common coordinate system, provides the surgeon with a reliable reference for lower limb alignment.

[0385] In some embodiments, a fifth step 810 in the method 800 is using the calculated ankle center as a reference for navigation. By way of example, once calculated, the ankle center serves as a critical reference point for intraoperative navigation and alignment. The surgeon can use this landmark to ensure precise alignment in TKA or other orthopedic procedures. This approach eliminates the need for additional optical trackers and provides a reliable,VIS-007-W01 straightforward method for integrating anatomical data beyond the scan field into the surgical workflow.

[0386] By way of example, the method 800 for identifying specific anatomical points that are beyond the scanner’s field of view without performing calculations on visible anatomy or using optical trackers disclosed herein applies broadly in TKA to locate necessary alignment points, such as the ankle center, using the medial and lateral malleoli as references (for example). However, this approach is adaptable to any procedure requiring anatomical point localization beyond the scan field, providing versatile use in various orthopedic and alignment-driven surgeries. By eliminating optical trackers and additional equipment, this method reduces setup complexity and enhances efficiency in the surgical workflow.

[0387] Unlike traditional tracking systems, the method 800 for identifying specific anatomical points that are beyond the scanner’s field of view without performing calculations on visible anatomy or using optical trackers disclosed herein achieves anatomical point localization without optical markers, allowing consistent reference even outside the scan field. No intraoperative calibration or setup is required for the pointer (e.g., the calibration described above may be done pre-operatively or optionally as regular check of pointer accuracy), reducing potential disruptions due to tracker shifts or line-of-sight interruptions. The fixed geometric reference from the finial 760 enables accurate inference of points like the ankle center, integrating any anatomical offsets automatically.

[0388] In some embodiments, for example, two 3D scanners may be used simultaneously, with one scanner essentially being the tip of the pointer. For example, a finial may be attached to a rod, and at the end of the rod could be a 3D scanner. In some embodiments, this scanner may scan the face of a cranial surgery patient in a prone position (face-down), where the face is out of view of personnel. By way of example, the finial could extend up and be scanned by a second scanner, while scanning the back of the head of the patient, thereby using the face to register the patient, and transferring that registration to its field of view (the back of the head). This is advantageous to current registration techniques which require a surgeon to gain access to the patient’s face anatomy for registration by crouching or crawling along the floor to gain an upward perspective on the patient’s face, and trace points for registration of that patient in theVIS-007-W01OR, while keeping their (relatively short) pointer in view of a stereoscopic camera system. Tn some embodiments, if there is no second scanner, the finial may be compatible with a hybrid navigation system where the finial is comprised of fiducial spheres or the like (optical markers, OMs).

[0389] Anatomical Verification Without Optical Trackers

[0390] By way of example, the methods for real-time verification of anatomical structures and bone resections disclosed herein utilize continuous or on-demand topological data scanning to replace conventional OT-based systems for checkpoints and resection validation. By eliminating the need for physical OTs, this invention provides a less invasive, streamlined, accurate solution for verifying anatomical consistency and bone resection accuracy. The approach significantly enhances workflow efficiency and precision in procedures such as TKA and other complex orthopedic surgeries where alignment and anatomical verification are critical.

[0391] In traditional surgical navigation systems, checkpoints can serve as verification points to ensure that OTs attached to anatomical structures, such as the femur, remain in a fixed position throughout the procedure. These checkpoints require a screw to be placed in the bone, creating a reference location that can be verified periodically using a pointer equipped with an OT. Any shift detected in the location of the checkpoint relative to the OT indicates potential movement, which requires the surgeon to pause and recalibrate the system before proceeding.

[0392] In some embodiments, this disclosure describes a method 830 for real-time verification of the positioning and orientation of anatomical structures that eliminates the need for both OTs and checkpoints by using real-time topological data to directly register and track an anatomical structure (e.g., the femur in a TKA procedure). Through continuous topological scanning, the system can monitor the femur’s position and orientation without requiring physical markers or trackers. This capability builds on the detailed registration and tracking methods disclosed above, which describe how topological data can be used to establish an accurate, continuous tracking framework.

[0393] By way of example, Fig. 35 is a block diagram illustrating several steps of a method for real-time verification of the positioning and orientation of an anatomical structures, accordingVIS-007-W01 to some embodiments. In some embodiments, a first step 832 in the method 830 for real-time verification of the positioning and orientation of an anatomical structure is to determine an initial baseline registration of the anatomical structure into common coordinate space (e.g., as described above). By way of example, at the outset of the surgical procedure, a plurality of high-resolution topological scans of the femur are captured for example using the 3D scanner 12, capturing surface contours and anatomical landmarks. These baseline scans provide a comprehensive dataset that defines the femur’s position within the surgical space, leveraging the topological registration principles described above.

[0394] In some embodiments, a second step 834 in the method 830 for real-time verification of the positioning and orientation of an anatomical structure is continuous topological tracking of the anatomical structure. By way of example, as the surgical procedure advances, the CAAT system 10 may continuously collect topological data, comparing live scans of the femur to the baseline registration. In some embodiments, any movement or shift is detected by the CAAT system 10 in real time, allowing the CAAT system 10 to track the femur’s exact position without physical trackers or checkpoints, using the tracking methodologies described above.

[0395] In some embodiments, the CAAT system 10 may employ a process referred to as “transfer registration” as a third step 836 of the method 830 for real-time verification of the positioning and orientation of an anatomical structure. By way of example, in transfer registration, each newly acquired optical scan is registered to a prior reference scan representing the immediately preceding anatomical state. In some embodiments, each registration thereby links the current intraoperative anatomical state to the previous anatomical state. Through a sequence of intermediate anatomical states, the current anatomical state is linked to the original anatomic reference model. Because each transformation is based on successive stages of anatomy exhibiting maximum geometric commonality, the resulting registrations are statistically robust to cumulative alignment error.

[0396] In some embodiments, successful registrations rely on statistical sufficiency of common anatomical features between consecutive stages. It is important in transfer registration to maintain common anatomy sufficient for confident registration in all six degrees of freedom. For this reason, it is likely that scans will take place at standard timepoints in aVIS-007-W01 procedure. Metrics on the number of inliers and their distance, and statistical outputs such as covariant matrices could enable the system to quantify the robustness of each stage’s registration.

[0397] In surgical contexts such as TKA or spine surgery, transfer registration may be used to update the reference model as anatomical modifications occur. For example, in TKA, the updated reference may exclude bone and / or cartilage that has been resected and include newly exposed cut surfaces. In spine procedures, for example, the reference may similarly exclude removed bony structures while incorporating newly revealed rigid anatomy. Rigidly affixed or relatively immobile tissue may also contribute to stable registration across stages when such tissue remains unaffected by the surgical manipulation. Although described primarily in the context of a TKA, transfer registration is suitable for many procedures where anatomy is affected, for example resection of bone, and is not limited to those discussed here.

[0398] In some embodiments, between major stages of anatomical change, the system may acquire new optical scans to update the current reference frame. For example, during stages in which the anatomy remains relatively unchanged (e.g., during drilling or probing operations), the system may rely on continuous motion tracking relative to the most recent reference, maintaining spatial accuracy in real time. In this manner, the system combines stage-based re-registration with intra-stage tracking to preserve registration accuracy throughout the procedure.

[0399] This approach allows intraoperative registration updates without cumulative drift exceeding acceptable limits, provided that each individual registration meets accuracy requirements and sufficient anatomical overlap exists between consecutive states.

[0400] Without the need for periodic OT verifications or recalibrations, the surgeon can proceed through the procedure uninterrupted (e.g., a third step 836 of the method 830). This continuous tracking reduces workflow disruptions, enhances accuracy, and removes reliance on physical markers that are subject to movement or accidental shifts. By replacing traditional OT- dependent checkpoints with real-time topological tracking, the method 830 streamlines the verification process, ensuring consistent and precise anatomical registration throughout the procedure.

[0401] Verification of Osteotomies and Bone Resections Using Topological DataVIS-007-W01

[0402] In TKA and similar procedures, accurate bone resection is critical for ensuring that implants align correctly with the surgical plan. Traditionally, OT-based tools with planar surfaces are used to verify each cut, comparing the resected surfaces against the preoperative plan. This approach requires precise OT calibration and positioning, which adds procedural complexity and can lead to cumulative error.

[0403] By way of example, Fig. 36 is a block diagram illustrating several steps of a method 850 for real-time verification of osteotomies and bone resections that eliminates the need for OT- based instruments by leveraging topological data in tandem with CT-based registration to verify location and orientation of osteotomies and bone resections relative to an original surgical plan. Using high-resolution scans before and after each resection, the method 850 provides the surgeon with a streamlined, tracker-free means of validating the accuracy of each cut.

[0404] In some embodiments, a first step 852 of the method 850 for real-time verification of osteotomies and bone resections is initial registration of CT and scan data. By way of example, before any bone resection, the anatomical site is registered to a common coordinate system by aligning preoperative CT data with an initial topological scan, for example according to the registration procedure described above. This foundational registration process aligns the CT data with the surgical field and sets up the common coordinate system. This enables the surgical navigation system to precisely understand the location and orientation of the femur (for example) relative to the planned resection planes.

[0405] In some embodiments, a second step 854 of the method 850 for real-time verification of osteotomies and bone resections is assisted bone resection. For example, once the femur is registered, the resection may be performed with robotic assistance or guided using real-time topological scanning to ensure precision during the cut. Methods for guiding cuts using topological data and robotic assistance are detailed above, providing reference for systems that integrate cutting and tracking functions.

[0406] In some embodiments, a third step 856 of the method 850 for real-time verification of osteotomies and bone resections is post-resection scan and idealized model comparison. For example, after completing the resection, one or more new topological scans of the surgical site may be taken. To facilitate accurate post-resection verification, the initial CT-based femurVIS-007-W01 model may be modified to reflect the planned resection by virtually removing the cut portion of the bone from the data set. This creates an idealized, post-resection model of the femur, preserving only the surfaces that remain after the resection. During verification and / or tracking, the system registers this modified CT dataset with the new post-resection scan.

[0407] In some embodiments, a fourth step 858 of the method 850 for real-time verification of osteotomies and bone resections is comparison of the resected surface with surgical plan. For example, with the modified CT model aligned to the current scan, the system can identify the planar surface of the bone cut and compare its position and orientation against the planned resection plane. This comparison allows the system to calculate any deviations, providing immediate feedback on the location and orientation of the cut relative to the original surgical plan.

[0408] In some embodiments, a fifth step 860 of the method 850 for real-time verification of osteotomies and bone resections is surgeon decision-making. For example, if the difference between the actual cut and the planned resection exceeds predefined tolerances, this system may alert the surgeon, who can then assess whether the cut is close enough to proceed or if adjustments to the surgical plan are needed to accommodate the discrepancy.

[0409] By registering and verifying the cut using both CT-based and real-time topological data, this disclosure provides precise and reliable feedback on resection accuracy without relying on OT-based tools. This method enables seamless, data-driven validation of each bone resection, ensuring high alignment accuracy and reducing the need for additional instrumentation or calibration.

[0410] Figs 37-38 are example block diagrams of computer-implemented electronic devices 500, 550 that may be used to implement the systems and methods described in this document, as either a client or as a server or plurality of servers. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers.Computing device 550 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart-phones, and other similar computing devices. In this example, computing device 500 may represent stationary computer or hub 12 and / or otherVIS-007-W01 computing systems referenced in this disclosure. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations described and / or claimed in this document.

[0411] Referring to Fig. 37, computing device 500 includes a processor 502, memory 504, a storage device 506, a high-speed interface 508 connecting to memory 504 and high-speed expansion ports 510, and a low-speed interface 512 connecting to low-speed bus 514 and storage device 506. Each of the components 502, 504, 506, 508, 510, and 512, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor 502 can process instructions for execution within the computing device 500, including instructions stored in the memory 504 or on the storage device 506 to display graphical information for a graphic user interface (GUI) on an external input / output device, such as display 516 coupled to high-speed interface 508. In other implementations, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices 500 may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

[0412] The memory 504 stores information within the computing device 500. By way of example only, the memory 504 may be a volatile memory unit, non-volatile memory unit, or another form of computer-readable medium, such as a magnetic or optical disk (for example).

[0413] The storage device 506 is capable of providing mass storage for the computing device 500. In one implementation, the storage device 506 may be or contain a non-transitory computer-readable medium (e.g., any and all computer-readable media except transitory, propagating signals), such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. A computer program product can be tangibly embodied in an information carrier. The computer program product may also contain instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 504, the storage device 506, or memory on processor 502.VIS-007-W01

[0414] The high-speed interface 508 manages bandwidth-intensive operations for the computing device 500, while the low-speed interface 512 manages lower bandwidth-intensive operations. Such allocation of functions is by way of example only. In one implementation, the high-speed interface 508 is coupled to memory 504, display 516 (e.g., through a graphics processor or accelerator), and to high-speed expansion ports 510, which may accept various expansion cards (not shown). In the implementation, low-speed interface 512 is coupled to storage device 506 and low-speed expansion port 514. The low-speed expansion port may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) and may be coupled to one or more input / output devices, such as a keyboard 518, a printer 520, a scanner 522, or a networking device such as a switch or router 524, e.g., through a network adapter.

[0415] The computing device 500 may be implemented in a number of different forms. For example, it may be implemented as a standard server, or multiple times in a group of such servers. It may also be implemented as part of a rack server system. In addition, it may be implemented in a personal computer such as a laptop computer. Alternatively, components from computing device 500 may be combined with other components in a mobile device, such as device 550 (Fig. 38). Each of such devices may contain one or more of computing device 500, 550, and an entire system may be made up of multiple computing devices 500, 550 communicating with each other.

[0416] Referring to Fig. 38, computing device 550 includes a processor 552, memory 554, an input / output device such as a display 556, a communication interface 558, and a transceiver 560, among other components. The device 550 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components 550, 552, 554, 556, 558, and 560, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

[0417] The processor 552 can execute instructions within the computing device 550, including instructions stored in the memory 554. The processor may be implemented as a chipset of chips that include separate and multiple analog and digital processors. Additionally, the processor may be implemented using any of a number of architectures. For example, theVIS-007-W01 processor 552 may be a CISC (Complex Instruction Set Computers) processor, a RISC (Reduced Instruction Set Computer) processor, or a MISC (Minimal Instruction Set Computer) processor. The processor may provide, for example, for coordination of the other components of the device 550, such as control of user interfaces, applications run by device 550, and wireless communication by device 550.

[0418] The processor 552 may communicate with a user through control interface 562 and display interface 564 coupled to a display 556. The display 556 may be, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display) display or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface 564 may comprise appropriate circuitry for driving the display 556 to present graphical and other information to a user. The control interface 562 may receive commands from a user and convert them for submission to the processor 552. In addition, an external interface 566 may be provided in communication with processor 552, so as to enable near area communication of device 550 with other devices. External interface 566 may provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used.

[0419] The memory 554 stores information within the computing device 550. The memory 554 can be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. Expansion memory 568 may also be provided and connected to device 550 through expansion interface 570, which may include, for example, a SIMM (Single In Line Memory Module) card interface. Such expansion memory 568 may provide extra storage space for device 550 or may also store applications or other information for device 550. Specifically, expansion memory 568 may include instructions to carry out or supplement the processes described above and may include secure information also. Thus, for example, expansion memory 568 may be provided as a security module for device 550 and may be programmed with instructions that permit secure use of device 550. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner. In some embodiments, and by way of example only, the memory 554 may also include one or more of aVIS-007-W01 log of scanner use, calibration information, hardware and software versions, date of assembly, hardware configuration, date last calibrated, service history, and the like.

[0420] The memory may include, for example, flash memory and / or NVRAM memory, as discussed below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, cause performance of one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 554, expansion memory 568, or memory on processor 552 that may be received, for example, over transceiver 560 or external interface 566.

[0421] Device 550 may communicate wirelessly through communication interface 558, which may include digital signal processing circuitry where necessary. Communication interface 558 may provide for communications under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication may occur, for example, through radio-frequency transceiver 560. In addition, short-range communication may occur, such as using a Bluetooth, WiFi, or other such transceiver (not shown). In addition, GPS (Global Positioning System) receiver module 572 may provide additional navigation- and location-related wireless data to device 550, which may be used as appropriate by applications running on device 550.

[0422] Device 550 may also communicate audibly using for example audio codec 574, which may receive spoken information from a user and convert it to usable digital information. Audio codec 574 may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of device 550. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, etc.) and may also include sound generated by applications operating on device 550.

[0423] The computing device 550 may be implemented in a number of different forms, some of which are shown in the figure. For example, it may be implemented as a cellular telephone. It may also be implemented as part of a smart-phone, personal digital assistant, or other similar mobile device.VIS-007-W01

[0424] Additionally computing device 500 or 550 can include Universal Serial Bus (USB) flash drives. The USB flash drives may store operating systems and other applications. The USB flash drives can include input / output components, such as a wireless transmitter or USB connector that may be inserted into a USB port of another computing device.

[0425] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0426] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor.

[0427] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0428] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middlewareVIS-007-W01 component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), peer-to-peer networks (having ad-hoc or static members), grid computing infrastructures, and the Internet.

[0429] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0430] Additionally, by way of example, the computer system may include one or more specialized components such as GPU, field programmable gate array (FPGA), digital signal processor (DSP), neural processing unit (NPU), high speed camera interfaces, PCle, etc.

[0431] Definitions

[0432] As used herein, a “3D Scanner” is a device that captures three-dimensional (3D) topological data of the surgical field, including anatomical structures and surgical tools, in real time. While there are multiple types of 3D scanners, this disclosure focuses on a proprietary 3D structured light scanner. However, any 3D topological scanner capable of collecting similar data may be used, as this disclosure emphasizes the use of the scanner and its data rather than the emphasizing the scanner itself.

[0433] As used herein, “continuous anatomical automatic tracking (CAAT)” is a system feature that continuously updates the position and orientation of anatomical structures and instruments by collecting data with a 3D scanner. In some embodiments, this process eliminates the need for physical optical markers, offering real-time tracking throughout the surgical procedure.VIS-007-W01

[0434] As used herein, “gap balancing” is a technique used to ensure equal spacing (gaps) between bones in both flexion and extension during surgeries like Total Knee Arthroplasty (TKA). Accurate gap balancing helps maintain proper joint stability and alignment.

[0435] As used herein, a “hub” is a computer responsible for processing raw data from the 3D scanner and automatically registering it to the Surgical Coordinate System.

[0436] As used herein, an “iterative closest point (ICP)” is an algorithm that refines the registration process by minimizing the distance between corresponding points in two datasets for precise alignment.

[0437] As used herein, an “osteotomy” is a surgical procedure that involves cutting and reshaping bones, commonly used in orthopedic surgeries to correct deformities or prepare the bone for implants.

[0438] As used herein, a “point cloud” is a set of points in 3D space, each representing a specific location on a scanned surface, used to generate a topological map of the surgical field. By way of example, the 3D scanner described herein provides a particularly dense point cloud, offering high-resolution data that enhances precision in surgical procedures, and it does it at >25 rps, with high accuracy, optionally in the near-infrared, and with an advantageous illumination coverage and depth of field in the pattern projection.

[0439] As used herein, a “random sample consensus (RANSAC)” is an algorithm used for initial coarse registration by selecting key points in the dataset and filtering out outliers.

[0440] As used herein, “real-time tracking” is defined as continuous registration of anatomical structures or tools, updating multiple times per second to reflect movements and changes in position within the surgical field. This ensures that data is captured and updated in real time without noticeable delays.

[0441] As used herein, a “robot-mounted scanner” is a 3D scanner attached to the robotic arm, or to the end effector of a robotic arm that allows the scanner to dynamically track anatomical structures as the robot moves during surgery.VIS-007-W01

[0442] As used herein, “sphere fitting” is a method used to estimate the center of rotation of a joint, such as the hip, by fitting a mathematical sphere to points collected during motion tracking of the femur.

[0443] As used herein, a “structured light pattern” is a method of projecting a predefined light pattern (such as grids, stripes, or dots) onto surfaces in order to capture 3D topological data.

[0444] As used herein, a “user interface (UI)” is a visual interface that provides real-time visualizations and guides the surgeon through key steps in the procedure.

[0445] As used herein, “varus angles” and “valgus angles” are the angles that represent the inward (varus) or outward (valgus) tilting of bones, often measured during surgeries to assess and correct misalignment in joints like the knee.

[0446] As used herein, the term “sagittal plane” is an anatomical plane that divides the body into left and right halves and is typically used to measure medial-lateral alignment. As used herein, the term “coronal or frontal plane” is an anatomical plane that divides the body into anterior (front) and posterior (back) sections, separating the joint into anterior and posterior components. As used herein, the term “transverse or axial plane” is an anatomical plane that divides the body into superior (upper) and inferior (lower) sections, often used for rotational alignment assessments.

[0447] As used herein, the term “flexion / extension (F / E) angle” is an angular measurement in the sagittal plane between the bones of the joint, with flexion being a positive angle representing bending, and extension being closer to or at 0°, representing a straightened joint. A negative number would indicate a hyperextended knee joint. As used herein, the term “internal / external rotation angle” is an angular measurement of rotational alignment in the transverse plane.Internal rotation is a positive value, indicating inward rotation, while external rotation is negative, indicating outward rotation. As used herein, the term “varus / valgus (V / V) angle” is an angular measurement between the bones in the joint in the coronal plane. A positive V / V angle indicates a varus (medial / inward) orientation, while a negative V / V angle represents a valgus (lateral / outward) orientation.VIS-007-W01

[0448] As used herein, the term “contact distance (D)” is the measured or estimated distance between two key contact points on the joint surfaces, such as the medial and lateral edges of the tibial plateau or femoral condyles. This distance serves as a lever arm in trigonometric calculations of joint gap changes under applied stress. This value can be preset in the software UI but would be changeable by the surgeon.

[0449] As used herein, the term “joint laxity” is the flexibility or looseness of a joint under applied stress, reflecting the joint’s capacity to withstand movement without compromising stability. Joint laxity measurements are critical for ensuring balanced load distribution and stability, particularly in joint replacement and repair procedures. Joint laxity is not normally a direct measurement but inferred by measuring gaps between bones.

[0450] As used herein, the term “peak gap” is the maximum gap achieved between two articulating surfaces under applied stress. Real-time tracking of peak gaps is essential for an accurate assessment of joint laxity, especially when continuous force application is challenging.

[0451] As used herein, the term “topological data collection” is a non-invasive scanning technique that captures the 3D surface geometry of anatomical structures, creating high- resolution data that maps out contours, ridges, and other surface features. This data provides a basis for real-time measurement and tracking of relative positions without requiring external markers or optical trackers.

[0452] As used herein, the term “zero-stress condition” is a baseline position where no additional forces (such as varus, valgus, or rotational stresses) are applied to the joint. In this state, the gap between the articulating bones is assumed to be neutral or zero, serving as a reference for all subsequent relative measurements under stress.

[0453] As used herein, the term “camera space” refers to the coordinate system within which the 3D scanner operates, typically based on the position and orientation of the scanner’s sensors. Data points, such as the femoral origin, are transformed into this space to maintain consistency in calculations relative to the scanner’s viewpoint.

[0454] As used herein, the term “confidence ellipsoid” refers to a three-dimensional, ellipsoidal boundary calculated around the estimated hip center to quantify the confidence in theVIS-007-W01 accuracy of this estimation. The confidence ellipsoid indicates the region within which the true hip center is likely to lie, based on statistical analysis. A 95% confidence ellipsoid means there is a 95% probability that the true center falls within this boundary.

[0455] As used herein the term “field of view” refers to the spatial area visible to the 3D scanner at any given time. As the knee moves within this area, the scanner captures data on the femoral origin’s position, contributing to the sphere-fitting calculations.

[0456] As used herein the term “femoral origin” refers to a reference point on the femur used to track the femur’s position during surgery. In the method disclosed herein, the femoral origin points are captured in 3D space as the knee rotates, providing data points that contribute to the sphere calculation.

[0457] As used herein the term “least-squares sphere fitting” is a mathematical method for finding the best-fit sphere to a set of 3D data points by minimizing the sum of squared residuals. By way of example, in the method disclosed herein it is used to estimate the center of rotation of the hip by fitting a sphere to the chosen femoral points captured by the scanner.

[0458] As used herein the term “residual” refers to the difference between the observed position of a data point and its expected position on the fitted spherical surface. By way of example, residuals are calculated for each data point to assess the variation of the error of the sphere fit and are minimized through the least-squares sphere-fitting method.

[0459] As used herein the term “topological data” refers to three-dimensional information representing the surface geometry of an object, in this case, the femur. Topological data enables accurate modeling of anatomical structures for surgical reference.

[0460] As used herein the term “advanced scanner” refers to a structured light 3D scanner used intraoperatively to capture topological data of anatomical structures, for example such as the scanner disclosed in the ‘973 app (attached as Appendix A). In this method, the advanced scanner collects real-time data on femoral origin points, which are used to estimate the hip center of rotation.VIS-007-W01

[0461] As used herein, the term “topological data collection” refers to a non-invasive scanning technique capturing the 3D surface geometry of anatomical structures, producing high- resolution data that maps anatomical contours.

[0462] As used herein, the term “anatomical point inference” refers to the process of inferring the position of anatomical landmarks outside the scanner’s field of view, based on the pointer’s known geometry and registered finial position.

[0463] As used herein, the term "coupled" is defined as connected, although not necessarily directly, and not necessarily mechanically. The use of the word "a" or "an" when used in conjunction with the term "comprising" in the claims and / or the specification may mean "one," but it is also consistent with the meaning of "one or more" or "at least one." The term "about" means, in general, the stated value plus or minus 5%. The use of the term "or" in the claims is used to mean "and / or" unless explicitly indicated to refer to alternatives only or the alternative are mutually exclusive, although the disclosure supports a definition that refers to only alternatives and "and / or."

[0464] As used herein, the terms "comprise" (and any form of comprise, such as "comprises" and "comprising"), "have" (and any form of have, such as "has" and "having"), "include" (and any form of include, such as "includes" and "including") and "contain" (and any form of contain, such as "contains" and "containing") are open-ended linking verbs. As a result, a method or device that "comprises," "has," "includes" or "contains" one or more steps or elements, possesses those one or more steps or elements, but is not limited to possessing only those one or more elements. Likewise, a step of a method or an element of a device that "comprises," "has," "includes" or "contains" one or more features, possesses those one or more features, but is not limited to possessing only those one or more features. Furthermore, a device or structure that is configured in a certain way is configured in at least that way but may also be configured in ways that are not listed.

[0465] All patents and publications mentioned in this specification are indicative of the levels of those skilled in the art to which the disclosure pertains. It is to be understood that while a certain form of the disclosure is illustrated, it is not to be limited to the specific form or arrangement herein described and shown. It will be apparent to those skilled in the art that noVIS-007-W01 various changes may be made without departing from the scope of the disclosure and the disclosure is not to be considered limited to what is shown and described in the specification and any drawings / figures included herein.

[0466] One skilled in the art will readily appreciate that the present disclosure is well adapted to carry out the objectives and obtain the ends and advantages mentioned, as well as those inherent therein. The embodiments, methods, procedures and techniques described herein are presently representative of the preferred embodiments, are intended to be exemplary and are not intended as limitations on the scope. Changes therein and other uses will occur to those skilled in the art which are encompassed within the spirit of the disclosure and are defined by the scope of the appended claims. Although the disclosure has been described in connection with specific preferred embodiments, it should be understood that the disclosure as claimed should not be unduly limited to such specific embodiments. Indeed, various modifications of the described modes for carrying out the disclosure which are obvious to those skilled in the art are intended to be within the scope of the following claims.

Claims

VIS-007-W01CLAIMSWhat is claimed is:

1. A system for measuring gaps between a first bone and a second bone within an affected joint having a medial side and a lateral side in an operating room environment during a surgical procedure, comprising: a structured light three-dimensional scanner, including: a light source: a liquid crystal matrix; and an image capture device; a computer system including a memory and at least one processor; and computer readable media embodied in a non-transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a first three-dimensional data representation of a topological surface of the first bone using a first dataset, the first dataset derived from one or more images of the topological surface of the first bone captured by the image capture device without optical trackers or applied fiducials; register the first three-dimensional data representation to a second dataset such that the first three-dimensional data representation and the second dataset are aligned within a common coordinate system, the registration occurring without optical trackers or applied fiducials; track the first bone relative to the common coordinate system without optical trackers or applied fiducials; construct a second three-dimensional data representation of a topological surface of a second bone using a third dataset, the third dataset derived from one or more images of the topological surface of the second bone captured by the image capture device without optical trackers or applied fiducials; register the second three-dimensional data representation to a fourth dataset such that the second three-dimensional data representation and the fourth dataset are aligned within the common coordinate system without optical trackers or applied fiducials;VIS-007-W01 track the second bone relative to the common coordinate system without optical trackers or applied fiducials; and calculate, with the affected joint under applied stress, at least one of a medial gap value, a lateral gap value, and a change in varus-valgus angle.

2. The system of claim 1, wherein the light source is a linear light source.

3. The system of claim 1, wherein the first three-dimensional data representation is a point cloud or three-dimensional data convertible into a point cloud.

4. The system of claim 1, wherein the captured images of the topological surface of the first bone are structured light patterned images that encode the surface of the first bone.

5. The system of claim 1, wherein the first three-dimensional data representation of the topological surface of the first bone is constructed by: decoding light patterns contained in the one or more captured images of the topological surface of the first bone by analyzing brightness history to create camera pixel correspondence to light angle projection; using triangulation to calculate a three-dimensional coordinate for each pixel in the one or more captured images of the topological surface of the first bone based on the decoded light patterns; and combining the triangulated three-dimensional coordinates of each pixel into the first three-dimensional point cloud representation of the surface of the first bone in physical space.

6. The system of claim 1, wherein the second dataset is derived from a preoperative computed tomography scan or magnetic resonance imaging scan of the first bone.

7. The system of claim 1, wherein the second dataset is derived from one or more subsequent images captured by the image capture device without said optical trackers or applied fiducials.

8. The system of claim 1, wherein the first bone is a femur, the second bone is a tibia, and the affected joint is a knee.VIS-007-W019. The system of claim 1, wherein: the computer system further comprises a user interface and a display device, and the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display, on the display device, the tracked first three-dimensional data representation of the first bone and the tracked second three-dimensional data representation of the second bone.

10. The system of claim 9, wherein the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display, on the display device, a real-time determination of at least one of a flexion-extension angle, varus-valgus angle, and interior-exterior rotation angle based on the tracked first three-dimensional data representation of the first bone and the tracked second three- dimensional data representation of the second bone.

11. The system of claim 1, wherein tracking the first bone relative to the common coordinate system comprises continuously updating the first bone’s position relative to the common coordinate system by repeatedly acquiring surface data, registering the repeatedly acquired surface data in the common coordinate system, and updating transformation matrices.

12. The system of claim 1, wherein the computer readable media includes a further set of instructions, that, when executed by a processor, cause the computer system to: establish a zero-gap varus-valgus angle and zero gap condition of the first bone and second bone while the affected joint is positioned in an extension orientation with no stress applied to the affected joint; and record the established zero-gap varus-valgus angle and zero gap condition as a baseline measurement.

13. The system of claim 1, wherein the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to calculate the lateral gap value by: measuring a test varus-valgus angle of the affected joint with varus stress applied to the affected joint, thereby opening the lateral gap between the first bone and secondVIS-007-W01 bone on the lateral side and creating a stable contact between the first bone and second bone on the medial side; comparing the measured test varus-valgus angle of the affected joint to the baseline measurement varus-valgus angle to calculate the change in varus-valgus angle of the affected joint; recording peak lateral gap measurements by continuously tracking the first bone and second bone in real time; determining a known articular contact distance between contact points on the first bone and second bone; calculating the lateral gap value based on the determined articular contact distance and the change in varus-valgus angle of the affected joint.

14. The system of claim 1, wherein the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to calculate the medial gap value by: measuring a test varus-valgus angle of the affected joint with valgus stress applied to the affected joint, thereby opening the medial gap between the first bone and second bone on the medial side and creating a stable contact between the first bone and second bone on the lateral side; comparing the measured test varus-valgus angle of the affected joint to the baseline measurement varus-valgus angle to calculate the change in varus-valgus angle of the affected joint; recording peak lateral gap measurements by continuously tracking the first bone and second bone in real time; determining a known articular contact distance between contact points on the first bone and second bone; calculating the medial gap value based on the determined articular contact distance and the change in varus-valgus angle of the affected joint.

15. A system for calculating hip center of rotation during a surgical procedure in an operating room environment, comprising: a structured light three-dimensional scanner, including:VIS-007-W01 a light source: a liquid crystal matrix; and an image capture device; a computer system including a memory and at least one processor; and computer readable media embodied in a non-transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: use a three-dimensional data representation of a topological surface of a femur constructed from a first dataset, the first dataset derived from a preoperative imaging event; obtain a plurality of second datasets from the image capture device during a plurality of image capture events, each second dataset of the plurality of second datasets comprising topological scan data of the femur derived from one or more images intraoperatively captured by the image capture device during an image capture event of the plurality of image capture events without optical trackers or applied fiducials; register the three-dimensional data representation to each second dataset of the plurality of second datasets in a series of registrations to find a transformation matrix such that the three-dimensional data representation and the plurality of second datasets are aligned within a common coordinate system, the registrations occurring without said optical trackers or applied fiducials; track the femur relative to the common coordinate system without said optical trackers or applied fiducials; use the found transformation matrix to compute a coordinate location of a chosen anatomic point on the three-dimensional data representation in a common coordinate system stationary with respect to the hip center of rotation for each registration of the series of registrations such that a plurality of coordinate locations is generated for sphere fitting; and calculate a size and location of a best-fit sphere to the plurality of generated coordinate locations using a sphere-fitting algorithm; establish and check the quality of the best-fit sphere; andVIS-007-W01 report the center of the best-fit sphere as the hip center of rotation.

16. The system of claim 15, wherein the sphere-fitting algorithm is a least squares fitting method.

17. The system of claim 15, wherein the light source is a linear light source.

18. The system of claim 15, wherein the three-dimensional data representation is a point cloud or three-dimensional data convertible into a point cloud.

19. The system of claim 15, wherein the captured images are structured light patterned images that encode the surface of the physical object.

20. The system of claim 15, wherein the first dataset is derived from a preoperative computed tomography scan or magnetic resonance imaging scan.

21. The system of claim 15, wherein: the computer system further comprises a user interface and a display device, and the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display, on a display device, the tracked three-dimensional data representation of the femur.

22. The system of claim 15, wherein tracking the femur relative to the common coordinate system comprises continuously updating the femur’s position relative to the common coordinate system by repeatedly acquiring surface data, registering the repeatedly acquired surface data in the common coordinate system, and updating transformation matrices.

23. A surgical pointer tool, comprising: a rigid body comprising a proximal end comprising a handle and a fiducial element, the fiducial element having a surface topology configured for optimized localization by a surgical navigation system using a 3D scanner; a distal end including a pointer tip, and an elongated shaft extending between the proximal end and the distal end;VIS-007-W01 wherein the fiducial element has a distinct orientation relative to the pointer tip such that the orientation of the fi ducial element reliably indicates the position and orientation of the pointer tip.

24. The surgical pointer tool of claim 23, wherein the fiducial element comprises a finial.

25. The surgical pointer tool of claim 23, wherein the fiducial element comprises a plurality of spheres.

26. The surgical pointer tool of claim 23, wherein the fiducial element comprises a unique geometric shape.

27. The surgical pointer tool of claim 23, wherein the elongated shaft includes a curved portion.

28. The surgical pointer tool of claim 27, wherein the curved portion is located near the distal end.

29. The surgical pointer tool of claim 23, wherein the elongated shaft has a linear configuration.

30. The surgical pointer tool of claim 23, wherein the pointer tip comprises a blunt tip.

31. The surgical pointer tool of claim 23, wherein the rigid body is composed of biocompatible material.

32. A system for instrument-based anatomical point location using topology registration, comprising: a structured light three-dimensional scanner, including: a light source: a liquid crystal matrix; and an image capture device having a field of view; a surgical pointer tool comprising a rigid body having a proximal end comprising a handle and a fiducial element, a distal end including a pointer tip, and an elongated shaft extending between the proximal end and the distal end;VIS-007-W01 a computer system including a memory and at least one processor; and computer readable media embodied in a non-transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a three-dimensional data representation of a topological surface of a physical object using a first dataset, the first dataset derived from one or more images captured by the image capture device without optical trackers or applied fiducials; register the three-dimensional data representation to a second dataset such that the three-dimensional data representation and the second dataset are aligned within a common coordinate system, the registration occurring without said optical trackers or applied fiducials; and track the physical object relative to the common coordinate system without said optical trackers or applied fiducials; determine a location of the pointer tip of the surgical pointer tool positioned outside of the field of view of the image capture device based on the position and orientation of the fiducial element positioned within the field of view of the image capture device.

33. The system of claim 32, wherein the light source is a linear light source.

34. The system of claim 32, wherein the fiducial element has a surface topology configured for optimized localization the structured light three-dimensional scanner.

35. The system of claim 32, wherein the three-dimensional data representation is a point cloud or three-dimensional data convertible into a point cloud.

36. The system of claim 32, wherein the captured images are structured light patterned images that encode the surface of the physical object.

37. The system of claim 32, wherein the physical object is an anatomical structure in a surgical procedure.

38. The system of claim 32, wherein the three-dimensional data representation of theVIS-007-W01 topological surface of the physical object is constructed by: decoding light patterns contained in the one or more captured images by analyzing brightness history to create camera pixel correspondence to light angle projection; using triangulation to calculate a three-dimensional coordinate for each pixel in the one or more captured images based on the decoded light patterns; and combining the triangulated three-dimensional coordinates of each pixel into the three-dimensional point cloud representation of the surface of the physical object in physical space.

39. The system of claim 32, wherein the three-dimensional data representation of the physical object comprises a first three-dimensional data representation of a first physical object, and the computer readable media includes a further set of instructions, that, when executed by a processor, cause the computer system to: construct a second digital three-dimensional data representation of a topological surface of a second physical object using a third dataset, the third dataset derived from one or more images of surface topology of the second physical object captured by the image capture device using infrared light and without the use of optical trackers or applied fiducials; register the second three-dimensional data representation to a fourth dataset such that the second three-dimensional data representation and the fourth dataset are aligned with one another and the with the first three-dimensional data representation and second dataset within the common coordinate system without the use of optical trackers or applied fiducials; continuously track the second physical object relative to the common coordinate system without the use of optical trackers or applied fiducials.

40. The system of claim 39, wherein the first physical object is a first anatomical structure and the second physical object is a second anatomical structure.

41. The system of claim 39, wherein the first physical object is an anatomical structure and the second physical object is a surgical instrument.VIS-007-W0142. The system of claim 41, wherein the surgical instrument is the pointer tool.

43. The system of claim 32, wherein the second dataset is derived from a preoperative computed tomography scan or magnetic resonance imaging scan.

44. The system of claim 32, wherein: the computer system further comprises a user interface and a display device, and the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display, on a display device, the tracked three-dimensional data representation of the physical object.

45. The system of claim 32, wherein tracking the physical object relative to the common coordinate system comprises continuously updating the physical object’s position relative to the common coordinate system by repeatedly acquiring surface data, registering the repeatedly acquired surface data in the common coordinate system, and updating transformation matrices.

46. The system of claim 32, wherein the surgical pointer tool is positioned such that the pointer tip is touching a target anatomy location.

47. The system of claim 32, wherein the fiducial element comprises a finial.

48. The system of claim 32, wherein the fiducial element comprises a plurality of spheres.

49. The system of claim 32, wherein the fiducial element comprises a unique geometric shape.

50. The system of claim 32, wherein the elongated shaft includes a curved portion.

51. The system of claim 32, wherein the curved portion is located near the distal end.

52. The system of claim 32, wherein the elongated shaft has a linear configuration.

53. The system of claim 32, wherein the pointer tip comprises a blunt tip.

54. The system of claim 32, wherein the rigid body is composed of biocompatible material.VIS-007-W0155. The system of claim 32, wherein the computer readable media includes a further set of instructions, that, when executed by a processor, cause the computer system to: calibrate the location of the pointer tip of the surgical pointer tool with respect to the fiducial element.

56. The system of claim 55, further comprising a calibration attachment configured to engage the pointer tip during a calibration event.

57. The system of claim 56, wherein the calibration attachment is configured to engage the pointer tip in a defined seating geometry.

58. The system of claim 32, wherein the computer readable media includes a further set of instructions, that, when executed by a processor, cause the computer system to: determine a location of an inaccessible anatomical point positioned between two accessible anatomical points, by: determining a first location of the pointer tip when the pointer tip is touching a first portion of patient anatomy; determining a second location of the pointer tip when the pointer tip is touching a second portion of patient anatomy; and calculating a midpoint between the first and second determined locations.

59. A system for continuous registration and tracking of objects in an operating room environment, comprising: a structured light three-dimensional scanner, including: a light source: a liquid crystal matrix; and an image capture device; a computer system including a memory and at least one processor; and computer readable media embodied in a non-transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a three-dimensional data representation of a topological surface of a physical object using a first dataset, the first dataset derived from one or moreVIS-007-W01 images captured by the image capture device and without optical trackers or applied fiducials; register the three-dimensional data representation to a second dataset such that the three-dimensional data representation and the second dataset are aligned within a common coordinate system, the registration occurring without said optical trackers or applied fiducials; and track the physical object relative to the common coordinate system without said optical trackers or applied fiducials; wherein tracking the physical object relative to the common coordinate system comprises: continuously updating the physical object’s position relative to the common coordinate system by repeatedly acquiring surface data through subsequent image capture events by the image capture device; registering the repeatedly acquired surface data in the common coordinate system; updating transformation matrices; and continuously updating the three-dimensional data representation based on the repeatedly acquired surface data.

60. The system of claim 59, wherein the light source is a linear light source.

61. The system of claim 59, wherein the three-dimensional data representation is a point cloud or three-dimensional data convertible into a point cloud.

62. The system of claim 59, wherein the captured images are structured light patterned images that encode the surface of the physical object.

63. The system of claim 59, wherein the physical object is an anatomical structure in a surgical procedure.

64. The system of claim 59, wherein the three-dimensional data representation of the topological surface of the physical object is constructed by: decoding light patterns contained in the one or more captured images byVIS-007-W01 analyzing brightness history to create camera pixel correspondence to light angle projection; using triangulation to calculate a three-dimensional coordinate for each pixel in the one or more captured images based on the decoded light patterns; and combining the triangulated three-dimensional coordinates of each pixel into the three-dimensional point cloud representation of the surface of the physical object in physical space.

65. The system of claim 59, wherein the second dataset is derived from a preoperative computed tomography scan or magnetic resonance imaging scan.

66. The system of claim 59, wherein: the computer system further comprises a user interface and a display device, and the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display, on a display device, the tracked three-dimensional data representation of the physical object.

Images