Patella tracking methods and systems for surgical planning

Abstract

Systems and computer-implemented methods of improving a patello-femoral response. A computing system operably coupled to a markerless navigation system classifies the knee anatomy of the patient into one or more segments, including the patella. The computing system further receives first simulation data of movement of the segmented patella at a first surgical phase, such as a pre-operative phase. The computing system receives second simulation data of the segmented patella at a second surgical phase, such as an intraoperative stage using a first implant type. The computing system generates comparison data based on a comparison of the first and second simulation data. The computing system analyzes the comparison data, along with patient-specific information, via a machine learning model to provide one or more recommendations to improve a patello-femoral response based on reducing, minimizing, or eliminating the differences between the first simulation data and the second simulation data.

Description

Attorney Docket No.: 8178.6140WOPATELLA TRACKING METHODS AND SYSTEMS FOR SURGICAL PLANNINGRELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 716,839, filed November 6, 2024, entitled “PATELLA TRACKING METHODS AND SYSTEMS FOR SURGICAL PLANNING”, the contents of which are incorporated herein in their entireties.FIELD OF THE DISCLOSURE

[0002] The present disclosure generally relates to methods, systems, and apparatuses related to a computer-assisted surgical system that includes various hardware and software components that work together to enhance surgical workflows. More specifically, the present disclosure relates to methods, systems, and apparatuses for the automated determination of patient patella information during one or more surgical stages for developing at least a portion of a knee arthroplasty surgical plan.BACKGROUND

[0003] The ultimate goal of a knee replacement procedure, such as a total knee arthroplasty (TKA) procedure, is to restore knee function and alleviate pain by matching the size and orientation of implant components to optimally match and reproduce the patient's original, pre-surgical anatomy. However, the knee is a sophisticated joint formed of multiple components (i.e., femur, tibia, and patella operating via a specific group of muscles and tendons) that flexes, extends, rotates, and translates during movement and, as such, makes characterization and surgical reproduction of the movement difficult.

[0004] The use of computers, robotics, and imaging to aid knee replacement surgery has improved surgical outcomes for patients, for instance, through computer-aided navigation and robotic systems. For example, surgical navigation systems can aid surgeons in locating and tracking patient anatomical structures, guiding surgical instruments, and implantingAttorney Docket No.: 8178.6140WO medical devices with a high degree of accuracy. These systems allow surgeons to plan, track, and navigate the placement of instruments and implants relative to the body of a patient, as well as conduct pre-operative and intra-operative body imaging. Even with such advances, the biomechanics of a complex system such as the knee are difficult to accurately and completely characterize by intraoperative observation using existing systems and techniques.

[0005] In one particular example, surgical navigation systems are not able to adequately characterization the patella and patellar tracking to develop optimal surgical plans. However, optimizing patella tracking can reduce or eliminate post-surgical patellofemoral pain and other negative surgical outcomes. Determining optimal patellar tracking for each patient is challenging as it is affected by multiple factors, such as surgical technique, implant selection, and / or resurfaced and non-resurfaced patella characteristics. Typical methods used to measure patellar tracking include computed tomography, nuclear magnetic resonance imaging, tracking systems using physical marker devices, and fluorescence capture systems. However, abnormalities and variations in patellar tracking continue to negatively affect surgical outcomes because existing techniques are unable to consistently and accurately predict post-surgical patellar tracking performance.SUMMARY

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended as an aid in determining the scope of the claimed subject matter.

[0007] Disclosed herein are improved systems and methods for planning a knee arthroplasty procedure, including the use of image-based markerless surgical navigation systems configured to segment the patient knee into one or more segments (e.g., femur, tibia, and / or patella) and to provide the surgeon with simulation outputs of patello-femoral kinematics to compare the pre-operative knee with the intraoperative and / or post-operative knee.Attorney Docket No.: 8178.6140WO

[0008] In any preceding or subsequent example, an operating environment may include a system for arthroscopic surgery video segmentation operative to segment a surgical video, such as an arthroscopic surgical video, automatically using artificial intelligence (Al) or machine learning (ML) models.

[0009] In any preceding or subsequent example, the system may operate to segment or classify the patient knee into one or more anatomical segments. The anatomical segments may include the femur (or a portion thereof), the tibia (or a portion thereof), and / or the patella (or a portion thereof). In any preceding or subsequent example, the classification may include a multiple classification in which each pixel (or other image region) is classified as belonging to one of multiple structures, such as determining whether a pixel represents the femur, tibia, patella, or a non-bone structure. In any preceding or subsequent example, the classification may include a binary classification in which each pixel (or other image region) is classified as representing or not representing a subject structure (e.g., bone or non-bone; patella or non-patella; and / or the like). In any preceding or subsequent example, the classification may include segmenting one or more features of the patient anatomy, including points, regions, landmarks, and / or the like.

[0010] In any preceding or subsequent example, classified patella anatomical features may include a proximal or superior end, a distal or inferior end, a base, an apex, a medial side, a lateral side, an articular surface, a posterior ridge, a medial condyle facet, a lateral condyle facet, a centroid, dimensions (e.g., length, width, thickness), patella tilt angle, and / or the like

[0011] In any preceding or subsequent example, a classified anatomical feature may include medial-lateral (ML) motion of the patella mid-point (for instance, designated on an anterior surface) defined in the direction of the femur ML / flexion axis. In any preceding or subsequent example, a classified anatomical feature may include the combined AP / SI motion of the patella mid-point (for instance, designated on an anterior surface) relative to femur ML / flexion axis (for example, located by the epicondyles) to provide a point-to-axis distance measurement related to retinaculum tension.Attorney Docket No.: 8178.6140WO

[0012] In any preceding or subsequent example, a classified anatomical feature may include patella tilt. In some examples, the patella tilt may be defined by a relative change in point-to-axis distance and / or a distance of a point on medial and lateral sides of the patella relative to femur flexion axis.

[0013] In any preceding or subsequent example, a surgical method may include determining comparison results capturing differences in knee characteristics or performance of the knee between surgical states. In any preceding or subsequent example, the surgical states include pre-operative, intraoperative, trialing, and post-operative. In any preceding or subsequent example, a surgical plan may be developed or updated based on the comparison results to perform the arthroplasty surgery to reduce, minimize, or eliminate the performance or characteristic differences.

[0014] In any preceding or subsequent example, performance may include patella tracking performance.

[0015] In any preceding or subsequent example, the surgical method may classify the patient anatomy into anatomy segments. For example, the segmentation process may operate to determine or classify segments of the knee joint, such as the femur, tibia, patella, and / or portions thereof. In any preceding or subsequent example, the segmentation process may process raw video data to determine one or more knee segments. In any preceding or subsequent example, the segments may include or be annotated with landmarks, points, user-defined features, and / or the like. In any preceding or subsequent example, segmentation may determine features of a segment, such as points, facets, borders, sides, edges, and / or the like of the patella.

[0016] In any preceding or subsequent example, the surgical method may access or determine patient information. The patient information may include patient physical and / or demographic information. Non-limiting types of patient information may include gender, age, ethnicity, height, weight, femur shape, Q-angle, varus / valgus angle, patella tilt, patella ML location, patella SI location, tendon locations, ligament locations, and / or the like.

[0017] In any preceding or subsequent example, the surgical method may simulate patella tracking. For example, the knee segmentation information and the patient information mayAttorney Docket No.: 8178.6140WO be input into a simulation system to simulate knee performance. The simulation may operate to emulate knee movement based on the input information and to generate knee performance or kinematic data. Simulated knee movements may include bone and / or ligament inputs with keep knee bends, gait analysis, and / or the like. In some examples, the simulation may include or may be focused on patella movement and patella tracking.

[0018] In any preceding or subsequent example, the surgical method may determine simulation output. In any preceding or subsequent example, the simulation output may include patella simulation output information directed toward simulated patella tracking. In any preceding or subsequent example, the patella simulation output information may include, patella kinematics, ligament strains, tendon strains, bone strains. In any preceding or subsequent example, the patella simulation output information may include feature values at various knee states (e.g., gate states, knee bend or flexion / extension states, and / or the like), such as patella tilt, distance between specified points (e.g., on patella and femur), and / or the like.

[0019] In any preceding or subsequent example, the surgical method may repeat the segmentation and / or performance steps intraoperatively. In any preceding or subsequent example, an initial process of the surgical method may include performing segmentation and / or performance steps on the pre-operative knee to obtain a set of simulation outputs indicating the characteristics and performance of the patient pre-operative knee. During the surgery, the intraoperative knee may be processed via the segmentation and / or performance steps to generate a set of intraoperative simulation outputs. The segmentation and / or performance steps may be performed repeatedly for different intraoperative stages, including different implant configurations and / or knee preparation configurations.

[0020] In any preceding or subsequent example, the surgical method may compare preoperative and intraoperative and / or post-operative patella tracking information and provide surgical recommendations. For example, pre-operative knee performance data (e.g., patella kinematics, patella tilt, point distances, ligament / tendon strain, and / or the like) may be compared to intraoperative knee performance data to generate comparison information. The comparison information (e.g., comparison information 674) may indicate differencesAttorney Docket No.: 8178.6140WO in knee performance for the pre-operative knee and the intraoperative (or post-operative knee). For example, patella tracking information in the pre-operative knee may be different compared with the intraoperative (or post-operative) knee.

[0021] In any preceding or subsequent example, the surgical method may provide a recommendation based on the comparison information. For example, a recommendation for all or part of a surgical plan may be generated based on the comparison information, for instance, to reduce, minimize, or even eliminate differences between the pre-operative knee and the intraoperative (or post-operative) knee.

[0022] In any preceding or subsequent example, the comparison information may be input into an algorithm or computational model that uses simulation information and patient specific factors to provide the surgeon with the surgical recommendations.

[0023] In any preceding or subsequent example, the computational model may be trained to receive the comparison results along with relevant patient information as input and to generate a surgical plan to optimize knee performance. In any preceding or subsequent example, optimizing knee performance may include optimizing patella tracking performance. In any preceding or subsequent example, optimizing performance may include reducing, minimizing, or eliminating differences in patella tracking performance between surgical stages (e.g., pre-operative, intraoperative, and / or post-operative).

[0024] In any preceding or subsequent example, the computational model may be trained using performance data of physical patients and / or simulated performance of actual or virtual patients of pre-operative, intraoperative, and / or post-operative knee performance. In any preceding or subsequent example, the training data used to train the computational model may include information indicating differences between feature values of preoperative, intraoperative, and / or post-operative knee anatomy and changes to surgical plans that moved the differences toward the computational model objective (i.e., reducing, minimizing, or eliminating the differences).

[0025] Examples described in the present disclosure provide numerous advantages over conventional systems and methods. In one non-limiting example advantage, surgical systems and methods may be used that provides a computer navigated or robotic assistedAttorney Docket No.: 8178.6140WO surgery that allows pre-operative, intraoperative, and post-operative patella tracking using markerless navigation and generates in real-time for the surgeon based on simulation outputs of patello-femoral kinematics to compare native knee function before and postsurgery. Surgical systems and methods include using computer vision and AI / ML techniques to determine present and predict future knee positions without the need for physical markers. In another non-limiting example advantage, surgical systems and methods provide for AI / ML techniques for treating patients to achieve optimized surgical outcomes based on AI / ML modeling of knee performance comparisons between surgical stages. AI / ML models trained according to some examples are able to analyze the comparisons and determine optimal surgical plans for each individual patient based on training data of surgical results a population of knee arthroplasty patients.

[0026] Further features and advantages of at least some of the examples described in the present disclosure, as well as the structure and operation of any preceding or subsequent example of the present disclosure, are described in detail below with reference to the accompanying drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0027] By way of example, specific examples of the disclosed device will now be described, with reference to the accompanying drawings, in which:

[0028] FIG. 1 depicts an operating theatre including an illustrative computer-assisted surgical system (CASS) in accordance with one or more features of the present disclosure;

[0029] FIG. 2 depicts an operating environment for performing a surgery planning process in accordance with one or more features of the present disclosure;

[0030] FIG. 3 depicts a system for performing the surgery planning process in accordance with one or more features of the present disclosure;

[0031] FIG. 4 depicts processed video data in accordance with one or more features of the present disclosure;

[0032] FIGS. 5A-5E depict anatomical features of patient bone anatomy in accordance with one or more features of the present disclosure;Attorney Docket No.: 8178.6140WO

[0033] FIG. 6 illustrates an example of a first surgical workflow in accordance with one or more features of the present disclosure;

[0034] FIG. 7 illustrates an example of performance data in accordance with one or more features of the present disclosure;

[0035] FIG. 8 illustrates an example of a second surgical workflow in accordance with one or more features of the present disclosure;

[0036] FIG. 9A illustrates an example of a processing flow in accordance with one or more features of the present disclosure;

[0037] FIG. 9B illustrates an example of a processing flow in accordance with one or more features of the present disclosure; and

[0038] FIG. 10 illustrates an example of a computing architecture in accordance with the present disclosure.

[0039] It should be understood that the drawings are not necessarily to scale and that the disclosed examples are sometimes illustrated diagrammatically and in partial views. In certain instances, details which are not necessary for an understanding of the disclosed methods and devices, or which render other details difficult to perceive may have been omitted. It should be further understood that this disclosure is not limited to the particular examples illustrated herein. In the drawings, like numbers refer to like elements throughout unless otherwise noted.DETAILED DESCRIPTION

[0040] This disclosure is not limited to the particular systems, devices and methods described, as these may vary. The terminology used in the description is for the purpose of describing the particular versions or examples only and is not intended to limit the scope.

[0041] As used in this document, the singular forms “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise. Unless defined otherwise, all technical and scientific terms used herein have the same meanings as commonly understood by one of ordinary skill in the art. Nothing in this disclosure is to be construed as an admission that the examples described in this disclosure are not entitled to antedateAttorney Docket No.: 8178.6140WO such disclosure by virtue of prior invention. As used in this document, the term “comprising” means “including, but not limited to.”

[0042] For the purposes of the present disclosure, the term “implant” is used to refer to a prosthetic device or structure manufactured to replace or enhance a biological structure. For example, in a total hip replacement procedure, a prosthetic acetabular cup (implant) is used to replace or enhance a patient’s worn or damaged acetabulum. While the term “implant” is generally considered to denote a man-made or manufactured structure (as contrasted with a transplant), for the purposes of this specification an implant can include a biological tissue or material transplanted to replace or enhance a biological structure.

[0043] For the purposes of this disclosure, the term “real-time” is used to refer to calculations or operations performed on-the-fly as events occur or input is received by the operable system. However, the use of the term “real-time” is not intended to preclude operations that cause some latency between input and response, so long as the latency is an unintended consequence induced by the performance characteristics of the machine.

[0044] Although much of this disclosure refers to surgeons or other medical professionals by specific job title or role, nothing in this disclosure is intended to be limited to a specific job title or function. Surgeons or medical professionals can include any doctor, nurse, medical professional, or technician. Any of these terms or job titles can be used interchangeably with the user of the systems disclosed herein unless otherwise explicitly demarcated. For example, a reference to a surgeon also could apply, in some examples, to a technician or nurse.

[0045] The systems, methods, and devices disclosed herein may be used for surgical procedures that utilize surgical navigation systems, robotic surgical systems, computer- assisted surgical systems, and / or the like. Non-limiting examples of surgical navigation systems include the NAVIO® or CORI® surgical navigation system provided by BLUE BELT TECHNOLOGIES, INC. of Pittsburgh, PA, which is a subsidiary of SMITH & NEPHEW, INC. of Memphis, TN.

[0046] The described technology generally relates to surgical processes, for example, knee arthroplasty procedures including, without limitation, a total knee arthroplasty (TKA)Attorney Docket No.: 8178.6140WO procedure, a revision knee arthroplasty procedure, a partial (unicondylar or unicompartmental) knee arthroplasty procedure, and / or the like. Although knee arthroplasty, and TKA in particular, is used in some examples, the present disclosure is not limited to knee arthroplasty, as processes according to some examples may operate with other orthopedic procedures.

[0047] In some examples, a surgical process may include a computer navigated or robot- assisted surgical process that uses markerless navigation for pre-operative and intraoperative patella tracking. In general, marker-based tracking uses markers or fiducials (e.g., reflective spheres, disks and / or infrared (IR)-LEDs) installed in or on patient anatomy and corresponding devices (e.g., IR sensor) configured to track the markers. Markerless navigation involves systems capable of classifying and tracking patient anatomy (e.g., femur, tibia, patella) and / or specific features of patient anatomy (e.g., femoral condyles, patella apex, and / or the like) relying on images instead of markers. In various examples, markerless navigation may be implemented via a camera or computer-vision system capable of determining and tracking the movement of patient knee joint anatomy (i.e., femur, tibia, and / or patella). Processes according to some examples allow pre-operative, intraoperative, and / or post-operative patella tracking determinations and / or simulated predictions using markerless navigation to provide the surgeon with simulation outputs of patello-f emoral kinematics to compare the native (i.e., pre-operative) before and after the knee arthroplasty procedure.

[0048] Conventional computer-assisted orthopedic surgical systems use marker-based tracking, for example, involving markers or fiducials that are rigidly fixed relative to the patient bones to track the movement of the bones after an initial registration process is performed. Such surgical systems conventionally use markers that are affixed to the patient bones either percutaneously or directly to an exposed portion of the bones. Once affixed to the patient's bones, the markers can then be identified by a tracking system (e.g., an optical tracking system) to monitor the movement of the bones throughout the surgical procedure.

[0049] However, there are several issues with such bone-mounted markers in both endoscopic procedures and open procedures. For example, the pins used to mount theAttorney Docket No.: 8178.6140WO markers to the bone are relatively invasive and can cause iatrogenic injuries, including pinsite infections, fractures, and vascular and nerve injuries. The patella is especially susceptible to negative impacts from attaching or implanting markers. As another example, the markers are conventionally placed manually by the surgeon and suboptimal postoperative results have been observed due to poor positioning of the markers during the procedure. As yet another example, a significant amount of time and expertise is required in order to properly affix the markers to the patient bones and intraoperatively register the patient's anatomy with preoperative 3D anatomical models.

[0050] Accordingly, surgical systems and techniques according to examples described in the present disclosure use markerless navigation implemented via surface scanning technology in combination with artificial intelligence and machine learning techniques for segmenting bony anatomy and extracting and measuring features of interest. A handheld or robotically-controlled three-dimensional (3D) scanner based upon laser or structured light (RGB-D) can be used to capture the surface structure of the bony anatomy, while a continuous stream of data can be processed to track the 3D movement of the patella without the need for physical markers. Tn some examples, computer vision and artificial intelligence (AI) / machine learning (ML) techniques such as U-Net can be used to automatically and accurately delineate bony structures from structured-light scans. By capturing these 3D models in a continuous video stream, clinicians can track the patella's movement in real-time, e.g., during knee flexion. It can also capture all six degrees of freedom of patellar motion — including translation (side-to-side, up-and-down, in-and-out) and rotation (tilt, spin, and rotation). The ability to visualize the 3D tracking path of the patella can improve the diagnosis of patellofem oral joint pathologies. For example, the "J- sign," where the patella subluxes laterally during knee extension, can be visualized and quantified using a structured light system, which cannot be achieved with static imaging like MRI or CT scans. The flexion-extension range of motion of the pre-operative intact knee may be performed to collect patella tracking (e g., medial -lateral (ML) position, superior-inferior (SI) position, ML tilt about the patella SI axis, and / or patella flexion) relative to the center of the femur, trochlea groove of the femur, center of the tibia, and / orAttorney Docket No.: 8178.6140WO tibial tubercle. The patella and patellar-tracking measurements of native patient anatomy may be stored for comparison to the intraoperative knee and / or the implanted knee (or implanted knee predictions based on intraoperative information) and input into an algorithm, computational model, AI / ML model, and / or the like that collects data and patient specific factors to provide the surgeon with recommendations, surgical plans, and / or the like. Non-limiting examples of recommendations may include information regarding the knee joint degrees of freedom (e.g., six degrees of freedom) of femoral, tibial, and / or patella implant positioning, treatment of the patella if the patella is remaining intact, tissue releases to optimize patella kinematics, patella forces, quad efficiency, tendon and ligament strains, patella pressure, and / or the like.

[0051] In various examples, patella and patellar-tracking analysis may be repeated intra- operatively once the surgeon has made an implant selection to determine during trialing if patella kinematics are similar to previously-simulated patella kinematics. In the case they are different, surgical processes according to some examples may provide recommendations to keep patella tracking as close as possible to native knee kinematics or otherwise produce “optimal” patella tracking.

[0052] Accordingly, patient information relating to patella and patella-femoral articulations may be determined and used to form a surgical plan, including implant component selection and / or positioning, bone preparations, patella resurfacing, and / or the like.

[0053] Introducing the patellar-tracking analysis steps of various examples of the present disclosure into the surgical workflow, to more accurately capture and predict post-surgical knee anatomy during natural patient motion and predict clinical outcomes, is critical in providing a surgeon with a better understanding as to how an implant system can be optimized and to reduce the risk of poor outcomes.

[0054] Accordingly, various examples may include technologies and processes for determining patient information to optimize implant component selection and / or positioning to improve post-operative biomechanics and performance of the implant system.Attorney Docket No.: 8178.6140WO

[0055] Processes according to some examples may provide a technological feature and advantage of determining recommendations on selecting implant components, determining implant component positioning, and intraoperative testing of implant components (or trials) using markerless, computer-vision techniques in combination with AI / ML models to determine and / or predict post-surgical patellar-tracking outcomes. Processes according to some examples may also provide possible clinical benefits including, without limitation, reduced anterior knee pain and implant failure.

[0056] As a result, surgical processes according to any preceding or subsequent example may provide surgeons with improved surgical methods that are more accurate, personalized for each patient, and reduce complexity and cognitive load (particularly during an active surgery), while also improving patient outcomes.

[0057] FIG. 1 provides an illustration of an example computer-assisted surgical system (CASS) 100 according to any preceding or subsequent example. As described in further detail in the sections that follow, the CASS uses computers, robotics, and imaging technology to aid surgeons in performing orthopedic surgery procedures such as knee arthroplasty (e.g., total knee arthroplasty (TKA)) or total hip arthroplasty (THA). For example, surgical navigation systems can aid surgeons in locating patient anatomical structures, guiding surgical instruments, and implanting medical devices with a high degree of accuracy. Surgical navigation systems such as the CASS 100 often employ various forms of computing technology to perform a wide variety of standard and minimally invasive surgical procedures and techniques. Moreover, these systems allow surgeons to more accurately plan, track, and navigate the placement of instruments and implants relative to the body of a patient, as well as conduct pre-operative and intra-operative body imaging.

[0058] As shown in FIG. 1, an Effector Platform 105 positions surgical tools relative to a patient during surgery. The exact components of the Effector Platform 105 will vary, depending on the example employed. For example, for a knee surgery, the Effector Platform 105 may include an End Effector 105B that holds surgical tools or instruments during their use. The End Effector 105B may be a handheld device or instrument used byAttorney Docket No.: 8178.6140WO the surgeon (e.g., a hand piece or a cutting guide or jig of a surgical system, such as the Navio® Surgical System from Blue Belt Technologies of Plymouth, Minnesota, United States of America) or, alternatively, the End Effector 105B can include a device or instrument held or positioned by a Robotic Arm 105 A.

[0059] The Effector Platform 105 can include a Limb Positioner 105C for positioning the patient’s limbs during surgery. One example of a Limb Positioner 105C may be the SPIDER2 system manufactured and sold by Smith & Nephew, Inc. of Cordova, Tennessee, United States of America. The Limb Positioner 105C may be operated manually by the surgeon or alternatively change limb positions based on instructions received from the Surgical Computer 150 (described below).

[0060] Resection Equipment 110 (not shown in FIG. 1) performs bone or tissue resection using, for example, mechanical, ultrasonic, or laser techniques. Examples of Resection Equipment 110 may include drilling devices, burring devices, oscillatory sawing devices, vibratory impaction devices, reamers, ultrasonic bone cutting devices, radio frequency ablation devices, and laser ablation systems. In some examples, the Resection Equipment 110 is held and operated by the surgeon during surgery. In other examples, the Effector Platform 105 may be used to hold the Resection Equipment 110 during use.

[0061] The Effector Platform 105 can also include a cutting guide or jig 105D that is used to guide saws or drills used to resect tissue during surgery. Such cutting guides 105D can be formed integrally as part of the Effector Platform 105 or Robotic Arm 105 A, or cutting guides can be separate structures that can be matingly and / or removably attached to the Effector Platform 105 or Robotic Arm 105 A. The Effector Platform 105 or Robotic Arm 105 A can be controlled by the CASS 100 to position a cutting guide or jig 105D adjacent to the patient’ s anatomy in accordance with a pre-operatively or intraoperatively developed surgical plan such that the cutting guide or jig will produce a precise bone cut in accordance with the surgical plan.

[0062] The Tracking System 115 uses one or more sensors to collect real-time position data that locates the patient’s anatomy and surgical instruments. For example, for TKA procedures, the Tracking System may provide a location and orientation of the EndAttorney Docket No.: 8178.6140WOEffector 105B during the procedure. In addition to positional data, data from the Tracking System 115 can also be used to infer velocity / acceleration of anatomy / instrumentation, which can be used for tool control. In some examples, the Tracking System 115 may use a tracker array attached to the End Effector 105B to determine the location and orientation of the End Effector 105B. The position of the End Effector 105B may be inferred based on the position and orientation of the Tracking System 115 and a known relationship in three- dimensional space between the Tracking System 115 and the End Effector 105B. Various types of tracking systems may be used in some examples of the present disclosure including, without limitation, Infrared (IR) tracking systems, electromagnetic (EM) tracking systems, video or image-based tracking systems, and ultrasound registration and tracking systems.

[0063] Any suitable tracking system can be used for tracking surgical objects and patient anatomy in the surgical theatre. For example, a combination of infrared (IR) and visible light cameras can be used in an array. Various illumination sources, such as an IR light emitting diode (LED) light source, can illuminate the scene allowing three-dimensional imaging to occur. In some examples, this can include stereoscopic, tri-scopic, quad-scopic, etc. imaging. In addition to the camera array, which in some examples is affixed to a cart, additional cameras can be placed throughout the surgical theatre. For example, handheld tools or headsets worn by operators / surgeons can include imaging capability that communicates images back to a central processor to correlate those images with images captured by the camera array. This can give a more robust image of the environment for modeling using multiple perspectives. Furthermore, some imaging devices may be of suitable resolution or have a suitable perspective on the scene to pick up information stored in quick response (QR) codes or barcodes. This can be helpful in identifying specific objects not manually registered with the system.

[0064] In some examples, specific objects can be manually registered by a surgeon with the system preoperatively or intraoperatively. For example, by interacting with a user interface, a surgeon may identify the starting location for a tool or a bone structure. By tracking fiducial marks associated with that tool or bone structure, or by using otherAttorney Docket No.: 8178.6140WO conventional image tracking modalities, a processor may track that tool or bone as it moves through the environment in a three-dimensional model.

[0065] In some examples, certain markers, such as fiducial marks that identify individuals, important tools, or bones in the theater may include passive or active identifiers that can be picked up by a camera or camera array associated with the tracking system. For example, an IR LED can flash a pattern that conveys a unique identifier to the source of that pattern, providing a dynamic identification mark. Similarly, one- or two-dimensional optical codes (e g., barcode, QR code, etc.) can be affixed to objects in the theater to provide passive identification that can occur based on image analysis. If these codes are placed asymmetrically on an object, they can also be used to determine an orientation of an object by comparing the location of the identifier with the extents of an object in an image. For example, a QR code may be placed in a corner of a tool tray, allowing the orientation and identity of that tray to be tracked. Other tracking modalities are explained throughout. For example, augmented reality headsets can be worn by surgeons and other staff to provide additional camera angles and tracking capabilities.

[0066] In addition to optical tracking, certain features of objects can be tracked by registering physical properties of the object and associating them with objects that can be tracked, such as fiducial marks fixed to a tool or bone. For example, a surgeon may perform a manual registration process whereby a tracked tool and a tracked bone can be manipulated relative to one another. By impinging the tip of the tool against the surface of the bone, a three-dimensional surface can be mapped for that bone that is associated with a position and orientation relative to the frame of reference of that fiducial mark. By optically tracking the position and orientation (pose) of the fiducial mark associated with that bone, a model of that surface can be tracked with an environment through extrapolation.

[0067] The registration process that registers the CASS 100 to the relevant anatomy of the patient can also involve the use of anatomical landmarks, such as landmarks on a bone or cartilage. For example, the CASS 100 can include a 3D model of the relevant bone or joint and the surgeon can intraoperatively collect data regarding the location of bony landmarks on the patient’s actual bone using a probe that is connected to the CASS. Bony landmarksAttorney Docket No.: 8178.6140WO can include, for example, the medial malleolus and lateral malleolus, the ends of the proximal femur and distal tibia, and the center of the hip joint. The CASS 100 can compare and register the location data of bony landmarks collected by the surgeon with the probe with the location data of the same landmarks in the 3D model. Alternatively, the CASS 100 can construct a 3D model of the bone or joint without pre-operative image data by using location data of bony landmarks and the bone surface that are collected by the surgeon using a CASS probe or other means. The registration process can also include determining various axes of a joint. For example, for a TKA the surgeon can use the CASS 100 to determine the anatomical and mechanical axes of the femur and tibia. The surgeon and the CASS 100 can identify the center of the hip joint by moving the patient’s leg in a spiral direction (i.e., circumduction) so the CASS can determine where the center of the hip joint is located.

[0068] A Tissue Navigation System 120 (not shown in FIG. 1) provides the surgeon with intraoperative, real-time visualization for the patient’s bone, cartilage, muscle, nervous, and / or vascular tissues surrounding the surgical area. Examples of systems that may be employed for tissue navigation include fluorescent imaging systems and ultrasound systems.

[0069] The Display 125 provides graphical user interfaces (GUIs) that display images collected by the Tissue Navigation System 120 as well other information relevant to the surgery. For example, in some examples, the Display 125 overlays image information collected from various modalities (e.g., CT, MRI, X-ray, fluorescent, ultrasound, etc.) collected pre-operatively or intra-operatively to give the surgeon various views of the patient’s anatomy as well as real-time conditions. The Display 125 may include, for example, one or more computer monitors. As an alternative or supplement to the Display 125, one or more members of the surgical staff may wear an Augmented Reality (AR) Head Mounted Device (HMD). For example, in FIG. 1 the Surgeon I l l is wearing an AR HMD 155 that may, for example, overlay pre-operative image data on the patient or provide surgical planning suggestions. Various example uses of the AR HMD 155 in surgical procedures are detailed in the sections that follow.Attorney Docket No.: 8178.6140WO

[0070] Surgical Computer 150 provides control instructions to various components of the CASS 100, collects data from those components, and provides general processing for various data needed during surgery. In some examples, the Surgical Computer 150 is a general-purpose computer. In some examples, the Surgical Computer 150 may be a parallel computing platform that uses multiple central processing units (CPUs) or graphics processing units (GPU) to perform processing. In some examples, the Surgical Computer 150 is connected to a remote server over one or more computer networks (e.g., the Internet). The remote server can be used, for example, for storage of data or execution of computationally intensive processing tasks.

[0071] Various techniques generally known in the art can be used for connecting the Surgical Computer 150 to the other components of the CASS 100. Moreover, the computers can connect to the Surgical Computer 150 using a mix of technologies. For example, the End Effector 105B may connect to the Surgical Computer 150 over a wired (i.e., serial) connection. The Tracking System 115, Tissue Navigation System 120, and Display 125 can similarly be connected to the Surgical Computer 150 using wired connections. Alternatively, the Tracking System 115, Tissue Navigation System 120, and Display 125 may connect to the Surgical Computer 150 using wireless technologies such as, without limitation, Wi-Fi, Bluetooth, Near Field Communication (NFC), and / or ZigBee.

[0072] Part of the flexibility of the CASS design described above with respect to FIG. 1 is that additional or alternative devices can be added to the CASS 100 as necessary to support particular surgical procedures.

[0073] In some examples, the CASS 100 includes a robotic arm 105A that serves as an interface to stabilize and hold a variety of instruments used during the surgical procedure. The robotic arm 105 A may have multiple degrees of freedom (e.g., like a SPIDER2 device) and have the ability to be locked in place (e.g., by a press of a button, voice activation, a surgeon removing a hand from the robotic arm, or other method).

[0074] In some examples, movement of the robotic arm 105 A may be effectuated by use of a control panel built into the robotic arm system. For example, a display screen mayAttorney Docket No.: 8178.6140WO include one or more input sources, such as physical buttons or a user interface having one or more icons, that direct movement of the robotic arm 105 A. The surgeon or other healthcare professional may engage with the one or more input sources to position the robotic arm 105 A when performing a surgical procedure.

[0075] A tool or an end effector 105B attached or integrated into a robotic arm 105 A may include, without limitation, a burring device, a scalpel, a cutting device, a retractor, a joint tensioning device, or the like. In some examples in which an end effector 105B is used, the end effector may be positioned at the end of the robotic arm 105 A such that any motor control operations are performed within the robotic arm system. In some examples in which a tool is used, the tool may be secured at a distal end of the robotic arm 105 A, but motor control operation may reside within the tool itself.

[0076] The robotic arm 105 A may be motorized internally to both stabilize the robotic arm, thereby preventing it from falling and hitting the patient, surgical table, surgical staff, etc., and to allow the surgeon to move the robotic arm without having to fully support its weight. While the surgeon is moving the robotic arm 105 A, the robotic arm may provide some resistance to prevent the robotic arm from moving too fast or having too many degrees of freedom active at once. The position and the lock status of the robotic arm 105 A may be tracked, for example, by a controller or the Surgical Computer 150.

[0077] In some examples, the robotic arm 105A can be moved by hand (e.g., by the surgeon) or with internal motors into its ideal position and orientation for the task being performed. In some examples, the robotic arm 105 A may be enabled to operate in a “free” mode that allows the surgeon to position the arm into a desired position without being restricted. While in the free mode, the position and orientation of the robotic arm 105 A may still be tracked as described above. In some examples, certain degrees of freedom can be selectively released upon input from user (e.g., surgeon) during specified portions of the surgical plan tracked by the Surgical Computer 150. Designs in which a robotic arm 105 A is internally powered through hydraulics or motors or provides resistance to external manual motion through similar means can be described as powered robotic arms, whileAttorney Docket No.: 8178.6140WO arms that are manually manipulated without power feedback, but which may be manually or automatically locked in place, may be described as passive robotic arms.

[0078] A robotic arm 105 A or end effector 105B can include a trigger or other means to control the power of a saw or drill. Engagement of the trigger or other means by the surgeon can cause the robotic arm 105 A or end effector 105B to transition from a motorized alignment mode to a mode where the saw or drill is engaged and powered on. Additionally, the CASS 100 can include a foot pedal (not shown) that causes the system to perform certain functions when activated. For example, the surgeon can activate the foot pedal to instruct the CASS 100 to place the robotic arm 105 A or end effector 105B in an automatic mode that brings the robotic arm or end effector into the proper position with respect to the patient’s anatomy in order to perform the necessary resections. The CASS 100 can also place the robotic arm 105 A or end effector 105B in a collaborative mode that allows the surgeon to manually manipulate and position the robotic arm or end effector into a particular location. The collaborative mode can be configured to allow the surgeon to move the robotic arm 105 A or end effector 105B medially or laterally, while restricting movement in other directions. As discussed, the robotic arm 105 A or end effector 105B can include a cutting device (saw, drill, and burr) or a cutting guide or jig 105D that will guide a cutting device. In some examples, movement of the robotic arm 105 A or robotically controlled end effector 105B can be controlled entirely by the CASS 100 without any, or with only minimal, assistance or input from a surgeon or other medical professional. In still any preceding or subsequent example, the movement of the robotic arm 105 A or robotically controlled end effector 105B can be controlled remotely by a surgeon or other medical professional using a control mechanism separate from the robotic arm or robotically controlled end effector device, for example using a joystick or interactive monitor or display control device.

[0079] The examples below describe uses of the robotic device in the context of a hip surgery; however, it should be understood that the robotic arm may have other applications for surgical procedures involving knees, shoulders, etc.Attorney Docket No.: 8178.6140WO

[0080] A robotic arm 105 A may be used for holding the retractor. For example, in some examples, the robotic arm 105 A may be moved into the desired position by the surgeon. At that point, the robotic arm 105 A may lock into place. In some examples, the robotic arm 105 A is provided with data regarding the patient’s position, such that if the patient moves, the robotic arm can adjust the retractor position accordingly. In some examples, multiple robotic arms may be used, thereby allowing multiple retractors to be held or for more than one activity to be performed simultaneously (e.g., retractor holding & reaming).

[0081] The robotic arm 105 A may also be used to help stabilize the surgeon’s hand while making a femoral neck cut. In this application, control of the robotic arm 105A may impose certain restrictions to prevent soft tissue damage from occurring. For example, in some examples, the Surgical Computer 150 tracks the position of the robotic arm 105 A as it operates. If the tracked location approaches an area where tissue damage is predicted, a command may be sent to the robotic arm 105 A causing it to stop. Alternatively, where the robotic arm 105 A is automatically controlled by the Surgical Computer 150, the Surgical Computer may ensure that the robotic arm is not provided with any instructions that cause it to enter areas where soft tissue damage is likely to occur. The Surgical Computer 150 may impose certain restrictions on the surgeon to prevent the surgeon from reaming too far into the medial wall of the acetabulum or reaming at an incorrect angle or orientation.

[0082] The robotic arm 105 A may also be used for resurfacing applications. For example, the robotic arm 105 A may stabilize the surgeon while using traditional instrumentation and provide certain restrictions or limitations to allow for proper placement of implant components (e.g., guide wire placement, chamfer cutter, sleeve cutter, plan cutter, etc.). Where only a burr is employed, the robotic arm 105 A may stabilize the surgeon’s handpiece and may impose restrictions on the handpiece to prevent the surgeon from removing unintended bone in contravention of the surgical plan.

[0083] The various services that are provided by medical professionals to treat a clinical condition are collectively referred to as an “episode of care.” For a particular surgical intervention, the episode of care can include three phases: pre-operative, intra-operative, and post-operative. During each phase, data is collected or generated that can be used toAttorney Docket No.: 8178.6140WO analyze the episode of care in order to understand various features of the procedure and identify patterns that may be used, for example, in training models to make decisions with minimal human intervention. The data collected over the episode of care may be stored at the Surgical Computer 150 or the Surgical Data Server 180 as a complete dataset. Thus, for each episode of care, a dataset exists that includes all of the data collectively pre- operatively about the patient, all of the data collected or stored by the CASS 100 intra- operatively, and any post-operative data provided by the patient or by a healthcare professional monitoring the patient.

[0084] As explained in further detail, the data collected during the episode of care may be used to enhance performance of the surgical procedure or to provide a holistic understanding of the surgical procedure and the patient outcomes. For example, in some examples, the data collected over the episode of care may be used to generate a surgical plan. In some examples, a high-level, pre-operative plan is refined intra-operatively as data is collected during surgery. In this way, the surgical plan can be viewed as dynamically changing in real-time or near real-time as new data is collected by the components of the CASS 100. In some examples, pre-operative images or other input data may be used to develop a robust plan preoperatively that is simply executed during surgery. In this case, the data collected by the CASS 100 during surgery may be used to make recommendations that ensure that the surgeon stays within the pre-operative surgical plan. For example, if the surgeon is unsure how to achieve a certain prescribed cut or implant alignment, the Surgical Computer 150 can be queried for a recommendation. In still any preceding or subsequent example, the pre-operative and intra-operative planning approaches can be combined such that a robust pre-operative plan can be dynamically modified, as necessary or desired, during the surgical procedure. In some examples, a biomechanics-based model of patient anatomy contributes simulation data to be considered by the CASS 100 in developing preoperative, intraoperative, and post-operative / rehabilitation procedures to optimize implant performance outcomes for the patient.

[0085] Aside from changing the surgical procedure itself, the data gathered during the episode of care may be used as an input to other procedures ancillary to the surgery. ForAttorney Docket No.: 8178.6140WO example, in some examples, implants can be designed using episode of care data. Example data-driven techniques for designing, sizing, and fitting implants are described in U.S. Patent Application No. 13 / 814,531 filed August 15, 2011 and entitled “Systems and Methods for Optimizing Parameters for Orthopaedic Procedures”; U.S. Patent Application No. 14 / 232,958 filed July 20, 2012 and entitled “Systems and Methods for Optimizing Fit of an Implant to Anatomy”; U.S. Patent Application No. 12 / 234,444 filed September 19, 2008 and entitled “Operatively Tuning Implants for Increased Performance,” and U.S. Patent Application No. 17 / 265,675 filed February 3, 2021 and entitled “Patella Tracking Method and System” the entire contents of each of which are hereby incorporated by reference into the present disclosure.

[0086] Data acquired during the pre-operative phase generally includes all information collected or generated prior to the surgery. Thus, for example, information about the patient may be acquired from a patient intake form or electronic medical record (EMR). Examples of patient information that may be collected include, without limitation, patient demographics, diagnoses, medical histories, progress notes, vital signs, medical history information, allergies, and lab results. The pre-operative data may also include images related to the anatomical area of interest. These images may be captured, for example, using Magnetic Resonance Imaging (MRI), Computed Tomography (CT), X-ray, ultrasound, or any other modality known in the art. The pre-operative data may also include quality of life data captured from the patient. For example, in some examples, pre-surgery patients use a mobile application (“app”) to answer questionnaires regarding their current quality of life. In some examples, preoperative data used by the CASS 100 includes demographic, anthropometric, cultural, or other specific traits about a patient that can coincide with activity levels and specific patient activities to customize the surgical plan to the patient.

[0087] FIG. 2 is a block diagram depicting a system for performing a surgery planning process in accordance with one or more features of the present disclosure. FIG. 3 depicts a system for performing the surgery planning process in accordance with one or more features of the present disclosure. A non-limiting example of a surgical procedure may include an orthopedic or arthroplasty procedure.Attorney Docket No.: 8178.6140WO

[0088] Referring to FIGS. 2 and 3, an operating environment 200 may include a system 300 for surgery video segmentation operative to segment a surgical video, such as an arthroscopic surgical video, automatically using AI / ML models. In one example, the segmentation on the video occurs in real-time in-line with the display of the video in order to provide real-time, automatic segmentation during the surgical procedure. The system 300 includes an arthroscopic camera 210 that provides raw unprocessed arthroscopic video data 230. The arthroscopic video data 230 is received by either a central control unit (CCU) 220 and / or a data processing unit 225.

[0089] In some examples, the data processing unit 225 may be implemented in hardware, software, or a combination of hardware and software. In some examples, the data processing unit 225 may be a software application. In some examples, the data processing unit 225 may be or may include an AI / ML application. In some examples, the data processing unit 225 may be included within the CCU 220. In other examples, the data processing unit 225 may be a separate device or application to provide AI / ML processing for segmenting the video in-line with the processing of the arthroscopic video data 230. In various examples, the data processing unit 225 or portions thereof may be implemented in the arthroscopic camera 210 (e.g., an AI / ML processing unit of the arthroscopic camera 210).

[0090] In some examples, the arthroscopic camera 210 and the CCU 220 can be part of the same device that generates video feeds of an arthroscopic procedure. Using the system 300, an operator can acquire the raw unprocessed arthroscopic video data 230 for an anatomical region of interest 280 of the patient during an arthroscopic surgical procedure, such as a patient's joint, although the system 300 can be used for other anatomical regions of interest. The joint can be, for example, a knee, hip, shoulder, or any other joint or structure.

[0091] In one example, the CCU 220 and / or the data processing unit 225 are communicably coupled to the surgical computer 150 of the CASS shown in FIG. 1, such as via the network 175. In an alternate example, the CCU 220 and / or the data processing unit 225 may operate as the surgical computer 150 of the CASS shown in FIG. 1. The CCU 220 and / or the data processing unit 225 can also be coupled to one or more cloud and / orAttorney Docket No.: 8178.6140WO local network server or storage devices 250 via one or more communication network(s) 252. The cloud and / or local network server or storage devices 250 can provide storage of the processed video 240 along with generated patella information, although the processed video could be stored locally on the data processing unit 225. The CCU 220 and / or cloud and / or local network server and / or storage on the data processing unit 225 itself or storage devices 250 may also serve as repositories containing data that enable AI / ML models stored in the data processing unit 225, for example, to segment the video and generate patella information, as described in further detail below.

[0092] The arthroscopic camera 210 may include a camera configured to provide images and / or a video feed of an arthroscopic procedure. The camera 210 may be configured to provide fully markerless navigation of at least a portion of the patient anatomy whereby the registration, planning, and bone removal stage are carried out using depth images.

[0093] In some examples, the camera 210 may be or may include imaging sensors configured to provide depth images or video, RGB images or video, RGB-D images or video, 2D images or video, 3D images or video, and / or IR images or video (including multidimensional IR images or video). Non-limiting examples of the camera 210 may include a SPRYTRACK camera (e.g., SPRYTRACK 300) provided by Atracsys LLC, the LENS™ system provided by Smith and Nephew, Inc., the Azure Kinect DK developer kit from Microsoft Corporation, the Acusense camera from Revopoint® 3D, the Traxis tracking camera and Al system provided by Zero Density, and / or the like.

[0094] In some examples, the camera 210 can be a single camera that has infrared and RGB-D (color and depth) capturing functionality. In such an example, the camera 210 can be handheld. Alternatively, the camera 210 can be attached to a robotic arm, such as robotic arm 105A. In some examples, the camera 210 is used to capture the raw video data of the region of interest, such as the patient’s knee, and this raw video and point cloud data is processed as described herein. In some examples, the camera 210 can be a revel point camera that is markerless. Since structured light captures only the skin surface, the patella's shape must be extracted from the surface geometry, a segmentation task that can be complicated by soft tissue. The clean point cloud is converted into a 3D surface mesh. ThisAttorney Docket No.: 8178.6140WO mesh, typically an STL file, is a digital, geometric representation of the patella's shape, composed of interconnected triangles. To create a coherent statistical model, all patella meshes from the training population are aligned to a common coordinate system. An iterative closest point (ICP) algorithm can be used during data alignment and registration to minimize the positional and rotational differences between each sample and a chosen template patella. Once the meshes are aligned, a script can be used to establish nodal correspondence, ensuring each point on one mesh corresponds to a point on all other meshes. Principal component analysis (PCA) is then performed on the coordinates of the corresponding points, which is a mathematical technique that identifies the principal modes of variation that account for most of the shape differences across the population. The output is a statistical shape model consisting of a mean patella shape and a set of variance modes. The first few modes typically explain the largest variations, such as overall size, while later modes describe more subtle differences. By comparing a patient's patellar shape to a statistical shape model (SSM), clinicians can identify and quantify shape abnormalities like trochlear dysplasia, which is a major factor in patellar instability. These SSMs can also be used in pre-operative simulation software to study joint kinematics and contact mechanics, helping clinicians understand how shape variations influence joint function and disease progression, such as osteoarthritis. They can also be used to help select the optimum implant fit for a patient by testing different implant shapes against a range of normal variations using finite element analysis.

[0095] In some examples, multiple cameras are used at the same time to capture both marker and markerless-based data can be captured simultaneously and aligned to a global world coordinate frame. For example, a stereo Infrared (IR) camera can be used to track the position of reflective markers attached to the femur and tibia whilst the structured light camera can be used to track and register the movement of the patella in the same coordinate reference frame. This dual-camera approach ensures that the entire knee joint is visible for markerless tracking, while markers are captured by a dedicated IR camera. Specialized hardware and software, often using timecodes, ensure that the two cameras in the system capture frames at precisely the same moment. An optical array can be attached to the 3DAttorney Docket No.: 8178.6140WO scanner to allow it to be visible to the IR stereo camera at the same time as the optical markers attached to the femur and tibia in the same working volume.

[0096] The extrinsic and intrinsic parameters of the two cameras are calibrated to establish their positions and orientations within a single, shared global coordinate frame. This is done by recording a calibration object with a known shape that can be pixilated from all camera viewpoints. This data can be used to calculate the relative positions of each camera / The calibration tool with a defined shape can be used to capture point cloud acquired with a tracked CORI robotic probe, which can be co-registered to a set of point clouds obtained from the 3D camera in the same scene. This calibration tool is visible both during the acquisition with the probe and the 3D camera, providing a known reference geometry that you can be used to register the two modalities.

[0097] During a data capture session, the IR cameras track the 2D position of the reflective markers in each frame. The system's software then uses the known camera positions to triangulate and reconstruct the 3D position of each marker in the global coordinate frame. Simultaneously, the video cameras record footage. Al-driven computer vision algorithms, such as OpenPose, analyze the video feeds to estimate the position of anatomical landmarks (e.g., joints) in 2D. The 2D landmark data from the markerless system is converted to 3D and aligned with the global coordinate frame established during calibration. This is often done using a Direct Linear Transformation (DLT) algorithm. The system fuses the markerbased and markerless data streams, which are now all in the same coordinate frame. The highly accurate marker data can be used to improve the precision of the markerless data, especially in areas where markerless tracking might be less reliable.

[0098] There are a number of benefits from establishing a hybrid tracking system. Firstly, marker data acts as a "ground truth" to validate and refine the less precise markerless estimates, leading to higher overall accuracy. Secondly, the system is more resilient to issues that affect a single method. If a marker is temporarily occluded, the markerless data can fill in the gaps. If the markerless Al struggles with a specific pose, the marker data can provide reliable tracking. Thirdly, combining both methods in a single capture session eliminates the need for separate markerless and marker-based sessions and complicatedAttorney Docket No.: 8178.6140WO post-capture alignment. Fourthly, it facilitates the tracking of both the tibiofemoral and patellofem oral joints within the same environment.

[0099] Table 1 below provides a summary of different examples for capturing patella imaging.Atorney Docket No.: 8178.6140WOTABLE 1

[0100] In some examples, the camera 210 (or the CCU 220 operably coupled to the camera 210) may include embedded hardware and / or software to allow for fast processing of captured image data. For example, the camera 210 (or the CCU 220 operably coupled to the camera 210) may include an embedded graphics processing unit (GPU). A non-limiting example of a GPU is the Nvidia® Jetson GPU processor. Embedding a GPU into the camera 210 (or the CCU 220 operably coupled to the camera 210) allows for the simultaneous or substantially simultaneous capturing and processing of image data.

[0101] In various examples, the camera 210 may include one or more sensors or cameras configured to detect and track markers or fiducials (e.g., reflective spheres, disks and / or infrared (IR)-LEDs) to implement marker-based tracking. For example, the camera 210 may include an IR tracking sensor.

[0102] In one example, the camera 210 includes an RGB-D imaging sensor operative to provide multiple output signals. In some examples, the output signals can include three output signals including 2D video data, 3D depth data, and / or infrared (IR) stereo data. The signals may be processed to determine and track patient anatomy. For example, the video data 230 may be processed to classify the femur portion of the knee, the tibial portion of the knee, and / or the patella. In another example, the video data 230 may be processed to determine features of the bones of the knee, such as the apex of the patella, the centroid or midpoint of the patella, the femoral condyles, the femoral flexion axis, and / or the like. In various examples, movement of the features of the anatomy classified from the video data 230 may be tracked, including relative to other features, for example, through a range of motion.

[0103] A non-limiting example of systems and methods for determining and / or tracking patient anatomy using images or video includes WIPO International Patent Application Publication No. WO2024044188, titled “Multi-Class Image Segmentation with W-Net Architecture,” the entire contents of each of which is incorporated by reference in the present disclosure. Another non-limiting example of systems and methods for determining and / or tracking patient anatomy using images or video includes U.S. Patent ApplicationAttorney Docket No.: 8178.6140WOPublication No. 20230360400, titled “Methods for Arthroscopic Surgery Video Segmentation and Devices Therefor,” the entire contents of each of which is incorporated by reference in the present disclosure. An additional non-limiting example of systems and methods for determining and / or tracking patient anatomy using images or video includes U.S. Patent Application Publication No. 2024 / 0013537, titled “Methods for Arthroscopic Video Analysis and Devices Therefor,” the entire contents of each of which is incorporated by reference in the present disclosure.

[0104] In one example, the video data 230 can be processed to register point clouds to a model (e.g., based on patient data and / or an atlas model). The registered point clouds can be used for registering other objects, tracking, and / or modeling. In some examples, the video data 230 is processed using an AI / ML algorithm or model to produce a classification of each pixel. In some examples, the classification may include a multiple classification in which each pixel is classified as belonging to one of multiple structures, such as determining whether a pixel represents the femur, tibia, patella, or a non-bone structure. In various examples, the classification may include a binary classification in which each pixel is classified as representing or not representing a subject structure (e.g., bone or non-bone; patella or non-patella; and / or the like).

[0105] In one example, an AI / ML model can be configured for end-to-end intra-operative image segmentation during robot-assisted orthopedic surgery. A training set can include labeled images of anatomy (e.g., RGB-D images of cadaveric knees). The AI / ML model may be or may include a deep learning model, a neural network (NN), and / or the like.

[0106] The AI / ML model can be configured to perform image-based or imageless registration or classification. The image-based registration can include acquiring a preoperative model of the patient anatomy based on imaging. Intraoperatively, the imagebased registration can include acquiring RGB-D frames with an image sensor of the camera 210, segmenting the images, identifying corresponding point clouds on a 3D point cloud, and registering the point clouds to the pre-operative model. Imageless registration can include acquiring RGB-D frames with an image sensor of the camera 210. The frames can be captured at high frame rates when compared to other systems (e.g., > 25 Hz). ImagelessAttorney Docket No.: 8178.6140WO registration can include segmenting the RGB images, for example, via a multi-class AI / ML approach. Imageless registration identifies corresponding point clouds on a 3D point cloud in a similar manner to the image-based registration. The 3D point clouds can be fed into an atlas model to obtain a 3D model. The atlas model can be modified based on intraoperative imaging to more closely mirror the patient anatomy.

[0107] To acquire the arthroscopic video data 230 using the system 300, an operator locates the arthroscopic camera 210 to provide the anatomical region of interest in the field of view of the arthroscopic camera 210. In another example, the arthroscopic camera 210 could be positioned using the computer-assisted surgical system disclosed herein. The arthroscopic video data 230 can include a video feed that extends for at least a portion of the arthroscopic surgical procedure. In this example, the arthroscopic video data 230 is a real-time stream as opposed to a static container, such as an image or video file. The arthroscopic camera 210 can be relocated during the procedure, either by the operator, or by computer assistance, to obtain arthroscopic video data 230 for different fields of view.

[0108] During the arthroscopic surgical procedure, the CCU 220 and / or the data processing unit 225 receives the raw unprocessed video data 230, generates the processed surgical video data 240 including the identified segments, and stores and / or transmits the processed surgical video data 240. In one example, the CCU 220 provides pre-processing of the arthroscopic raw unprocessed video data 230 prior to analysis by the data processing unit 225, although pre-processing can also be performed by the data processing unit 225 in another example. In some examples, digital markers or digital annotations inserted manually by the surgeon during the surgical procedure can also be stored along with the raw video 230 in a specialized file and / or metadata associated with the processed video surgical data 240. The CCU 220 or data processing unit 225 may store the processed video data 240 for post-operative video processing, or transmit the processed video data 240 to other devices, such as the network server or storage devices 250. In other examples, the CCU 220 or data processing unit 225 provides a real-time segmented video playback to the display device 260.Attorney Docket No.: 8178.6140WO

[0109] FIG. 4 depicts processed video data in accordance with one or more features of the present disclosure. As shown in FIG. 4, the processing of raw video data 230 to generate processed video data 240 may include multiple steps. For example, in a first step 450, coarse detection of subject patient anatomy may be performed. For example, a knee region 404 bounding the knee joint 402 of the patient may be detected. In a second step 451, precision segmentation or classification may be performed using AI / ML processes described according to various examples. During segmentation, the subject patient anatomy (i.e., the knee joint 402) may be segmented into segments of interest, such as the tibia 410 (or the proximal end of the tibia) and the femur 412 (or the distal end of the femur). Other segments may be detected, such as the patella (not shown) and non-bone anatomy, such as tendons, ligaments, muscle, and / or the like. In some examples, the segmentation may include a binary segmentation in which only one portion of the subject anatomy is segmented from the remaining anatomy. For instance, in a binary patella segmentation, the knee joint 402 may be segmented into patella and non-patella image regions, pixels, and / or the like.

[0110] In some examples, a third step 452 may include a validation step performed by an operator to confirm or modify the automated segmentation. For example, a surgeon may select to confirm that the region designated via the segmentation as the distal end of the femur is correct. In another example, the surgeon may indicate that a certain portion designated as part of the proximal end of the tibia is actually a portion of the patella. The system 300 may modify the segmentation to conform to the operator validation.[OHl] In various examples, segmentation may include determining one or more features, points, regions, landmarks, and / or the like of the segmented anatomy. FIGS. 5A-5E depict anatomical features of patient bone anatomy in accordance with one or more features of the present disclosure. For example, the anatomical features of the patella 500 depicted in FIGS. 5A-5D may be automatically determined according to segmentation processes of examples of the present disclosure. Non-limiting examples of patella anatomical features may include a proximal or superior end, a distal or inferior end, a base, an apex, a medial side, a lateral side, an articular surface, a posterior ridge, a medial condyle facet, a lateralAttorney Docket No.: 8178.6140WO condyle facet, a centroid, dimensions (e.g., length, width, thickness), patella tilt angle, and / or the like.

[0112] In another example, referring to FIG. 5E, anatomical features of the femur 580 and / or tibia 590 may be determined via the segmentation process. For example, a flexion axis 550 of the femur 580 may be determined (e.g., extending a line between a medial condyle arc center 587 and a lateral condyle arc center 588). In another example, the medial and lateral condyles 585, 586 may be determined. In another example, a tibial mechanical axis 591 of the tibia 590 may be determined.

[0113] Examples are not limited in this context as other anatomical features of the patella, femur, tibia, and / or the knee joint may be determined via segmentation processes according to some examples. In one example, the surgeon may create an anatomical feature, for instance, via selecting or defining a point or other portion of the patient anatomy. For example, the surgeon may define a mid-point or centroid of the patella and the segmentation process may automatically designate this portion of the patella in future processed video data 240. In some examples, the anatomical features may be defined in correlation with other features.

[0114] For example, a first feature may be defined as a distance between feature A and feature B. In this example, an operator may define a first point on a first bone structure (for instance, by selecting a point, landmark, or other region on the patella) and a second point on a second bone structure (for instance, the flexion axis of the femur). The selected points may be defined as part of the segmentation of the knee joint. Characteristics of interest of the points, such as a distance between the points, and angle between the points, and / or the like may be determined during segmentation and for knee performance data collection (see, for example, FIGS. 6-8).

[0115] In some examples, point or feature selection criteria may be determined to minimize sensitivity to artifacts from tissue movement relative to bone to reduce need to clear tissue off anterior side of patella or other regions of the knee joint.

[0116] One non-limiting example of an anatomical feature may include ML motion of the patella mid-point (for instance, designated on an anterior surface) defined in the directionAttorney Docket No.: 8178.6140WO of the femur ML / flexion axis. Another non-limiting example of an anatomical feature may include the combined AP / SI motion of the patella mid-point (for instance, designated on an anterior surface) relative to femur ML / flexion axis (for example, located by the epicondyles) to provide a point-to-axis distance measurement related to retinaculum tension.

[0117] A further non-limiting example of an anatomical feature may include patella tilt. In some examples, the patella tilt may be defined by a relative change in point-to-axis distance and / or a distance of a point on medial and lateral sides of the patella relative to femur flexion axis. In various examples, points may be defined a certain distance from the patella mid-point. For example, the points may be defined in the direction of femur flexion axis or based on femur flexion axis direction while in extension and maintained, if possible, during patella motion. In another example, differences in a relative change of a medial point compared to a lateral point using spacing between the points may be used to calculate the patella tilt angle.

[0118] In some examples, the anatomical features may be associated with the posterior articular surface of the patella. In various examples, the anatomical features may be associated with the posterior articular surface of the patella through direct visualization (e.g., inversion of the patella and segmentation according to various examples) or correlation.

[0119] Included herein are one or more workflows representative of exemplary methodologies for performing novel features of the disclosed examples. While, for purposes of simplicity of explanation, the one or more methodologies shown herein are shown and described as a series of acts, those skilled in the art will understand and appreciate that the methodologies are not limited by the order of acts. Some acts may, in accordance therewith, occur in a different order and / or concurrently with other acts from that shown and described herein. For example, those skilled in the art will understand and appreciate that a methodology could alternatively be represented as a series of interrelated states or events, such as in a state diagram. Moreover, not all acts illustrated in aAttorney Docket No.: 8178.6140WO methodology may be required for a novel implementation. Blocks designated with dotted lines may be optional blocks of a workflow.

[0120] A workflow or portions (or steps) thereof may be implemented manually and / or via software, firmware, hardware, or any combination thereof. In software and firmware examples, one or more workflow steps (or logic flow) may be implemented by computer executable instructions stored on a non-transitory computer readable medium or machine- readable medium (for instance, executed by CASS 100, CCU 220, data processing unit 225, and / or similar systems). The examples are not limited in this context.

[0121] FIG. 6 illustrates an example of a first surgical workflow in accordance with one or more features of the present disclosure. Workflow 600 may be representative of some or all of the operations executed by or according to any preceding or subsequent example described in the present disclosure to determine a surgical plan.

[0122] The workflow 600 may include a data collection or generation phase 601. Performance data 702a-n of patella tracking during one or more stages 670a-n may be generated for the tracked anatomical features classified via the segmenting process according to some examples. The stages 670a-n may include various surgical stages, such as pre-operative, intraoperative, trialing, post-operative, and / or the like. In some examples, the performance data 702a-n may include physical data based on data collected via physical manipulation of the patient anatomy. In various examples, the performance data 702a-n may include simulated data generated via simulation of knee movement of the segmented knee during a particular stage 670a-n. A non-limiting example of a simulation system may include the LifeMOD™ simulation software by Smith and Nephew, Inc.

[0123] For example, during the data generation stage 601, pre-operative 670a simulation of flexion / extension of a patient knee may generate performance data 702a. The simulation may include flexion / extension through various degrees of flexion / extension (i.e., 0 degrees of flexion through 90 degrees of flexion), deep knee bends, gait, and / or the like. The performance data 702a-n may include kinematic information relating to the pattern of motion, for example, six degrees of freedom for one or more anatomical structures and / or features. In some examples, the flexion-extension range of motion of the intact knee mayAttorney Docket No.: 8178.6140WO be simulated to collect patella tracking (e.g., medial-lateral position, superior-inferior position, medial-lateral tilt about the patella superior-inferior axis, and / or patella flexion) relative to the center of the femur, trochlea groove of the femur, center of the tibia, and / or tibial tubercle.

[0124] FIG. 7 illustrates an example of performance data in accordance with one or more features of the present disclosure. As shown in FIG. 7, a performance data set 702a for a pre-operative phase 670a may be collected or generated for various knee states, such as flexion angles 704a-n. In some examples, the performance data 702a may be generated via simulation of a knee having the features of the segmented knee (including any determined points, landmarks, or other features and any relationships therebetween) and patient characteristics (e.g., height, weight, gender, age, and / or the like). For example, state 704a may be for a flexion angle of 0 degrees, state 704b may be for a flexion angle of 10 degrees, and so on until state 704n. Accordingly, in some examples, a set of data or “snapshot” of the knee joint (or portions thereof, such as the patella or patella tracking state) may be generated and stored.

[0125] For each state, a set of feature data 710a relating to patella tracking may be collected for each subject feature 712a. For example, the feature 712a may be a patella tilt angle of a patella when the knee is in flexion angle A 704a, which is determined to have a particular feature value 714a (e.g., 2 degrees). At flexion angle N 704n, the Feature A 712a may have a value of feature value 714n (e.g., 10 degrees). In the alternative or in addition, kinematic data 720 (e.g., joint angles, translations, joint performance, and / or the like) may be determined via physical manipulation and / or simulation of the knee for performance data 702a (e g., the pre-operative knee).

[0126] Non-limiting examples of features 712a-n may include boundary detection such as patella shape, size and contours, and morphological measurements such as articular facets (size and shape of the medial and lateral articular facets on the posterior surface can be analyzed for irregularities), trochlear groove morphology, e.g. sulcus angle, shape and depth of the femoral trochlear groove, degree of medial and lateral patellar tilt and rotation, varus and valgus angle, and distance between points (e.g., a first point on the patella andAttorney Docket No.: 8178.6140WO a second point on the femur), patella superior-inferior (SI) location, patella medio-lateral (ML) location, ligament strains, tendon strains, bone strains, and / or the like.

[0127] Accordingly, for each surgical stage, a set of performance data of the knee may be collected or generated for one or more knee states. For example, performance data 702a is for the pre-operative knee, performance data 702b may include a set of data for an intraoperative knee according to a first surgical plan (e.g., implant type, implant size, resection information, patella resurfacing, and / or the like), performance data 702c may include a set of data for an intraoperative knee according to a second surgical plan, performance data 702n may include a set of data for a post-operative knee according to a the first surgical plan, and so on.

[0128] Referring back to FIG. 6, during a comparison phase 602, performance data 702a- n from two or more phases are compared to generate comparison results 674. For example, the performance data 702a set for the pre-operative knee may be compared to the performance data set 702b for an intraoperative knee according to a first surgical plan. In another example, the performance data 702a set for the pre-operative knee may be compared to the performance data set 702n for the post-operative knee according to the first surgical plan.

[0129] In some examples, the comparison results 674 may indicate differences between the performance data 702a-n of the different surgical phases (e.g., pre-operative, intraoperative, post-operative). For example, the comparison results 676 may indicate that the patella tilt angle of the pre-operative knee is 1 degree smaller than the patella tilt angle of the post-surgical knee. In another example, the comparison results 674 may indicate that a distance between a first point on the patella and a second point on the femur is 5 millimeters smaller in the pre-operative knee compared with the post-surgical knee. The comparison results 674 may be formed of different types of values, such as positive / negative integers (e.g., subtracting a first value from a second value), percentages (percentage different between two different values), and / or the like.

[0130] During an analysis phase, the comparison results 674 along with relevant patient information 676 may be used as input to a trained computational model 680 (see, forAttorney Docket No.: 8178.6140WO example, FIGS. 9A and 9B) to generate a surgical plan 690, which may include updating an existing surgical plan. In some examples, the patient information 676 may include patient anatomical information, including measurements, features, and / or the like of the patient knee anatomy (e.g., femur shape, patella shape, patella tilt angle, Q-angle, varus / valgus angle, ML / SI location of patella bone and tendons, and / or the like). In various examples, the patient information 676 may include physical and / or demographic information of the patient (e.g., height, weight, BMI, gender, ethnicity, and / or the like).

[0131] The computational model 680 may be trained to receive the comparison results 674 and the patient information 676 as input and to generate a surgical plan 690 (or to update an existing surgical plan). A goal of an arthroplasty surgery is to install the implant while providing the patient with the same or substantially similar knee characteristics and performance. For example, in an optimal surgery, the patella tilt of the pre-operative or pre-arthritic patella throughout the range flexion / extension would be the same or substantially the same as the patella tilt of the post-operative patella. Accordingly, in some examples, the objective or objective function of the computational model 680 is to reduce, minimize, or even eliminate differences between the pre-operative feature values and the post-operative feature values. The computational model 680 may operate to determine a surgical plan that reduces, minimizes, or even eliminates differences between the preoperative feature values and the post-operative feature values.

[0132] In some examples, the objective or objective function of the computational model 680 is to achieve optimal knee performance, which may include optimal patella tracking. In various examples, optimal knee performance may include achieving the same or substantially similar performance as the pre-operative or pre-arthritic knee. In other examples, optimal knee performance may include achieving pre-defined performance goals, which may be the same or different than the pre-operative knee (e.g., an improved varus / valgus angle, an improved patella tilt, and / or the like).

[0133] The computational model 680 may be trained using performance data of physical patients (or cadavers) and / or simulated performance of actual or virtual patients of preoperative and post-operative knee performance. The training data used to train theAttorney Docket No.: 8178.6140WO computational model 680 may include information indicating differences between feature values of pre-operative, intraoperative, and / or post-operative knee anatomy and changes to surgical plans that moved the differences toward the computational model objective (i.e., reducing, minimizing, or eliminating the differences).

[0134] In some examples, the computational model 680 may be trained or configured posttraining to provide more weight to certain feature values over other feature values. For example, the computational model 680 may operate to provide more weight to maintaining patella tilt versus a varus angle. In another example, the computational model 680 may be trained to operate according to certain difference thresholds. For instance, the patella tilt at 0 degrees flexion may be 5% different between the pre-operative and post-operative knee without requiring a change to the surgical plan. In another instance, a threshold may be configured that the varus angle cannot change more than 10% between the pre-operative and post-operative knee.

[0135] In some examples, historical data sets from one or more databases may be used as inputs to the computational model 680. The computational model 680 may be or may include one or more types of deep learning, Al, ML, and / or the like models. Non-limiting examples of computational model 680 may include a neural network (NN), a recurrent NNs (RNN), a convolutional NN (CNN), an artificial NN (ANN), a large language model (LLM), combinations thereof, variations thereof, and / or the like. The computational model 680 may be trained using various ML training techniques, including supervised, semisupervised, unsupervised, and reinforcement training.

[0136] The computational model 680 is trained to predict one or more values based on the input data. In some examples, the computational model 680 is configured to determine the optimal size, position, and orientation of the implants to achieve an optimal surgical outcome.

[0137] Changes to the surgical plan 690 recommended by the computational model 680 may include, without limitation, changes to implant type, implant size(s), tibial resection properties, femoral resection properties, patella resurfacing, and / or the like.Attorney Docket No.: 8178.6140WO

[0138] The phases 601-603 of the workflow 600 may be repeated and / or performed in a different order. For example, performance data 702a and 702b may be generated for states 670a and 670b during phase 601. After determining the comparison results 674 during phase 602, the surgeon or operator may return to phase 601 to determine different performance data 702a-n for different knee states before proceeding to phase 603. Examples are not limited in this context.

[0139] FIG. 8 illustrates an example of a second surgical workflow in accordance with one or more features of the present disclosure. Workflow 800 may be representative of some or all of the operations executed by or according to any preceding or subsequent example described in the present disclosure to determine a surgical plan or recommendations.

[0140] At block 802, the workflow 800 may determine anatomy segments. For example, the segmentation process may operate to determine or classify segments of the knee joint, such as the femur, tibia, patella, and / or portions thereof. In some examples, the segmentation process may process raw video data to determine one or more knee segments (which may include or be annotated with landmarks, points, user-defined features, and / or the like). In some examples, the segmentation process may operate to only determine one segment, such as the patella. In various examples, segmentation may determine features of a segment, such as points, facets, borders, sides, edges, and / or the like of the patella. For instance, the segmentation process may determine a patella segment with a mid-point feature. In another instance, the segmentation process may produce segments including the patella bone anatomy, femur bone anatomy, tibia bone anatomy, ligament attachment points, and / or tendon attachment points. The segmentation process may be performed by a markerless navigation system according to the present disclosure (see, for example, FIGS. 2 and 3).

[0141] Workflow 800 may determine patient information at block 804. The patient information may include patient physical and / or demographic information. Non-limiting types of patient information may include gender, age, ethnicity, height, weight, femur shape, Q-angle, varus / valgus angle, patella tilt, patella ML location, patella SI location, tendon locations, ligament locations, and / or the like. In general, the patient informationAttorney Docket No.: 8178.6140WO may include any information that may be used to determine or update a surgical plan according to various examples.

[0142] At block 806, the workflow 800 may simulate patella tracking. For example, the knee segmentation information and the patient information may be input into a simulation system to simulate knee performance. Non-limiting examples of simulation systems may include computational models, AI / ML models, simulation algorithms, simulation equations, simulation software platforms, and / or the like. One example of a simulation software platform is the LifeMOD™ simulation software by Smith and Nephew, Inc. The simulation may operate to emulate knee movement based on the input information and to generate knee performance or kinematic data. Simulated knee movements may include bone and / or ligament inputs with deep knee bends, gait analysis, and / or the like. In some examples, the simulation may include or may be focused on patella movement and patella tracking.

[0143] The workflow 800 may determine simulation output at block 808. For example, the simulation process of block 806 may generate output including patella simulation output information directed toward simulated patella tracking. In various examples, the simulation output may be the same or similar to performance data 702a-n. In some examples, the patella simulation output information may include, without limitation, patella kinematics, ligament strains, tendon strains, bone strains. In various examples, the patella simulation output information may include feature values at various knee states (e.g., gate states, knee bend or flexion / extension states, and / or the like), such as patella tilt, distance between specified points (e.g., on patella and femur), and / or the like.

[0144] At block 810, the workflow 800 may repeat one or more of blocks 802-808 intraoperatively. In some examples, an initial process of the workflow 800 may include performing blocks 802-808 on the pre-operative knee to obtain a set of simulation outputs indicating the characteristics and performance of the patient preoperative knee. During the surgery, the intraoperative knee may be processed via blocks 802-808 to generate a set of intraoperative simulation outputs. Blocks 802-808 may be performed repeatedly for different intraoperative stages, including different implant configurations (e.g., for a firstAttorney Docket No.: 8178.6140WO selected implant trial, for a second selected input trial, and so on) and / or knee preparation configurations (e.g., different femoral preparations, patella resurfacing, and / or the like). In this manner, processes according to some examples are able to determine simulation outputs, including patella tracking simulation information, for the pre-operative knee, one or more intraoperative knee configurations, and / or the post-operative knee.

[0145] For instance, the method of blocks 802-808 may be repeated intraoperatively once the surgeon has made the implant selection to determine during trialing if patella kinematics are similar (or otherwise optimal) to previous simulated patella kinematics (or pre-operative patella kinematics). In the case they are different (or otherwise sub-optimal), the computer method would allow for recommendations to keep patella tracking as in native knee or to produce “optimal” patella tracking results.

[0146] The workflow 800 may compare pre-operative and intraoperative and / or postoperative patella tracking information and provide recommendations at block 812. For example, pre-operative knee performance data (e.g., patella kinematics, patella tilt, point distances, ligament / tendon strain, and / or the like) may be compared to intraoperative knee performance data to generate comparison information. The comparison information (e.g., comparison information 674) may indicate differences in knee performance for the preoperative knee and the intraoperative (or post-operative knee). For example, patella tracking information in the pre-operative knee may be different compared with the intraoperative (or post-operative) knee.

[0147] For example, the patella tilt angle at 30 degrees of flexion in the pre-operative knee may be 2 degrees smaller compared with the intraoperative knee at the same degree of flexion. Another example of a patella tracking information may include ML motion of the patella mid-point defined in the direction of the femur ML / fl exion axis. A further example of patella tracking information may include the combined AP / SI motion of the patella midpoint relative to femur ML / flexion axis, for example, to provide a point-to-axis distance measurement related to retinaculum tension. An additional example of patella tracking information may include patella tilt. In some examples, the patella tilt may be defined by a relative change in point-to-axis distance and / or a distance of a point on medial and lateralAttorney Docket No.: 8178.6140WO sides of the patella relative to femur flexion axis. In various examples, points may be defined a certain distance from the patella mid-point. For example, the points may be defined in the direction of femur flexion axis or based on femur flexion axis direction while in extension and maintained if possible, during patella motion. In another example, differences in a relative change of a medial point compared to a lateral point using spacing between the points may be used to calculate the patella tilt angle.

[0148] The workflow 800 at block 812 may provide a recommendation based on the comparison information. For example, a recommendation for all or part of a surgical plan may be generated based on the comparison information, for instance, to reduce, minimize, or even eliminate differences between the pre-operative knee and the intraoperative (or post-operative) knee. In one example, the difference between a pre-operative and intraoperative point-to-axis distance measurement related to retinaculum tension may be X using a femoral implant of size Y. The recommendation may be to use a femoral implant of size Z to reduce the point-to-axis distance measurement value X (including to zero).

[0149] In some examples, the comparison information may be input into an algorithm or computerized model that uses simulation information and patient specific factors to provide the surgeon with recommendations, for instance, six degree of freedom of femoral, tibial, and patella implant positioning or other treatment of the patella if the patella is remaining intact, tissue releases to optimize patella kinematics, patella forces, quad efficiency, tendon and ligament strains, patella pressure, and / or the like.

[0150] In various examples, the comparison information may be input into an AI / ML model (see, for example, FIGS. 6, 9A, and 9B) to determine a surgical plan and / or recommendations to achieve optimal knee performance (e.g., optimal patella tracking). In some examples, optimal knee performance may include achieving the same or substantially similar performance as the pre-operative knee. In other examples, optimal knee performance may include achieving a pre-defined optimal range of knee performance, which may be the same or different than the knee performance of the pre-operative knee (e g., an improved varus / valgus angle, an improved patella tilt, and / or the like).Attorney Docket No.: 8178.6140WO

[0151] FIG. 9A illustrates an example of a processing flow in accordance with examples described in the present disclosure. More specifically, FIG. 9A illustrates an example of a processing flow for training a computational model 950. In some examples, computational model 950 may be or may include computational model 680.

[0152] As shown in FIG. 9A, a training module 930 may be configured to access training data 910. In some examples, the training data 910 includes real-world clinical data 912 associated with arthroscopic surgery patients. In various examples, the training data 910 includes simulated data 914 associated with arthroscopic surgery patients (including actual and / or virtual patients). In some examples, the training data 910 may include historical data sets from one or more databases.

[0153] In some examples, the training data 910 may include knee performance data, comparison data, and surgical plans associated with known patient outcomes (real or simulated). The training data 910 may be configured to provide real -world and simulated outcomes of patients with specific knee performance data (e.g., kinematics and / or feature values (e.g., patella tilt, point distances, and / or the like)) and comparison data with one or more surgical plans.

[0154] For example, the training data 910 may include data for patients and associated patient information sets (e.g., personal information, knee performance data, and comparison results for multiple surgical stages (e.g., a pre-operative stage, one or more intraoperative stages, and / or a post-operative stage). The training data 910 may include surgical plans or changes to surgical plans (e.g., changes to implant type, implant size, bone preparation / resection, patella resurfacing, and / or the like) and the effect of the changes on the comparison information. For instance, for a patient with information set X and a preoperative and intraoperative change in patella tilt of +2 degrees, changing the femoral implant from type Y to type Z eliminated the patella tilt difference. In another instance, a patient with information set X may have a sub-optimal intraoperative ML motion of the patella mid-point defined in the direction of the femur ML / flexion (i.e., outside of an acceptable range) based on a first surgical plan. Modifying the first surgical plan with different femoral resurfacing parameters according to a second surgical plan may cause theAttorney Docket No.: 8178.6140WO intraoperative ML motion of the patella mid-point defined in the direction of the femur ML / flexion to be within an optimal range. Examples are not limited in this context.

[0155] In some examples, the training data 910 may include labeled data, unlabeled data, semi-labeled data, structured data, unstructured data, semi -structured data, combinations thereof, variations thereof, and / or the like.

[0156] The training data 910 is fed into the training module 930 configured to train computational model 950. The computational model 950 may include various types of data models, mathematical models, regression models, algorithms, predictor equations, deep learning models, Al models, ML models, NNs, CNNs, RNNs, ANNs, LLMs, combinations thereof, variations thereof, and / or the like. The computational model 950 may be trained to determine output of a surgical plan (or surgical recommendation) based on input of patient information and comparison results.

[0157] In some examples, the computational model 950 may be trained or configured posttraining to provide more weight to certain performance goals (e.g., feature values over other performance goals). For example, the computational model 950 may operate to provide more weight to maintaining patella tilt versus a varus angle. In another example, the computational model performance goals may be trained to operate according to certain difference thresholds. For instance, the patella tilt at 0 degrees flexion may be 5% different between pre-operative and post-operative knee without requiring a change to the surgical plan. In another instance, the surgical plan is only changed if the varus angle difference is more than 10% between the pre-operative and intraoperative knee. Examples are not limited in this context.

[0158] FIG. 9B illustrates an example of a processing flow in accordance with one or more features of the present disclosure. More specifically, FIG. 9B illustrates an example of a processing flow for determining a surgical plan based on comparison results determined according to some examples. As shown in FIG. 9B, patient information 960 and comparison information 962 may be fed into the trained computational model 950. The patient information 960 may include patient information (e.g., demographics, age, gender, health condition(s), and / or the like) and patient knee or knee performance information (e.g.,Attorney Docket No.: 8178.6140WO varus / valgus angle, Q-angle, point distances, patella tilt, patella tracking information) at one or more surgical stages (e.g., pre-operative, intraoperative, post-operative). The comparison information 962 (e.g., comparison results 674) may indicate differences between the performance data of the different surgical phases (e.g., pre-operative, intraoperative, post-operative).

[0159] The computational model 950 may operate to generate a surgical plan 970 configured to provide an optimal outcome for one or more knee functions. For instance, in some examples, the computational model 950 may be trained for optimizing patella tracking functions. In various examples, the surgical plan 970 may include information associated with performing a knee arthroplasty procedure, in particular, that provides optimal outcomes. In some examples, an optimal outcome may be or may include postoperative knee performance that is the same or substantially similar to pre-operative knee performance. In various examples, an optimal outcome may be or may include postoperative knee performance that meets certain metrics or goals (e.g., a varus / valgus angle within a specified range, a patella tilt angle within a specified range, lack of patient knee pain, and / or the like).

[0160] In some examples, a healthcare provider 980 may access the surgical plan 970 for use in treating a patient 985. For example, the surgical plan 970 may be provided to a surgeon intraoperatively to allow the surgeon to proceed with an arthroplasty procedure according to the surgical plan 970 (for instance, updated to reduce, minimize, or eliminate certain knee performance outcomes indicated by the comparison information 962). The surgical plan 970 may be provided to the surgeon intraoperatively via a display device of the CASS in the operating theatre.

[0161] FIG. 10 illustrates an example of an exemplary computing architecture 1000 suitable for implementing various examples as previously described. In various examples, the computing architecture 1000 may comprise or be implemented as part of an electronic device. In some examples, the computing architecture 1000 may be representative, for example, of computing device 1002 and / or components thereof. The examples are not limited in this context.Attorney Docket No.: 8178.6140WO

[0162] As used in this application, the terms “system” and “component” and “module” are intended to refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution, examples of which are provided by the exemplary computing architecture 1000. For example, a component can be, but is not limited to being, a process running on a processor, a processor, a hard disk drive, multiple storage drives (of optical and / or magnetic storage medium), an object, an executable, a thread of execution, a program, and / or a computer. By way of illustration, both an application running on a server and the server can be a component. One or more components can reside within a process and / or thread of execution, and a component can be localized on one computer and / or distributed between two or more computers. Further, components may be communicatively coupled to each other by various types of communications media to coordinate operations. The coordination may involve the unidirectional or bi-directional exchange of information. For instance, the components may communicate information in the form of signals communicated over the communications media. The information can be implemented as signals allocated to various signal lines. In such allocations, each message is a signal. Further examples, however, may alternatively employ data messages. Such data messages may be sent across various connections. Exemplary connections include parallel interfaces, serial interfaces, and bus interfaces.

[0163] The computing architecture 1000 includes various common computing elements, such as one or more processors, multi-core processors, co-processors, memory units, chipsets, controllers, peripherals, interfaces, oscillators, timing devices, video cards, audio cards, multimedia input / output (I / O) components, power supplies, and so forth. The examples, however, are not limited to implementation by the computing architecture 1000.

[0164] As shown in FIG. 10, the computing architecture 1000 comprises a processing unit 1004, a system memory 1006 and a system bus 1008. The processing unit 1004 can be any of various commercially available processors, including without limitation an AMD® Athlon®, Duron® and Opteron® processors; ARM® application, embedded and secure processors; IBM® and Motorola® DragonBall® and PowerPC® processors; IBM and Sony® Cell processors; Intel® Celeron®, Core (2) Duo®, Itanium®, Pentium®, Xeon®,Attorney Docket No.: 8178.6140WO and XScale® processors; and similar processors. Dual microprocessors, multi-core processors, and other multi-processor architectures may also be employed as the processing unit 1004.

[0165] The system bus 1008 provides an interface for system components including, but not limited to, the system memory 1006 to the processing unit 1004. The system bus 1008 can be any of several types of bus structure that may further interconnect to a memory bus (with or without a memory controller), a peripheral bus, and a local bus using any of a variety of commercially available bus architectures. Interface adapters may connect to the system bus 1008 via a slot architecture. Example slot architectures may include without limitation Accelerated Graphics Port (AGP), Card Bus, (Extended) Industry Standard Architecture ((E)ISA), Micro Channel Architecture (MCA), NuBus, Peripheral Component Interconnect (Extended) (PCI(X)), PCI Express, Personal Computer Memory Card International Association (PCMCIA), and the like.

[0166] The system memory 1006 may include various types of computer-readable storage media in the form of one or more higher speed memory units, such as read-only memory (ROM), random-access memory (RAM), dynamic RAM (DRAM), Double-Data-Rate DRAM (DDRAM), synchronous DRAM (SDRAM), static RAM (SRAM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, polymer memory such as ferroelectric polymer memory, ovonic memory, phase change or ferroelectric memory, silicon-oxide- nitride-oxide-silicon (SONOS) memory, magnetic or optical cards, an array of devices such as Redundant Array of Independent Disks (RAID) drives, solid state memory devices (e.g., USB memory, solid state drives (SSD) and any other type of storage media suitable for storing information. In the illustrated example shown in FIG. 10, the system memory 1006 can include non-volatile memory 1010 and / or volatile memory 1012. A basic input / output system (BIOS) can be stored in the non-volatile memory 1010.

[0167] The computer 1002 may include various types of computer-readable storage media in the form of one or more lower speed memory units, including an internal (or external) hard disk drive (HDD) 1014, a magnetic floppy disk drive (FDD) 1016 to read from orAttorney Docket No.: 8178.6140WO write to a removable magnetic disk 1018, and an optical disk drive 1020 to read from or write to a removable optical disk 1022 (e g., a CD-ROM or DVD). The HDD 1014, FDD 1016 and optical disk drive 1020 can be connected to the system bus 1008 by an HDD interface 1024, an FDD interface 1026 and an optical drive interface 1029, respectively. The HDD interface 1024 for external drive implementations can include at least one or both of Universal Serial Bus (USB) and IEEE 1384 interface technologies.

[0168] The drives and associated computer-readable media provide volatile and / or nonvolatile storage of data, data structures, computer-executable instructions, and so forth. For example, a number of program modules can be stored in the drives and memory units 1010, 1012, including an operating system 1030, one or more application programs 1032, other program modules 1034, and program data 1036. In one example, the one or more application programs 1032, other program modules 1034, and program data 1036 can include, for example, the various applications and / or components of computing device 110.

[0169] A user can enter commands and information into the computer 1002 through one or more wire / wireless input devices, for example, a keyboard 1038 and a pointing device, such as a mouse 1040. Other input devices may include microphones, infra-red (IR) remote controls, radio-frequency (RF) remote controls, game pads, stylus pens, card readers, dongles, finger print readers, gloves, graphics tablets, joysticks, keyboards, retina readers, touch screens (e.g., capacitive, resistive, etc.), trackballs, trackpads, sensors, styluses, and the like. These and other input devices are often connected to the processing unit 1004 through an input device interface 1042 that is coupled to the system bus 1008, but can be connected by other interfaces such as a parallel port, IEEE 994 serial port, a game port, a USB port, an IR interface, and so forth.

[0170] A monitor 1044 or other type of display device is also connected to the system bus 1008 via an interface, such as a video adaptor 1046. The monitor 1044 may be internal or external to the computer 1002. In addition to the monitor 1044, a computer typically includes other peripheral output devices, such as speakers, printers, and so forth.

[0171] The computer 1002 may operate in a networked environment using logical connections via wire and / or wireless communications to one or more remote computers,Attorney Docket No.: 8178.6140WO such as a remote computer 1049. The remote computer 1049 can be a workstation, a server computer, a router, a personal computer, portable computer, microprocessor-based entertainment appliance, a peer device or other common network node, and typically includes many or all of the elements described relative to the computer 1002, although, for purposes of brevity, only a memory / storage device 1050 is illustrated. The logical connections depicted include wire / wireless connectivity to a local area network (LAN) 1052 and / or larger networks, for example, a wide area network (WAN) 1054. Such LAN and WAN networking environments are commonplace in offices and companies, and facilitate enterprise-wide computer networks, such as intranets, all of which may connect to a global communications network, for example, the Internet.

[0172] When used in a LAN networking environment, the computer 1002 is connected to the LAN 1052 through a wire and / or wireless communication network interface or adaptor 1056. The adaptor 1056 can facilitate wire and / or wireless communications to the LAN 1052, which may also include a wireless access point disposed thereon for communicating with the wireless functionality of the adaptor 1056.

[0173] When used in a WAN networking environment, the computer 1002 can include a modem 1058, or is connected to a communications server on the WAN 1054, or has other means for establishing communications over the WAN 1054, such as by way of the Internet. The modem 1059, which can be internal or external and a wire and / or wireless device, connects to the system bus 1008 via the input device interface 1042. In a networked environment, program modules depicted relative to the computer 1002, or portions thereof, can be stored in the remote memory / storage device 1050. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers can be used.

[0174] The computer 1002 is operable to communicate with wire and wireless devices or entities using the IEEE 802 family of standards, such as wireless devices operatively disposed in wireless communication (e g., IEEE 802.16 over-the-air modulation techniques). This includes at least Wi-Fi (or Wireless Fidelity), WiMax, and Bluetooth™ wireless technologies, among others. Thus, the communication can be a predefinedAttorney Docket No.: 8178.6140WO structure as with a conventional network or simply an ad hoc communication between at least two devices. Wi-Fi networks use radio technologies called IEEE 802.1 lx (a, b, g, n, etc.) to provide secure, reliable, fast wireless connectivity. A Wi-Fi network can be used to connect computers to each other, to the Internet, and to wire networks (which use IEEE 802.3-related media and functions).

[0175] Several implementation examples of the present disclosure are provided hereafter. Namely, in one example, a method is disclosed, the method including accessing surface image data of a knee joint of a patient; processing the surface image data to generate a set of knee segments; accessing patient information associated with the patient, the patient information including physical or demographic information; using a simulation system to generate a simulation of the knee joint, the simulation generating knee performance data, wherein the simulation is generated based on the set of knee segments and the patient information; determining, from the simulation of the knee joint and knee performance data, simulated patella tracking of the knee joint; processing intra-operative surface image data of the knee joint to generate a set of intra-operative knee segments; generating, based on the intra-operative surface image data and the set of intra-operative knee segments, an intra-operative representation of knee performance of the knee joint; determining, from the intra-operative representation of the knee performance, intra-operative patella tracking of the knee joint; comparing the simulated patella tracking of the knee joint to the intraoperative patella tracking of the knee joint to determine a deviation therebetween; generating, based on the deviation, a recommendation to alter a surgical plan for the knee joint.

[0176] In some examples, the surface image data comprises image data for a tissue surface of the knee joint. In some examples, the intra-operative surface image data comprises image data for an implant surface of the knee joint. In some examples, wherein the intraoperative surface image data comprises image data for a trial surface or instrument surface.

[0177] In some examples, processing the surface image data includes producing a three- dimensional (3D) reconstruction of the knee joint from point clouds based upon calculated 3D coordinates from the surface image data. In some examples, the surface image dataAttorney Docket No.: 8178.6140WO includes raw video data of the knee joint of the patient. In some examples, generating the set of knee segments includes processing the raw video data of the knee joint to classify segments of the knee joint as a femur, a tibia, a patella, or portions thereof; identifying landmarks on the femur, tibia, patella, or portions thereof; and using the landmarks to determine features of a corresponding segment of the knee joint, the features including points, facets, borders, sides, or edges of the corresponding segment. In some examples, the simulation of the knee joint and the knee performance data are based on the determined features of the corresponding segment.

[0178] In some examples, the patient information includes patient physical or demographic information. In some examples, the patient physical or demographic information includes at least one selected from gender, age, ethnicity, height, weight, femur shape, Q-angle, varus / valgus angle, patella tilt, patella medio-lateral (ML) location, patella superiorinferior (SI) location, tendon locations, or ligament locations.

[0179] In some examples, the simulation of the knee joint includes simulating knee activities that include bone or ligament inputs with knee bends, gait analysis, or patella movement or tracking. In some examples, the simulation is configured to provide knee performance or kinematic data.

[0180] In some examples, the simulation system includes a computational model, an artificial intelligence (Al) or machine learning (ML) model, a simulation algorithm, one or more simulation equations, or one or more simulation software platforms. In some examples, the intra-operative surface image data includes raw video data of the knee joint of the patient during a knee operation. In some examples, generating the set of intraoperative knee segments includes: processing the raw video data of the intra-operative knee joint to classify segments of the intra-operative knee joint as a femur, a tibia, a patella, or portions thereof; identifying landmarks on the femur, tibia, patella, or portions thereof; using the landmarks to determine features of a corresponding intra-operative segment of the intra-operative knee joint, the features including points, facets, borders, sides, or edges of the corresponding intra-operative segment. In some examples, the intra-operativeAttorney Docket No.: 8178.6140WO representation of the knee performance of the knee joint is based on the determined features of the corresponding intra-operative segment.

[0181] In some examples, the intra-operative representation of the knee performance is configured to provide knee performance or kinematic data of the intra-operative knee joint. In some examples, the simulated patella tracking of the knee joint includes simulated patella output information including simulated patella kinematic data, simulated ligament strain data, simulated tendon strain data, or simulated bone strain data. In some examples, the intra-operative patella tracking of the knee joint includes intra-operative patella output information including intra-operative patella kinematic data, intra-operative ligament strain data, intra-operative tendon strain data, or intra-operative bone strain data.

[0182] In some examples, comparing the simulated patella tracking of the knee joint to the intra-operative patella tracking of the knee joint includes: comparing the patella output information to the intra-operative patella output information to determine a deviation between: the simulated patella kinematic data and the intra-operative patella kinematic data; the simulated ligament strain data and the intra-operative ligament strain data; the simulated tendon strain data and the intra-operative tendon strain data; or the simulated bone strain data and the intra-operative bone strain data.

[0183] In a second example implementation, a computing device is provided, the computing device comprising a processing circuit and a memory coupled to the processing circuit, the memory including executable instructions, which when executed causes the processing circuit to perform various operations. In some examples, the processing circuit is caused to access surface image data of a knee joint of a patient. In some examples, the processing circuit is caused to process the surface image data to generate a set of knee segments. In some examples, the processing circuit is caused to access patient information associated with the patient, the patient information including physical or demographic information. In some examples, the processing circuit is caused to use a simulation system to generate a simulation of the knee joint, the simulation generating knee performance data, wherein the simulation is generated based on the set of knee segments and the patient information. In some examples, the processing circuit is caused to determine, from theAttorney Docket No.: 8178.6140WO simulation of the knee performance, simulated patella tracking of the knee joint. In some examples, the processing circuit is caused to process intra-operative surface image data of the knee joint to generate a set of intra-operative knee segments. In some examples, the processing circuit is caused to generate, based on the intra-operative surface image data and the set of intra-operative knee segments, an intra-operative representation of knee performance of the knee joint. In some examples, the processing circuit is caused to determine, from the intra-operative representation of the knee performance, intra-operative patella tracking of the knee joint. In some examples, the processing circuit is caused to compare the simulated patella tracking of the knee joint to the intra-operative patella tracking of the knee joint to determine a deviation therebetween. In some examples, the processing circuit is caused to generate, based on the deviation, a recommendation to alter a surgical plan for the knee joint.

[0184] In some examples, the surface image data comprises image data for a tissue surface of the knee joint. In some examples, the processing circuit is caused to the intra-operative surface image data comprises image data for an implant surface of the knee joint. In some examples, the intra-operative surface image data comprises image data for a trial surface or instrument surface. In some examples, processing the surface image data includes the processing circuit being caused to produce a three-dimensional (3D) reconstruction of the knee joint from point clouds based upon calculated 3D coordinates from the surface image data.

[0185] In some examples, the surface image data includes raw video data of the knee joint of the patient. In some examples, generating the set of knee segments includes the processing circuit being caused to: process the raw video data of the knee joint to classify segments of the knee joint as a femur, a tibia, a patella, or portions thereof; identify landmarks on the femur, tibia, patella, or portions thereof; and use the landmarks to determine features of a corresponding segment of the knee joint, the features including points, facets, borders, sides, or edges of the corresponding segment. In some examples, the simulation of the knee joint and the knee performance data is based on the determined features of the corresponding segment.Attorney Docket No.: 8178.6140WO

[0186] In some examples, the patient information includes patient physical or demographic information. In some examples, the patient physical or demographic information includes at least one selected from gender, age, ethnicity, height, weight, femur shape, Q-angle, varus / valgus angle, patella tilt, patella medio-lateral (ML) location, patella superiorinferior (SI) location, tendon locations, or ligament locations.

[0187] In some examples, the simulation of the knee joint includes the processing circuit being caused to simulate knee activities that include bone or ligament inputs with knee bends, gait analysis, or patella movement or tracking. In some examples, the simulation is configured to provide knee performance or kinematic data. In some examples, the simulation system includes a computational model, an artificial intelligence (Al) or machine learning (ML) model, a simulation algorithm, one or more simulation equations, or one or more simulation software platforms.

[0188] In some examples, the surface intra-operative image data includes raw video data of the knee joint of the patient during a knee operation. In some examples, generating the set of intra-operative knee segments includes the processing circuit being caused to: process the raw video data of the intra-operative knee joint to classify segments of the intraoperative knee joint as a femur, a tibia, a patella, or portions thereof; identify landmarks on the femur, tibia, patella, or portions thereof; and use the landmarks to determine features of a corresponding intra-operative segment of the intra-operative knee joint, the features including points, facets, borders, sides, or edges of the corresponding intra-operative segment. In some examples, intra-operative representation of the knee performance of the knee joint is based on the determined features of the corresponding intra-operative segment.

[0189] In some examples, the intra-operative representation of the knee performance is configured to provide knee performance or kinematic data of the intra-operative knee joint. In some examples, the simulated patella tracking of the knee joint includes simulated patella output information including simulated patella kinematic data, simulated ligament strain data, simulated tendon strain data, or simulated bone strain data. In some examples, the intra-operative patella tracking of the knee joint includes intra-operative patella outputAttorney Docket No.: 8178.6140WO information including intra-operative patella kinematic data, intra-operative ligament strain data, intra-operative tendon strain data, or intra-operative bone strain data.

[0190] In some examples, comparing the simulated patella tracking of the knee joint to the intra-operative patella tracking of the knee joint includes the processing circuit being caused to: compare the simulated patella output information to the intra-operative patella output information to determine a deviation between: the simulated patella kinematic data and the intra-operative patella kinematic data; the simulated ligament strain data and the intra-operative ligament strain data; the simulated tendon strain data and the intra-operative tendon strain data; or the simulated bone strain data and the intra-operative bone strain data.

[0191] The foregoing description has broad application. While the present disclosure refers to certain some examples, numerous modifications, alterations, and changes to the described examples are possible without departing from the sphere and scope of the present disclosure, as defined in the appended claim(s). Accordingly, it is intended that the present disclosure not be limited to the described examples. Rather these examples should be considered as illustrative and not restrictive in character. All changes and modifications that come within the spirit of the disclosure are to be considered within the scope of the disclosure. The present disclosure should be given the full scope defined by the language of the following claims, and equivalents thereof. The discussion of any example is meant only to be explanatory and is not intended to suggest that the scope of the disclosure, including the claims, is limited to these examples. In other words, while illustrative examples of the disclosure have been described in detail herein, it is to be understood that the inventive concepts may be otherwise variously embodied and employed, and that the appended claims are intended to be construed to include such variations, except as limited by the prior art. Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the disclosure belongs.

[0192] Directional terms such as top, bottom, superior, inferior, medial, lateral, anterior, posterior, proximal, distal, upper, lower, upward, downward, left, right, longitudinal, front,Attorney Docket No.: 8178.6140WO back, above, below, vertical, horizontal, radial, axial, clockwise, and counter-clockwise) and the like may have been used herein. Such directional references are only used for identification purposes to aid the reader’s understanding of the present disclosure. For example, the term “distal” may refer to the end farthest away from the medical professional / operator when introducing a device into a patient, while the term “proximal” may refer to the end closest to the medical professional when introducing a device into a patient. Such directional references do not necessarily create limitations, particularly as to the position, orientation, or use of this disclosure. As such, directional references should not be limited to specific coordinate orientations, distances, or sizes, but are used to describe relative positions referencing particular examples. Such terms are not generally limiting to the scope of the claims made herein. Any example or feature of any section, portion, or any other component shown or particularly described in relation to any preceding or subsequent example of similar sections, portions, or components herein may be interchangeably applied to any other similar example or feature shown or described herein.

[0193] It should be understood that, as described herein, an “example” (such as illustrated in the accompanying Figures) or “example” (such as “in some examples”) may refer to an illustrative representation of an environment or article or component in which a disclosed concept or feature may be provided or embodied, or to the representation of a manner in which just the concept or feature may be provided or embodied. However, such illustrated examples are to be understood as examples (unless otherwise stated), and other manners of embodying the described concepts or features, such as may be understood by one of ordinary skill in the art upon learning the concepts or features from the present disclosure, are within the scope of the disclosure. Furthermore, references to “one example” of the present disclosure are not intended to be interpreted as excluding the existence of additional implementations, configurations, and / or examples that also incorporate the recited features.

[0194] In addition, it will be appreciated that while the Figures may show one or more examples of concepts or features together in a single example of an environment, article, or component incorporating such concepts or features, such concepts or features are to beAttorney Docket No.: 8178.6140WO understood (unless otherwise specified) as independent of and separate from one another and are shown together for the sake of convenience and without intent to limit to being present or used together. For instance, features illustrated or described as part of one example can be used separately, or with another example to yield a still further example. Thus, it is intended that the present subject matter covers such modifications and variations as come within the scope of the appended claims and their equivalents.

[0195] As used herein, an element or step recited in the singular and proceeded with the word “a” or “an” should be understood as not excluding plural elements or steps, unless such exclusion is explicitly recited. It will be further understood that the terms “comprises” and / or “comprising,” or “includes” and / or “including” when used herein, specify the presence of stated features, regions, steps, elements and / or components, but do not preclude the presence or addition of one or more other features, regions, integers, steps, operations, elements, components and / or groups thereof.

[0196] The phrases “at least one,” “one or more,” and “and / or,” as used herein, are open- ended expressions that are both conjunctive and disjunctive in operation. The terms “a” (or “an”), “one or more” and “at least one” can be used interchangeably herein.

[0197] Connection references (e.g., engaged, attached, coupled, connected, and joined) are to be construed broadly and may include intermediate members between a collection of elements and relative to movement between elements unless otherwise indicated. As such, connection references do not necessarily infer that two elements are directly connected and in fixed relation to each other. Identification references (e.g., primary, secondary, first, second, third, fourth, etc.) are not intended to connote importance or priority but are used to distinguish one feature from another. The drawings are for purposes of illustration only and the dimensions, positions, order and relative to sizes reflected in the drawings attached hereto may vary.

[0198] The foregoing discussion has been presented for purposes of illustration and description and is not intended to limit the disclosure to the form or forms disclosed herein. For example, various features of the disclosure are grouped together in one or more examples or configurations for the purpose of streamlining the disclosure. However, itAttorney Docket No.: 8178.6140WO should be understood that various features of the certain some examples or configurations of the disclosure may be combined in alternate examples or configurations. Moreover, the following claims are hereby incorporated into this Detailed Description by this reference, with each claim standing on its own as a separate example of the present disclosure.

Claims

Attorney Docket No.: 8178.6140WOCLAIMSWhat is claimed is:

1. A method comprising: accessing surface image data of a knee joint of a patient; processing the surface image data to generate a set of knee segments; accessing patient information associated with the patient, the patient information including physical or demographic information; using a simulation system to generate a simulation of the knee joint, the simulation generating knee performance data, wherein the simulation is generated based on the set of knee segments and the patient information; determining, from the simulation of the knee joint and knee performance data, simulated patella tracking of the knee joint; processing intra-operative surface image data of the knee joint to generate a set of intra-operative knee segments; generating, based on the intra-operative surface image data and the set of intraoperative knee segments, an intra-operative representation of knee performance of the knee joint; determining, from the intra-operative representation of the knee performance, intra-operative patella tracking of the knee joint; comparing the simulated patella tracking of the knee joint to the intra-operative patella tracking of the knee joint to determine a deviation therebetween; generating, based on the deviation, a recommendation to alter a surgical plan for the knee joint.

2. The method of claim 1, wherein the surface image data comprises image data for a tissue surface of the knee joint.

3. The method of claim 2, wherein the intra-operative surface image data comprises image data for an implant surface of the knee joint.Attorney Docket No.: 8178.6140WO4. The method of claim 2, wherein the intra-operative surface image data comprises image data for a trial surface or instrument surface.

5. The method of claim 1, wherein processing the surface image data includes producing a three-dimensional (3D) reconstruction of the knee joint from point clouds based upon calculated 3D coordinates from the surface image data.

6. The method of claim 1, wherein the surface image data includes raw video data of the knee joint of the patient; wherein generating the set of knee segments includes: processing the raw video data of the knee joint to classify segments of the knee joint as a femur, a tibia, a patella, or portions thereof; identifying landmarks on the femur, tibia, patella, or portions thereof; and using the landmarks to determine features of a corresponding segment of the knee joint, the features including points, facets, borders, sides, or edges of the corresponding segment; and wherein the simulation of the knee joint and the knee performance data are based on the determined features of the corresponding segment.

7. The method of claim 1, wherein the patient information includes patient physical or demographic information; and wherein the patient physical or demographic information includes at least one selected from gender, age, ethnicity, height, weight, femur shape, Q-angle, varus / valgus angle, patella tilt, patella medio-lateral (ML) location, patella superior-inferior (SI) location, tendon locations, or ligament locations.Attorney Docket No.: 8178.6140WO8. The method of claim 1, wherein the simulation of the knee joint includes simulating knee activities that include bone or ligament inputs with knee bends, gait analysis, or patella movement or tracking; and wherein the simulation is configured to provide knee performance or kinematic data.

9. The method of claim 1, wherein the simulation system includes a computational model, an artificial intelligence (Al) or machine learning (ML) model, a simulation algorithm, one or more simulation equations, or one or more simulation software platforms.

10. The method of claim 1, wherein the intra-operative surface image data includes raw video data of the knee joint of the patient during a knee operation; wherein generating the set of intra-operative knee segments includes: processing the raw video data of the intra-operative knee joint to classify segments of the intra-operative knee joint as a femur, a tibia, a patella, or portions thereof; identifying landmarks on the femur, tibia, patella, or portions thereof; and using the landmarks to determine features of a corresponding intraoperative segment of the intra-operative knee joint, the features including points, facets, borders, sides, or edges of the corresponding intra-operative segment; and wherein the intra-operative representation of the knee performance of the knee joint is based on the determined features of the corresponding intra-operative segment.

11. The method of claim 1, wherein the intra-operative representation of the knee performance is configured to provide knee performance or kinematic data of the intraoperative knee joint.Attorney Docket No.: 8178.6140WO12. The method of claim 1, wherein the simulated patella tracking of the knee joint includes simulated patella output information including simulated patella kinematic data, simulated ligament strain data, simulated tendon strain data, or simulated bone strain data.

13. The method of claim 12, wherein the intra-operative patella tracking of the knee joint includes intra-operative patella output information including intra-operative patella kinematic data, intra-operative ligament strain data, intra-operative tendon strain data, or intra-operative bone strain data.

14. The method of claim 13, wherein comparing the simulated patella tracking of the knee joint to the intra-operative patella tracking of the knee joint includes: comparing the patella output information to the intra-operative patella output information to determine a deviation between: the simulated patella kinematic data and the intra-operative patella kinematic data; the simulated ligament strain data and the intra-operative ligament strain data; the simulated tendon strain data and the intra-operative tendon strain data; or the simulated bone strain data and the intra-operative bone strain data.

15. A computing device comprising: a processing circuit; and a memory coupled to the processing circuit, the memory including executable instructions, which when executed causes the processing circuit to: access surface image data of a knee joint of a patient; process the surface image data to generate a set of knee segments;Attorney Docket No.: 8178.6140WO access patient information associated with the patient, the patient information including physical or demographic information; use a simulation system to generate a simulation of the knee joint, the simulation generating knee performance data, wherein the simulation is generated based on the set of knee segments and the patient information; determine, from the simulation of the knee performance, simulated patella tracking of the knee joint; process intra-operative surface image data of the knee joint to generate a set of intra-operative knee segments; generate, based on the intra-operative surface image data and the set of intraoperative knee segments, an intra-operative representation of knee performance of the knee joint; determine, from the intra-operative representation of the knee performance, intraoperative patella tracking of the knee joint; compare the simulated patella tracking of the knee joint to the intra-operative patella tracking of the knee joint to determine a deviation therebetween; generate, based on the deviation, a recommendation to alter a surgical plan for the knee joint.

16. The computing device of claim 15, wherein the surface image data comprises image data for a tissue surface of the knee joint.

17. The computing device of claim 16, wherein the intra-operative surface image data comprises image data for an implant surface of the knee joint.

18. The computing device of claim 16, wherein the intra-operative surface image data comprises image data for a trial surface or instrument surface.Attorney Docket No.: 8178.6140WO19. The computing device of claim 15, wherein processing the surface image data includes the processing circuit being caused to produce a three-dimensional (3D) reconstruction of the knee joint from point clouds based upon calculated 3D coordinates from the surface image data.

20. The computing device of claim 15, wherein the surface image data includes raw video data of the knee joint of the patient; wherein generating the set of knee segments includes the processing circuit being caused to: process the raw video data of the knee joint to classify segments of the knee joint as a femur, a tibia, a patella, or portions thereof; identify landmarks on the femur, tibia, patella, or portions thereof; and use the landmarks to determine features of a corresponding segment of the knee joint, the features including points, facets, borders, sides, or edges of the corresponding segment; and wherein the simulation of the knee joint and the knee performance data is based on the determined features of the corresponding segment.

21. The computing device of claim 15, wherein the patient information includes patient physical or demographic information; and wherein the patient physical or demographic information includes at least one selected from gender, age, ethnicity, height, weight, femur shape, Q-angle, varus / valgus angle, patella tilt, patella medio-lateral (ML) location, patella superior-inferior (SI) location, tendon locations, or ligament locations.

22. The computing device of claim 15, wherein the simulation of the knee joint includes the processing circuit being caused to simulate knee activities that include bone or ligament inputs with knee bends, gait analysis, or patella movement or tracking; andAttorney Docket No.: 8178.6140WO wherein the simulation is configured to provide knee performance or kinematic data.

23. The computing device of claim 15, wherein the simulation system includes a computational model, an artificial intelligence (Al) or machine learning (ML) model, a simulation algorithm, one or more simulation equations, or one or more simulation software platforms.

24. The computing device of claim 15, wherein the surface intra-operative image data includes raw video data of the knee joint of the patient during a knee operation; wherein generating the set of intra-operative knee segments includes the processing circuit being caused to: process the raw video data of the intra-operative knee joint to classify segments of the intra-operative knee joint as a femur, a tibia, a patella, or portions thereof; identify landmarks on the femur, tibia, patella, or portions thereof; and use the landmarks to determine features of a corresponding intra-operative segment of the intra-operative knee joint, the features including points, facets, borders, sides, or edges of the corresponding intra-operative segment; and wherein intra-operative representation of the knee performance of the knee joint is based on the determined features of the corresponding intra-operative segment.

25. The computing device of claim 15, wherein the intra-operative representation of the knee performance is configured to provide knee performance or kinematic data of the intra-operative knee joint.

26. The computing device of claim 15, wherein the simulated patella tracking of the knee joint includes simulated patella output information including simulated patellaAttorney Docket No.: 8178.6140WO kinematic data, simulated ligament strain data, simulated tendon strain data, or simulated bone strain data.

27. The computing device of claim 26, wherein the intra-operative patella tracking of the knee joint includes intra-operative patella output information including intraoperative patella kinematic data, intra-operative ligament strain data, intra-operative tendon strain data, or intra-operative bone strain data.

28. The computing device of claim 28, wherein comparing the simulated patella tracking of the knee joint to the intra-operative patella tracking of the knee joint includes the processing circuit being caused to: compare the simulated patella output information to the intra-operative patella output information to determine a deviation between: the simulated patella kinematic data and the intra-operative patella kinematic data; the simulated ligament strain data and the intra-operative ligament strain data; the simulated tendon strain data and the intra-operative tendon strain data; or the simulated bone strain data and the intra-operative bone strain data.

Images