Portal estimaton in minimally invasive surgery

Abstract

A system for determining a location of a portal in an anatomical structure or surface includes memory storing instructions and one or more processing devices configured to execute the instructions. Executing the instructions causes the one or more processing devices to determine a position of at least one of a camera and an instrument within a surgical environment, identify a set of geometric entities associated with the at least one of the camera and the instrument, determine the location of the portal based on the identified set of geometric entities, and at least one of present, on a display, a visualization of the location of the portal and store data indicating the location of the portal.

Description

PT-6210- WO- PCTPORTAL ESTIMATON IN MINIMALLY INVASIVE SURGERYCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional App. 63 / 755,782 filed February 7, 2025, the entire contents of which are incorporated herein by reference.FIELD

[0002] The present disclosure relates surgical navigation systems and methods, and more particularly to surgical navigation systems and methods for estimating portal locations for minimally invasive surgery.BACKGROUND

[0003] The background description provided here is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.

[0004] Arthroscopic surgical procedures are minimally invasive surgical procedures in which access to the surgical site within the body is by way of small keyholes or ports through the patient’s skin. The various tissues within the surgical site are visualized by way of an arthroscope placed through a port or portal, and the internal scene is shown on an external display device. The tissue may be repaired or replaced through the same or additional ports. In computer-assisted surgical procedures (e.g., surgical procedures associated with a knee or knee joint, surgical procedures associated with a hip or hip joint, etc.), the location of various objects with the surgical site may be tracked relative to the bone by way of images captured by an arthroscope and a three-dimensional model of the bone.PT-6210-WO-PCTSUMMARY

[0005] A system for determining a location of a portal in an anatomical structure or surface includes memory storing instructions and one or more processing devices configured to execute the instructions. Executing the instructions causes the one or more processing devices to determine a position of at least one of a camera and an instrument within a surgical environment, identify a set of geometric entities associated with the at least one of the camera and the instrument, determine the location of the portal based on the identified set of geometric entities, and at least one of present, on a display, a visualization of the location of the portal and store data indicating the determined location of the portal.

[0006] In other features, the set of geometric entities includes a set of lines corresponding to an axis of the instrument. The set of geometric entities includes at least one of a set of lines, a set of points, and a set of planes. Determining the position of the instrument includes obtaining an image feed of the surgical environment, detecting a fiducial marker associated with the instrument, and determining the position of the instrument based on the fiducial marker. Determining the position of the instrument includes obtaining and using data indicating a relationship between a location of the fiducial marker and the position of the instrument. Determining the position of the instrument includes determining the position of the instrument further based on a relationship between a location of the fiducial marker and a base marker fixed to patient anatomy.

[0007] In other features, determining the location of the portal further comprises performing clustering to identify the set of geometric entities. Performing clustering includes defining a three-dimensional surface containing the surgical environment and identifying a cluster of points based on an intersection between the set of geometric entities and the three-dimensional surface. Determining the location of the portal includes modeling the portal as a point on the anatomical structure or surface. Determining the location of the portal includes modeling the portal as a line segment passing through the anatomical structure or surface. Determining the location of the portal includes modeling the portal as a three-dimensional surface. Executing the instructions causes the one or more processing devices to obtain, from the camera,PT-6210-WO-PCTan image feed of the surgical environment and determine the position of the instrument based on the image feed. The camera corresponds to an arthroscopic camera.

[0008] A method includes, using one or more processing devices, obtaining an image feed of a surgical environment, determining, based on the image feed, a position of at least one of a camera and an instrument within the surgical environment, identifying a set of geometric entities associated with the at least one of the camera and the instrument, determining a location of a portal through an anatomical structure or surface based on the identified set of geometric entities, and at least one of presenting, on a display, a visualization of the location of the portal and storing data indicative of the determined location of the portal.

[0009] In other features, the set of geometric entities includes at least one of a set of lines, a set of points, and a set of planes. Determining the position of the instrument includes detecting, based on the image feed, a fiducial marker associated with the instrument and determining the position of the instrument based on the fiducial marker. Determining the position of the instrument includes at least one of obtaining and using data indicating a relationship between a location of the fiducial marker and the position of the instrument and determining the position of the instrument further based on a relationship between the location of the fiducial marker and a base marker fixed to patient anatomy. Determining the location of the portal further comprises performing clustering to identify the set of geometric entities. Performing clustering includes defining a three-dimensional surface containing the surgical environment and identifying a cluster of points based on an intersection between the set of geometric entities and the three-dimensional surface. Determining the location of the portal includes at least one of modeling the portal as a point on the anatomical structure or surface, modeling the portal as a line segment passing through the anatomical structure or surface, and modeling the portal as a three-dimensional surface.

[0010] Further areas of applicability of the present disclosure will become apparent from the detailed description, the claims and the drawings. The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the disclosure.PT-6210-WO-PCTBRIEF DESCRIPTION OF THE DRAWINGS

[0011] For a detailed description of example embodiments, reference will now be made to the accompanying drawings in which:

[0012] FIG. 1 shows a surgical system in accordance with at least some embodiments;

[0013] FIG. 2 shows a conceptual drawing of a surgical site with various objects within the surgical site tracked, in accordance with at least some embodiments;

[0014] FIG. 3 shows a method in accordance with at least some embodiments;

[0015] FIG. 4 is an example video display showing portions of a femur and a bone fiducial during a registration procedure, in accordance with at least some embodiments;

[0016] FIG. 5 shows a method in accordance with at least some embodiments;

[0017] FIGS. 6A, 6B, and 6C illustrate example image frames acquired during videobased surgical navigation (VBSN) in accordance with at least some embodiments;

[0018] FIGS. 7A, 7B, and 7C illustrate various example surgical tools / instruments in accordance with at least some embodiments;

[0019] FIGS. 8A and 8B show an example sagittal view and coronal view of a femur in accordance with at least some embodiments;

[0020] FIG. 9 shows an example of clustering of sets of lines in accordance with at least some embodiments;

[0021] FIG. 10 shows an example method for performing portal estimation in accordance with at least some embodiments; and

[0022] FIG. 11 shows an example computer system or computing device configured to implement the various systems and methods of the present disclosure.

[0023] In the drawings, reference numbers may be reused to identify similar and / or identical elements.DEFINITIONS

[0024] Various terms are used to refer to particular system components. Different companies may refer to a component by different names - this document does not intend to distinguish between components that differ in name but not function. In the following discussion and in the claims, the terms “including” and “comprising” are used in an open-ended fashion, and thus should be interpreted to mean “including, but notPT-6210-WO-PCTlimited to... .” Also, the term “couple” or “couples” is intended to mean either an indirect or direct connection. Thus, if a first device couples to a second device, that connection may be through a direct connection or through an indirect connection via other devices and connections.

[0025] Similarly, spatial and functional relationships between elements (for example, between device, modules, circuit elements, etc.) are described using various terms, including “connected,” “engaged,” “coupled,” “adjacent,” “next to,” “on top of,” “above,” “below,” and “disposed.” Unless explicitly described as being “direct,” when a relationship between first and second elements is described in the above disclosure, that relationship can be a direct relationship where no other intervening elements are present between the first and second elements, but can also be an indirect relationship where one or more intervening elements are present (either spatially or functionally) between the first and second elements. Nevertheless, this paragraph shall serve as antecedent basis in the claims for referencing any electrical connection as “directly coupled” for electrical connections shown in the drawing with no intervening element(s).

[0026] Terms of degree, such as “substantially” or “approximately,” are understood by those skilled in the art to refer to reasonable ranges around and including the given value and ranges outside the given value, for example, general tolerances associated with manufacturing, assembly, and use of the embodiments. The term “substantially,” when referring to a structure or characteristic, includes the characteristic that is mostly or entirely present in the characteristic or structure. As one example, numerical values that are described as “approximate” or “approximately” as used herein may refer to a value within + / - 5% of the stated value.

[0027] “A”, “an”, and “the” as used herein refers to both singular and plural referents unless the context clearly dictates otherwise. By way of example, “a processor” programmed to perform various functions refers to one processor programmed to perform each and every function, or more than one processor collectively programmed to perform each of the various functions. To be clear, an initial reference to “a [referent]”, and then a later reference for antecedent basis purposes to “the [referent]”, shall not obviate the fact the recited referent may be plural.PT-6210-WO-PCT

[0028] In general, terminology may be understood at least in part from usage in context. For example, terms, such as “and”, “or”, or “and / or,” as used herein may include a variety of meanings that may depend at least in part upon the context in which such terms are used. Typically, “or” if used to associate a list, such as A, B or C, is intended to mean A, B, and C, here used in the inclusive sense, as well as A, B or C, here used in the exclusive sense. In addition, the term “one or more” as used herein, depending at least in part upon context, may be used to describe any feature, structure, or characteristic in a singular sense or may be used to describe combinations of features, structures or characteristics in a plural sense. Similarly, terms, such as “a,” “an,” or “the,” again, may be understood to convey a singular usage or to convey a plural usage, depending at least in part upon context. In addition, the term “based on” may be understood as not necessarily intended to convey an exclusive set of factors and may, instead, allow for existence of additional factors not necessarily expressly described, again, depending at least in part on context. As used herein, the phrase at least one of A, B, and C should be construed to mean a logical (A OR B OR C), using a non-exclusive logical OR, and should not be construed to mean “at least one of A, at least one of B, and at least one of C.”

[0029] The terms “input” and “output” when used as nouns refer to connections (e.g., electrical, software) and / or signals, and shall not be read as verbs requiring action. For example, a timer circuit may define a clock output. The example timer circuit may create or drive a clock signal on the clock output. In systems implemented directly in hardware (e.g., on a semiconductor substrate), these “inputs” and “outputs” define electrical connections and / or signals transmitted or received by those connections. In systems implemented in software, these “inputs” and “outputs” define parameters read by or written by, respectively, the instructions implementing the function. In examples where used in the context of user input, “input” may refer to actions of a user, interactions with input devices or interfaces by the user, etc.

[0030] “Controller,” “module,” or “circuitry” shall mean, alone or in combination, individual circuit components, an application specific integrated circuit (ASIC), a microcontroller with controlling software, a reduced-instruction-set computer (RISC) with controlling software, a digital signal processor (DSP), a processor with controlling software, a programmable logic device (PLD), a field programmable gate arrayPT-6210-WO-PCT(FPGA), or a programmable system-on-a-chip (PSOC), configured to read inputs and drive outputs responsive to the inputs.

[0031] As used to describe various surgical instruments or devices, such as a probe, the term “proximal” refers to a point or direction nearest a handle of the probe (e.g., a direction opposite the probe tip). Conversely, the term “distal” refers to a point or direction nearest the probe tip (e.g., a direction opposite the handle).

[0032] For the purposes of this disclosure, a non-transitory computer readable medium (or computer-readable storage medium / media) stores computer data, which data can include computer program code (or computer-executable instructions) that is executable by a computer, in machine-readable form. By way of example, and not limitation, a computer readable medium may comprise computer readable storage media, for tangible or fixed storage of data, or communication media for transient interpretation of code-containing signals. Computer readable storage media, as used herein, refers to physical or tangible storage (as opposed to signals) and includes without limitation volatile and non-volatile, removable and non-removable media implemented in any method or technology for the tangible storage of information such as computer-readable instructions, data structures, program modules or other data. Computer readable storage media includes, but is not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, optical storage, cloud storage, magnetic storage devices, or any other physical or material medium which can be used to tangibly store the desired information or data or instructions and which can be accessed by a computer or processor.

[0033] For the purposes of this disclosure, the term “server” should be understood to refer to a service point that provides processing, database, and communication facilities. By way of example, and not limitation, the term “server” can refer to a single, physical processor with associated communications and data storage and database facilities, or it can refer to a networked or clustered complex of processors and associated network and storage devices, as well as operating software and one or more database systems and application software that support the services provided by the server. Cloud servers are examples.

[0034] For the purposes of this disclosure, a “network” should be understood to refer to a network that may couple devices so that communications may be exchanged, suchPT-6210-WO-PCTas between a server and a client device or other types of devices, including between wireless devices coupled via a wireless network, for example. A network may also include mass storage, such as network attached storage (NAS), a storage area network (SAN), a content delivery network (CDN) or other forms of computer or machine-readable media, for example. A network may include the Internet, one or more local area networks (LANs), one or more wide area networks (WANs), wire-line type connections, wireless type connections, cellular or any combination thereof. Likewise, sub-networks, which may employ differing architectures or may be compliant or compatible with differing protocols, may interoperate within a larger network.

[0035] For purposes of this disclosure, a “wireless network” should be understood to couple client devices with a network. A wireless network may employ stand-alone ad-hoc networks, mesh networks, Wireless LAN (WLAN) networks, cellular networks, or the like. A wireless network may further employ a plurality of network access technologies, including Wi-Fi, Long Term Evolution (LTE), WLAN, Wireless Router (WR) mesh, or 2nd, 3rd, 4thor 5thgeneration (2G, 3G, 4G or 5G) cellular technology, mobile edge computing (MEC), Bluetooth, 802.11 b / g / n, or the like. Network access technologies may enable wide area coverage for devices, such as client devices with varying degrees of mobility, for example. In short, a wireless network may include virtually any type of wireless communication mechanism by which signals may be communicated between devices, such as a client device or a computing device, between or within a network, or the like.

[0036] A computing device may be capable of sending or receiving signals, such as via a wired or wireless network, or may be capable of processing or storing signals, such as in memory as physical memory states, and may, therefore, operate as a server. Thus, devices capable of operating as a server may include, as examples, dedicated rackmounted servers, desktop computers, laptop computers, set top boxes, integrated devices combining various features, such as two or more features of the foregoing devices, or the like.

[0037] For purposes of this disclosure, a client (or consumer or user) device, referred to as user equipment (UE)), may include a computing device capable of sending or receiving signals, such as via a wired or a wireless network. A client device may, for example, include a desktop computer or a portable device, such as a cellular telephone,PT-6210-WO-PCTa smart phone, a display pager, a radio frequency (RF) device, an infrared (IR) device a Near Field Communication (NFC) device, a Personal Digital Assistant (PDA), a handheld computer, a tablet computer, a phablet, a laptop computer, a set top box, a wearable computer, smart watch, an integrated or distributed device combining various features, such as features of the forgoing devices, or the like.

[0038] In some embodiments, as discussed below, the client device can also be, or can communicatively be coupled to, any type of known or to be known medical device (e.g., any type of Class I, II or III medical device), such as, but not limited to, a MRI machine, CT scanner, Electrocardiogram (ECG or EKG) device, photopletismograph (PPG), Doppler and transmit-time flow meter, laser Doppler, an endoscopic device neuromodulation device, a neurostimulation device, and the like, or some combination thereof.DETAILED DESCRIPTION

[0039] The present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, which form a part hereof, and which show, by way of non-limiting illustration, certain example embodiments. Subject matter may, however, be embodied in a variety of different forms and, therefore, covered or claimed subject matter is intended to be construed as not being limited to any example embodiments set forth herein; example embodiments are provided merely to be illustrative. Likewise, a reasonably broad scope for claimed or covered subject matter is intended. Among other things, for example, subject matter may be embodied as methods, devices, components, or systems. Accordingly, embodiments may, for example, take the form of hardware, software, firmware or any combination thereof (other than software per se). The following detailed description is, therefore, not intended to be taken in a limiting sense.

[0040] Throughout the specification and claims, terms may have nuanced meanings suggested or implied in context beyond an explicitly stated meaning. Likewise, the phrase “in one embodiment” as used herein does not necessarily refer to the same embodiment and the phrase “in another embodiment” as used herein does not necessarily refer to a different embodiment. It is intended, for example, that claimed subject matter include combinations of example embodiments in whole or in part.PT-6210-WO-PCT

[0041] The present disclosure is described below with reference to block diagrams and operational illustrations of methods and devices. It is understood that each block of the block diagrams or operational illustrations, and combinations of blocks in the block diagrams or operational illustrations, can be implemented by means of analog or digital hardware and computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer to alter its function as detailed herein, a special purpose computer, ASIC, or other programmable data processing apparatus, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, implement the functions / acts specified in the block diagrams or operational block or blocks. In some alternate implementations, the functions / acts noted in the blocks can occur out of the order noted in the operational illustrations. For example, two blocks shown in succession can in fact be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality / acts involved.

[0042] Computer-Aided Surgery (CAS) and surgical navigation systems support surgeons in planning and performing complex surgical procedures with increased precision and accuracy. As one example surgical procedure, arthroscopy is a minimally invasive medical procedure for diagnosing and treating joint problems. An orthopedic surgeon makes a small incision in the skin of the patient and inserts a lens into the incision. The lens is attached to a camera and coupled to a light source, allowing the joint to be visualized and treated. Surgical navigation and CAS systems have had significant impact in minimally invasive surgeries (MIS) such as arthroscopic procedures because the increased difficulty in visualizing the anatomy of the patient further complicates the surgical workflow.

[0043] Video-based surgical navigation (VBSN) leverages visual fiducials or markers (also called visual markers) attached to patient anatomy to guide the surgeon throughout the medical procedure. The video-based navigation process requires the precise registration of a pre-operative anatomical model with data acquired intra-operatively. The registration process requires the surgeon to digitize the surface of interest that corresponds to the pre-operative model. The visual markers attached toPT-6210-WO-PCTthe anatomies define reference frames to which the pre-operative model and the intraoperative acquired data are aligned.

[0044] Typically, MIS is performed by inserting surgical instruments and a camera, which can be an endoscope, a laparoscope, an arthroscope, etc., through small incisions called portals (or ports). The efficiency and success of MIS is significantly impacted by the location of the portals as it dictates, for instance, the ability to reach a surgical site, the ability to complete surgical tasks due to relative orientation of the instruments and camera and the existence of internal instrument / scope collisions. Poorly-placed portals greatly increase the difficulty of the procedure, which in turn can lead to costly delays and possibly even additional incisions.

[0045] In VBSN, fiducial markers with known visual patterns may be attached both to the targeted anatomy and to the instruments and subsequently tracked such that their relative poses can be accurately estimated (e.g., by applying 3D computer vision methods on the images / video acquired by a camera). These relative poses allow to locate the instruments with respect to the anatomy at every frame time instant. While VSBN can be applied in open surgery, the main field of application for VSBN is sports medicine, in which case the video can be acquired by a standard arthroscopic camera. However, misplaced portals can preclude a pre-operative plan from being achieved and hinder visualization of the surgical site or relevant parts of the anatomy. Knowing the location of the portals with respect to the relevant anatomy is, thus, important for the success of arthroscopic VBSN.

[0046] In some examples of MIS, portals may be located and opened based on external anatomical landmarks (e.g., a location of the patella, a height of the tibial plateau, etc.). However, the high anatomical variability among patients and the difficulty in identifying landmarks make this process error-prone, non-repeatable, and inaccurate. Given the importance of accurately knowing the location of portals, extensive research has been done to devise methods for portal (portal location) estimation, with the majority of methods focusing on robot-assisted surgeries. One common aspect among these methods is that the portal is modeled as a 3D point (e.g., instruments are considered as pivoting about the incision point, such that the incision point acts as a fulcrum).PT-6210-WO-PCT

[0047] In some example methods for robot-assisted procedures, various instrument axes are detected using external measurement equipment (e.g., cameras position within the line-of-sight of the surgical site) and a mathematical approach is used to estimate an optimal intersection between the detected axes / lines. In other example methods, force sensors are used to measure forces generated as the instrument pivots about the incision. In still other examples, robot kinematic data is used to identify a point that minimizes the distance to all recorded instrument axes. However, in any of these methods, the elasticity of the skin causes displacements of the incision portals which can impact robot performance whenever the portals are modeled as 3D points.

[0048] Portal estimation systems and methods according to the principles of the present disclosure implement techniques for locating portals for MIS that includes VBSN. The techniques described herein improve portal estimation / location (e.g., by modeling the portal location relative to a position of an instrument).

[0049] VBSN facilitates the tracking of instruments with respect to the targeted anatomy to which a fiducial is rigidly attached (which may be referred to as a “base marker”). During arthroscopic procedures, maneuverability inside the joint is limited because the instruments are inserted through portals. Properly locating the portals with respect to the anatomy allows visualization of the relevant parts of the anatomy to be maximized and facilitates execution and achievement of the desired preoperative plan. For existing portals, knowing their location provides an indication to the surgeon whether the plan is achievable prior to an actual guidance step.

[0050] The surgical instruments may be pre-calibrated (e.g., the location of certain geometric entities such as points, lines and planes, etc. of the instruments may be known with respect to the fiducial that is rigidly attached to the instruments). While the instruments are moving, motion of the instruments is tracked and the location of the geometric entities can be represented in the reference system of the base marker. Similarly, the mechanical axis of the camera or other relevant geometric entity can be estimated on a per-frame basis and represented in the reference system of the base marker. Knowing that the motion of the instruments and the camera, and thus the location of the geometric entities, is constrained by the portal facilitates estimation of the location of the portal using the data represented in the coordinate system of thePT-6210-WO-PCTbase marker as described below in more detail. For example, since skin has non-zero thickness and is elastic, the portal may not be well-modeled by a single 3D point (e.g., the fulcrum). Accordingly, portal estimation techniques according to the present disclosure consider more complex models such as spheres, lines (e.g., sets orclusters of lines), cylinders, sets of 3D points, etc. to describe / describe portal location.

[0051] FIG. 1 shows an example surgical system (e.g., a system including or implementing an arthroscopic video-based navigation system) 100 in accordance with at least some embodiments of the present disclosure. In particular, the example surgical system 100 comprises a tower or device cart 102 and various tools or instruments, such as an example mechanical resection instrument 104, an example plasma-based ablation instrument (hereafter just ablation instrument 106), and an endoscope in the example form of an arthroscope 108 and attached camera head or camera 110. In the example systems, the arthroscope 108 may be a rigid device, unlike endoscopes for other procedures, such as upper-endoscopies. The device cart 102 may comprise a display device 114, a resection controller 116, and a camera control unit (CCU) together with an endoscopic light source and video (e.g., a VBN) controller 118. In example cases the combined CCU and video controller 118 not only provides light to the arthroscope 108 and displays images received from the camera 110, but also implements various additional aspects, such as registering a three-dimensional bone model with the bone visible in the video images, and providing computer-assisted navigation during the surgery. Thus, the combined CCU and video controller are hereafter referred to as surgical controller 118. In other cases, however, the CCU and video controller may be a separate and distinct system from the controller that handles registration and computer-assisted navigation, yet the separate devices would nevertheless be operationally coupled.

[0052] The example device cart 102 further includes a pump controller 122 (e.g., single or dual peristaltic pump). Fluidic connections of the mechanical resection instrument 104 and ablation instrument 106 to the pump controller 122 are not shown so as not to unduly complicate the figure. Similarly, fluidic connections between the pump controller 122 and the patient are not shown so as not to unduly complicate the figure. In the example system, both the mechanical resection instrument 104 and the ablation instrument 106 are coupled to the resection controller 116 being a dual-PT-6210-WO-PCTfunction controller. In other cases, however, there may be a mechanical resection controller separate and distinct from an ablation controller. The example devices and controllers associated with the device cart 102 are merely examples, and other examples include vacuum pumps, patient-positioning systems, robotic arms holding various instruments, ultrasonic cutting devices and related controllers, patientpositioning controllers, and robotic surgical systems.

[0053] FIGS. 1 and 2 further show additional instruments that may be present during an arthroscopic surgical procedure. In particular, an example probe 124 (e.g., shown as a touch probe, but which may be a touchless probe in other examples), a drill guide or aimer 126, and a bone fiducial 128 are shown. The probe 124 may be used during the surgical procedure to provide information to the surgical controller 118, such as information to register a three-dimensional bone model to an underlying bone visible in images captured by the arthroscope 108 and camera head 110. In some surgical procedures, the aimer 126 may be used as a guide for placement and drilling with a drill wire to create an initial or pilot tunnel through the bone. The bone fiducial 128 may be affixed or rigidly attached to the bone and serve as an anchor location for the surgical controller 118 to know the position and orientation of the bone (e.g., after registration of a three-dimensional bone model). Additional tools and instruments may be present, such as the drill wire, various reamers for creating the throughbore and counterbore aspects of a tunnel through the bone, and various tools, such as for suturing and anchoring a graft. These additional tools and instruments are not shown so as not to further complicate the figure.

[0054] Example workflow for a surgical procedure is described below. While described with respect to an example anterior cruciate ligament repair procedure, the below techniques may also be performed for other types of surgical procedures, such as hip procedures or other procedures that include joint distraction. A surgical procedure may begin with a planning phase. An example procedure may start with imaging (e.g., X-ray imaging, computed tomography (CT), magnetic resonance imaging (MRI)) of the anatomy of the patient, including the relevant anatomy (e.g., for a knee procedure the lower portion of the femur, the upper portion of the tibia, and the articular cartilage; for a hip procedure, an upper portion of the femur, the acetabulum / hip joint, pelvis, etc.). The imaging may be preoperative imaging, hoursPT-6210-WO-PCTor days before the intraoperative repair, or the imaging may take place within the surgical setting just prior to the intraoperative repair. The discussion that follows assumes MRI imaging, but again many different types of imaging may be used. The image slices from the MRI imaging can be segmented such that a volumetric model or three-dimensional model of the anatomy is created. Any suitable currently available, or after developed, segmentation technology may be used to create the three-dimensional model. More specifically to the example of anterior cruciate ligament repair, a three-dimensional bone model of the lower portion of the femur, including the femoral condyles, is created. Conversely, for a hip procedure, a three-dimensional model of the upper portion of the femur and at least a portion of the pelvis (e.g., the acetabulum) is created.

[0055] Using the three-dimensional bone model, an operative plan is created. For a knee procedure, the results of the planning may include: a three-dimensional bone model of the distal end of the femur; a three-dimensional bone model for a proximal end of the tibia; an entry location and exit location through the femur and thus a planned-tunnel path for the femur; and an entry location and exit location through the tibia and thus a planned-tunnel path through the tibia. Other surgical parameters may also be selected during the planning, such as tunnel throughbore diameters, tunnel counterbore diameters and depth, desired post-repair flexion, and the like, but those additional surgical parameters are omitted so as not to unduly complicate the specification.

[0056] Conversely, for a hip procedure, the results of the planning may include a three-dimensional bone model of the proximal end of the femur; a three-dimensional bone model for at least a portion of the pelvis / hip joint (e.g., a region of the pelvis corresponding to the acetabulum); a surgical area of interest within the hip joint; and parameters associated with achieving an amount of distraction in the surgical area of interest to provide sufficient access to the surgical area of interest. For example, example hip procedures may include, but are not limited to, labral repair, femoroacetabular impingement (FAI) debridement (e.g., removal of bone spurs / growths), cartilage repair, and synovectomy (e.g., removal of inflamed tissue). These example procedures typically require access to a specific surgical area of interest within the hip joint (i.e., in a specific area within an interface between the pelvisPT-6210-WO-PCTand the femoral head, such as an area around / surrounding a bone spur or growth, cartilage or tissue to be repaired or removed, etc.).

[0057] The intraoperative aspects include steps and procedures for setting up the surgical system to perform the various repairs. It is noted, however, that some of the intraoperative aspects (e.g., optical system calibration) may take place before any portals or incisions are made through the patient’s skin, and in fact before the patient is wheeled into the surgical room. Nevertheless, such steps and procedures may be considered intraoperative as they take place in the surgical setting and with the surgical equipment and instruments used to perform the actual repair.

[0058] An example procedure can be conducted arthroscopically and is computer-assisted in the sense that the surgical controller 118 is used for arthroscopic navigation within the surgical site. More particularly, in example systems the surgical controller 118 provides computer-assisted navigation during the procedure by tracking locations of various objects within the surgical site, such as the location of the bone within the three-dimensional coordinate space of the view of the arthroscope, and location of the various instruments within the three-dimensional coordinate space of the view of the arthroscope. A brief description of such tracking techniques is described below.

[0059] FIG. 2 shows a conceptual drawing of a surgical site with various objects (e.g., surgical instruments / tools) within the surgical site. In particular, visible in FIG. 2 is a distal end of the arthroscope 108, a portion of a bone 200 (e.g., femur), the bone fiducial 128 within the surgical site, and the probe 124.

[0060] The arthroscope 108 illuminates the surgical site with visible light. In the example of FIG. 2, the illumination is illustrated by arrows 208. The illumination provided to the surgical site is reflected by various objects and tissues within the surgical site, and the reflected light that returns to the distal end enters the arthroscope 108, propagates along an optical channel within the arthroscope 108, and is eventually incident upon a capture array within the camera 110 (FIG. 1). The images detected by the capture array within the camera 110 are sent electronically to the surgical controller 118 (FIG. 1) and displayed on the display device 114 (FIG. 1). In one example, the arthroscope 108 is monocular or has a single optical path through the arthroscope for capturing images of the surgical site, notwithstanding that the single optical path may be constructed of two or more optical members (e.g., glass rods,PT-6210-WO-PCToptical fibers). That is to say, in example systems and methods the computer-assisted navigation provided by the arthroscope 108, the camera 110, and the surgical controller 118 is provided with the arthroscope 108 that is not a stereoscopic endoscope having two distinct optical paths separated by an interocular distance at the distal end endoscope.

[0061] During a surgical procedure, a surgeon selects an arthroscope with a viewing direction beneficial for the planned surgical procedure. Viewing direction refers to a line residing at the center of an angle subtended by the outside edges or peripheral edges of the view of an endoscope. The viewing direction for some arthroscopes is aligned with the longitudinal central axis of the arthroscope, and such arthroscopes are referred to as “zero degree” arthroscopes (e.g., the angle between the viewing direction and the longitudinal central axis of the arthroscope is zero degrees). The viewing direction of other arthroscopes forms a non-zero angle with the longitudinal central axis of the arthroscope. For example, for a 30° arthroscope the viewing direction forms a 30° angle to the longitudinal central axis of the arthroscope, the angle measured as an obtuse angle beyond the distal end of the arthroscope. In the example of FIG. 2, the view angle 210 of the arthroscope 108 forms a non-zero angle to the longitudinal central axis 212 of the arthroscope 108.

[0062] Still referring to FIG. 2, within the view of the arthroscope 108 is a portion of the bone 200 (in this example, within the intercondylar notch), along with the example bone fiducial 128, and the example probe 124. The example bone fiducial 128 is multifaceted element, with each face or facet having a fiducial disposed or created thereon. However, the bone fiducial need not have multiple faces, and in fact may take any shape so long as that shape can be tracked within the video images. The bone fiducial, such as bone fiducial 128, may be attached to the bone 200 in any suitable form (e.g., via the screw portion of the bone fiducial 128 visible in FIG. 1). The patterns of the fiducials on each facet are designed to provide information regarding the position and orientation of the bone fiducial 128 in the three-dimensional coordinate space of the view of the arthroscope 108. More particularly, the pattern is selected such that the position and orientation of the bone fiducial 128 may be determined from images captured by the arthroscope 108 and attached camera (FIG. 1).PT-6210-WO-PCT

[0063] The probe 124 is also shown as partially visible within the view of the arthroscope 108. The probe 124 may be used, as discussed more below, to identify a plurality of surface features on the bone 200 as part of the registration of the bone 200 to the three-dimensional bone model. In some cases the probe 124 and / or the aimer 126 may carry their own, unique fiducials, such that their respective poses may be calculated from the one or more fiducial present in the video stream. However, in other cases, and as shown, the medical instrument used to help with registration of the three-dimensional bone model, be it the probe 124, the aimer 126, or any other suitable medical device, may omit carrying fiducials. Stated otherwise, in such examples the medical instrument has no fiducial markings. In such cases, the pose of the medical instrument may be determined by a machine learning model, discussed in more detail below.

[0064] The images captured by the arthroscope 108 and attached camera are subject to optical distortion in many forms. For example, the visual field between distal end of the arthroscope 108 and the bone 200 within the surgical site is filled with fluid, such as bodily fluids and saline used to distend the joint. Many arthroscopes have one or more lenses at the distal end that widen the field of view, and the wider field of view causes a “fish eye” effect in the captured images. Further, the optical elements within the arthroscope (e.g., rod lenses) may have optical aberrations inherent to the manufacturing and / or assembly process. Further still, the camera may have various optical elements for focusing the images received onto the capture array, and the various optical elements may have aberrations inherent to the manufacturing and / or assembly process. In example systems, prior to use within each surgical procedure, the endoscopic optical system is calibrated to account for the various optical distortions. The calibration creates a characterization function that characterizes the optical distortion, and further analysis of the frames of the video stream may be, prior to further analysis, compensated using the characterization function.

[0065] The next example step in the intraoperative procedure is the registration of the bone model created during the planning stage. During the intraoperative repair, the three-dimensional bone model is obtained by or provided to the surgical controller 118. Again using the example of anterior cruciate ligament repair, and specifically computer-assisted navigation for tunnel paths through the femur, the three-PT-6210-WO-PCTdimensional bone model of the lower portion of the femur is obtained by or provided to the surgical controller 118. Thus, the surgical controller 118 receives the three-dimensional bone model, and assuming the arthroscope 108 is inserted into the knee by way of a port or portal through the patient’s skin, the surgical controller 118 also receives video images of a portion of the lower end of the femur. In order to relate the three-dimensional bone model to the images received by way of the arthroscope 108 and camera 110, the surgical controller 118 registers the three-dimensional bone model to the images of the femur received by way of the arthroscope 108 and camera 110.

[0066] In order to perform the registration, and in accordance with example methods, the bone fiducial 128 is attached to the femur. The bone fiducial placement is such that the bone fiducial is within the field of view of the arthroscope 108. In examples for knee procedures, the bone fiducial 128 is placed within the intercondylar notch superior to the expected location of the tunnel through lateral condyle. Conversely, in examples for hip procedures, the bone fiducial 128 is placed on the femoral head. To relate or register bone visible in the video images to the three-dimensional bone model, the surgical controller 118 (FIG. 1) is provided or determines a plurality of surface features of an outer surface of the bone. Identifying the surface features may take several forms, including a touch-based registration using the probe 124 without a carried fiducial, a touchless registration technique in which the surface features are identified after resolving the motion of the arthroscope 108 and camera relative to the bone fiducial 128, and a third technique in which uses a patient-specific instrument.

[0067] In the example touch-based registration, the surgeon may touch a plurality of locations using the probe 124 (FIG. 1). In some cases, particularly when portions of the outer surface of the bone are exposed to view, receiving the plurality of surface features of the outer surface of the bone may involve the surgeon “painting” the outer surface of the bone. “Painting” is a term of art that does not involve application of color or pigment, but instead implies motion of the probe 124 when the distal end of the probe 124 is touching bone. In this example, the probe 124 does not carry or have a fiducial visible to the arthroscope 108 and the camera 110. It follows that the pose of the probe 124 and the location of the distal tip of the probe 124 needs to be determinedPT-6210-WO-PCTin order to gather the surface features for purposes of registering the three-dimensional bone model.

[0068] FIG. 3 shows a method 300 in accordance with at least some embodiments of the present disclosure. The example method 300 may be implemented in software within a computer system, such as the surgical controller 118. In particular, the example method 300 comprises obtaining a three-dimensional bone model (block 302). That is to say, in the example method 300, what is obtained is the three-dimensional bone model that may be created by segmenting a plurality of non-invasive images (e.g., CT, MRI) taken preoperatively or intraoperatively. With the bone segmented from or within the images, the three-dimensional bone model may be created. The three-dimensional bone may take any suitable form, such as a computer-aided design (CAD) model, a point cloud of data points with respect to an arbitrary origin, or a parametric representation of a surface expressed using analytical mathematical equations. Thus, the three-dimensional bone model is defined with respect to the origin and in any suitable an orthogonal basis.

[0069] The next step in the example method 300 is capturing video images of the bone fiducial attached to the bone (block 304). The capturing is performed intraoperatively. In an example, the capturing of video images is by way of the arthroscope 108 and camera 110. Other endoscopes may be used, such as endoscopes in which the capture array resides at the distal end of the device (e.g., chip-on-the-tip devices). However, in open procedures where the skin is cut and pulled away, exposing the bone to the open air, the capturing may be by any suitable camera device, such as one or both cameras of a stereoscopic camera system, or a portable computing device, such as a tablet or smart-phone device. The video images may be provided to the surgical controller 118 in any suitable form.

[0070] The next step in the example method 300 is determining locations of a distal tip of the medical instrument visible within the video images (block 306), where the distal tip is touching the bone in at least some of the frames of the video images, and the medical instrument does not have a fiducial. Determining the locations of the distal tip of the medical instrument may take any suitable form. In one example, determining the locations may include segmenting the medical instrument in the frames of the video images (block 308). The segmenting may take any suitable form, such asPT-6210-WO-PCTapplying the video images to a segmentation machine learning algorithm. The segmentation machine learning algorithm may take any suitable form, such as neural network or convolution neural network trained with a training data set showing the medical instrument in a plurality of known orientations. The segmentation machine learning algorithm may produce segmented video images where the medical instrument is identified or highlighted in some way (e.g., box, brightness increased, other objects removed).

[0071] With the segmented video images, the example method 300 may estimate a plurality of poses of the medical instrument within a respective plurality of frames of the video images (block 310). The estimating the poses may take any suitable form, such as applying the video images to a pose machine learning algorithm. The pose machine learning algorithm may take any suitable form, such as neural network or convolution neural network trained to perform six-dimensional pose estimation. The resultant of the pose machine learning algorithm may be, for at least some of the frames of the video image, an estimated pose of the medical instrument in the reference frame of the video images and / or in the reference frame provided by the bone fiducial. That is, the resultant of the pose machine learning algorithm may be a plurality of poses, one pose each for at least some of the frames of the segmented video images. While in many cases a pose may be determined for each frame, in other cases it may not be possible to make a pose estimation for at least some frame because of video quality issues, such as motion blur caused by electronic shutter operation.

[0072] The next step in the example method 300 is determining the locations based on the plurality of poses (block 312). In particular, for each frame for which a pose can be estimated, based on a model of the medical device the location of the distal tip can be determined in the reference frame of the video images and / or the bone fiducial. Thus, the resultant is a set of locations that, at least some of which, represent locations of the outer surface of the bone.

[0073] FIG. 3 shows an example three-step process for determining the locations of the distal tip of the medial instrument. However, the method 300 is merely an example, and many variations are possible. For example, a single machine learning model, such as a convolution neural network, may be set up and trained to perform all threePT-6210-WO-PCTsteps as a single overall process, though there may be many hidden layers of the convolution neural network. That is, the convolution neural network may segment the medical instrument, perform the six-dimensional pose estimation, and determine the location of the distal tip in each frame. The training data set in such a situation would include a data set in which each frame has the medical device segmented, the sixdimensional pose identified, and the location of the distal tip identified. The output of the determining step 306 may be a segmented video stream distinct from the video images captured at step 304. In such cases, the later method steps may use both segmented video stream and the video images to perform the further tasks. In other cases, the location information may be combined with the video images, such as being embedded in the video images, or added as metadata to each frame of the video images.

[0074] FIG. 4 is an example video display showing portions of a femur and a bone fiducial during a registration procedure. Although described with respect to a distal end of a femur, the principles and techniques described and shown in FIG. 4 can be applied to other anatomical structures / procedures, such as a femoral head for hip procedures as described herein. The display may be shown, for example, on the display device 114 associated with the device cart 102, or any other suitable location. In particular, visible in the main part of the display of FIG. 4 is an intercondylar notch 400, a portion of the lateral condyle 402, a portion the medial condyle 404, and the example bone fiducial 128. Shown in the upper right corner of the example display is a depiction of the bone, which may be a rendering 406 of the bone created from the three-dimensional bone model. Shown on the rendering 406 is a recommended area 408, the recommended area 408 being portions of the surface of the bone to be “painted” as part of the registration process. Shown in the lower right corner of the example display is a depiction of the bone, which again may be a rendering 412 of the bone created from the three-dimensional bone model. Shown on the rendering 412 are a plurality of surface features 416 on the bone model that have been identified as part of the registration process. Further shown in the lower right corner of the example display is progress indicator 418, showing the progress of providing and receiving of locations on the bone. The example progress indicator 418 is a horizontal bar havingPT-6210-WO-PCTa length that is proportional to the number of locations received, but any suitable graphic or numerical display showing progress may be used (e.g., 0% to 100%).

[0075] Referring to both the main display and the lower right rendering, as the surgeon touches the outer surface of the bone within the images captured by the arthroscope 108 and camera 110, the surgical controller 118 receives the surface features on the bone, and may display each location both within the main display as dots or locations 416, and within the rendering shown in the lower right corner. More specifically, the example surgical controller 118 overlays indications of identified surface features 416 on the display of the images captured by the arthroscope 108 and camera 110, and in the example case shown, also overlays indications of identified surface features 416 on the rendering 412 of the bone model. Moreover, as the number of identified locations 416 increases, the surgical controller 118 also updates the progress indicator 418.

[0076] Still referring to FIG. 4, in spite of the diligence of the surgeon, not all locations identified by the surgical controller 118 based on the surgeon’s movement of the probe 124 result in valid locations on the surface of the bone. In the example of FIG. 4, as the surgeon moves the probe 124 from the inside surface of the lateral condyle 102 to the inside surface of the medial condyle 104, the surgical controller 118, based on the example six-dimensional pose estimation, receives several locations 420 that likely represent locations at which the distal end of the probe 124 was not in contact with the bone.

[0077] With reference to FIG. 3, the plurality of surface features 416 may be, or the example surgical controller 118 may generate, a registration model relative to the bone fiducial 128 (block 314). The registration model may take any suitable form, such as a computer-aided design (CAD) model or point cloud of data points in any suitable orthogonal basis. The registration model, regardless of the form, may have fewer overall data points or less “structure” than the bone model created by the non-invasive computer imaging (e.g., MRI). However, the goal of the registration model is to provide the basis for the coordinate transforms and scaling used to correlate the bone model to the registration model and relative to the bone fiducial 128. Thus, the next step in the example method 300 is registering the bone model relative to the location of the bone fiducial based on the registration model (block 316). Registration mayPT-6210-WO-PCTconceptually involve testing a plurality of coordinate transformations and scaling values to find a correlation that has a sufficiently high correlation or confidence factor. Once a correlation is found with the sufficiently high confidence factor, the bone model is said to be registered to the location of the bone fiducial. Thereafter, the example registration method 300 may end (block 318); however, the surgical controller 118 may then use the registered bone model to provide computer-assisted navigation regarding a procedure involving the bone.

[0078] In the examples discussed to this point, registration of the bone model involves a touch-based registration technique using the probe 124 without a carried fiducial. However, other registration techniques are possible, such as a touchless registration technique. The example touchless registration technique again relies on placement of the bone fiducial 128. As before, when the viewing direction of the arthroscope 108 is relatively constant, the bone fiducial may have fewer faces with respective fiducials. Once placed, the bone fiducial 128 represents a fixed location on the outer surface of the bone in the view of the arthroscope 108, even as the position of the arthroscope 108 is moved and changed relative to the bone fiducial 128. Again, in order to relate or register the bone visible in the video images to the three-dimensional bone model, the surgical controller 118 (FIG. 1) determines a plurality of surface features of an outer surface of the bone, and in this example determining the plurality of surface features is based on a touchless registration technique in which the surface features are identified based on motion of the arthroscope 108 and camera 110 relative to the bone fiducial 128.

[0079] Another technique for registering the bone model to the bone uses a patientspecific instrument. In both touch-based and touchless registration techniques, a registration model is created, and the registration model is used to register the bone model to the bone visible in the video images. Conceptually, the registration model is used to determine a coordinate transformation and scaling to align the bone model to the actual bone. However, if the orientation of the bone in the video images is known or can be determined, use of the registration model may be omitted, and instead the coordinate transformations and scaling may be calculated directly.

[0080] FIG. 5 shows a method 500 in accordance with at least some embodiments. The example method may be implemented in software within one or more computerPT-6210-WO-PCTsystems, such as, in part, the surgical controller 118. In particular, the example method 500 comprises obtaining a three-dimensional bone model (block 502). In the patient-specific instrument registration technique, what is obtained is the three-dimensional bone model that may be created by segmenting a plurality of non-invasive images (e.g., MRI) taken preoperatively or intraoperatively.

[0081] The method 500 further includes generating a patient-specific instrument that has a feature designed to couple to the bone represented in the bone model in only one orientation (block 504). Generating the patient-specific instrument may first involve selecting a location at which the patient-specific instrument will attach. For example, a device or computer system may analyze the bone model and select the attachment location. In various examples, the attachment location may be a unique location in the sense that, if a patient-specific instrument is made to couple to the unique location, the patient-specific instrument will not couple to the bone at any other location. In the example case of an anterior cruciate ligament repair, the location selected may be at or near the upper or superior portion on the intercondylar notch. If the bone model shows another location with a unique feature, such as a bone spur or other raised or sunken surface anomaly, such a unique location may be selected as the attachment location for the patient-specific instrument. For example, for hip procedures, the location may be selected based on a location, within the hip joint, of a bone spur or other anatomical feature associated with the hip procedure.

[0082] Moreover, forming the patient-specific instrument may take any suitable form. In one example, a device or computer system may directly print, such as using a 3D printer, the patient-specific instrument. In other cases, the device or computer system may print a model of the attachment location, and the model may then become the mold for creating the patient-specific instrument. For example, the model may be the mold for an injection-molded plastic or casting technique. In some examples, the patient-specific instrument carries one or more fiducials, but as mentioned above, in other cases the patient-specific instrument may itself be tracked and thus carry no fiducials.

[0083] The method 500 further includes coupling the patient-specific instrument to the bone, in some cases the patient-specific instrument having the fiducial coupled to an exterior surface (block 506). As described above, the attachment location for thePT-6210-WO-PCTpatient-specific instrument can be selected to be unique such that the patient-specific instrument couples to the bone in only one location and in only one orientation. In the example case of an arthroscopic procedure, the patient-specific instrument may be inserted arthroscopically. That is, the attachment location may be selected such that a physical size of the patient-specific instrument enables insertion through the ports / portals in the patient’s skin. In other cases, the patient-specific instrument may be made or constructed of a flexible material that enables the patient-specific instrument to deform for insertion in the surgical site, yet return to the predetermined shape for coupling to the attachment location. However, in open procedures where the skin is cut and pulled away, exposing the bone to the open air, the patient-specific instrument may be a rigid device with fewer size restrictions.

[0084] The method 500 further includes capturing video images of the patientspecific instrument (block 508). Here again, the capturing may be performed intraoperatively. In the example case of an arthroscopic anterior cruciate ligament repair, the capturing of video images is by the surgical controller 118 by way of arthroscope 108 and camera 110. However, in open procedures where the skin is cut and pulled away, exposing the bone to the open air, the capturing may be by any suitable camera device, such as one or both cameras of a stereoscopic camera systems, or a portable computing device, such as a tablet or smart-phone device. In such cases, the video images may be provided to the surgical controller 118 in any suitable form.

[0085] The example method 500 further includes registering the bone model based on the location of the patient-specific instrument (block 510). That is, given that the patient-specific instrument couples to the bone at only one location and in only one orientation, the location and orientation of the patient-specific instrument is directly related to the location and origination of the bone, and thus the coordinate transformations and scaling for the registration may be calculated directly. Thereafter, the example method 500 may end; however, the surgical controller 118 may then use the registered bone model to provide computer-assisted navigation regarding a surgical task or surgical procedure involving the bone.

[0086] For example, with the registered bone model the surgical controller 118 may provide guidance regarding a surgical task of a surgical procedure. The specificPT-6210-WO-PCTguidance is dependent upon the surgical procedure being performed and the stage of the surgical procedure. A non-exhaustive list of guidance comprises: changing a drill path entry point; changing a drill path exit point; aligning an aimer along a planned drill path; showing location at which to cut and / or resect the bone; reaming the bone by a certain depth along a certain direction; placing a device (suture, anchor or other) at a certain location; placing a suture at a certain location; placing an anchor at a certain location; showing regions of the bone to touch and / or avoid; and identifying regions and / or landmarks of the anatomy. In yet still other cases, the guidance may include highlighting within a version of the video images displayed on a display device, which can be the arthroscopic display or a see-through display, or by communicating to a virtual reality device or a robotic tool.

[0087] Various steps, actions, functions, etc. of the systems and methods described above in FIGS. 1-5 require ports or portals (which may be used interchangeably herein) through the skin of the patient to provide access to patient anatomy. Knowledge of locations of these portals can be used to provide guidance during MIS performed using VBSN. Accordingly, portal estimation systems and methods according to the principles of the present disclosure implement techniques for locating portals for MIS using VBSN as described below in more detail. The techniques described herein improve portal estimation / location (e.g., by modeling the portal location relative to a position of an instrument). Modeling can be performed using standard arthroscopic footage that is acquired during VBSN. In VBSN, the anatomy and surgical tools are instrumented with visual fiducials / fiducial markers (e.g., a base marker) that can be detected by the arthroscopic camera. Poses of different surgical instruments (e.g., a touch probe, tibial and femoral aimers, etc.) can be tracked and, at every frame time instant, the surgical instruments can be represented in the coordinate system of the base marker. A location of a portal can be determined based on positions of one or more of the surgical instruments relative to patient anatomy.

[0088] Considering a fiducial that is rigidly attached to the bodies of the tools (e.g., a tool marker), it is possible to represent a set of geometric entities that are relevant for portal estimation in the coordinate system of the tool marker. At each frame-time instant, by tracking both the base marker and the tool marker, the geometric entity can be represented in the reference frame of the base marker. Similarly to surgical tools,PT-6210-WO-PCTthe arthroscopic camera is inserted into the joint through a portal. At every frame-time instant, the mechanical axis of the camera can be estimated based on the current lens position and its coordinates may be represented in the reference frame of the base marker. Further, if the anatomy has been previously registered with a pre-operative model as described above, the geometric entities can be represented in the same reference system as the model, providing a visualization of one or more geometric entities in the same coordinate system as the pre-operative bone model (e.g., a set of lines corresponding to touch probe lines passing through a given portal).

[0089] While the example above is provided for lines passing through a single portal, it may be the case that the surgeon inserted the surgical instruments through multiple portals during the procedure and / or that the camera’s mechanical axis is also being considered for portal estimation. In this scenario, there are lines / planes corresponding to more than one portal that cannot be considered simultaneously in the estimation process. Before applying the estimation method, the set of geometric entities can be divided into different subsets, each corresponding to a different portal, using, for instance, a clustering scheme.

[0090] In case the geometric entities are lines, clustering may be performed by considering a 3D surface that encloses the targeted anatomy, such as a sphere or a cylinder, and intersecting each line with the 3D surface. This will generate a set of 3D points that can be clustered using methods such as mean shift, in case the number of clusters is not known, or, in other cases, k-means.

[0091] Locations of portals can then be estimated using the clustered data. In some examples, the portal can be modeled as a 3D point (e.g., a point corresponding to an intersection of all lines or planes). As another example, the portal can be modeled as a 3D point based on an intersection of the lines with cylinders having a range of radii.

[0092] In other examples, a portal can be estimated as a 3D line or line segment (e.g., by computing the median or average direction of all the lines, finding the line that is simultaneously closest to all lines, etc.).

[0093] In still other examples, a portal can be modeled as a 3D surface (e.g., a 3D entity with an associated area or volume). As one example, a sphere centered on a 3D point and having a radius based on the spread of intersections of the lines with a 3D surface that encloses the targeted bone is determined.PT-6210-WO-PCT

[0094] The techniques for locating portals (or determination locations of portals) as described herein can be used for both determining and modeling (e.g., for display / visualization) locations of existing portals and determining and modeling locations for future / subsequent placement of (new) portals.

[0095] Accordingly, portal estimation techniques according to the present disclosure provide various improvements to MIS procedures using VBSN. For example, portal location may depend on surgeon preference (e.g., surgeons, including surgeons performing robot-assisted surgery, may choose portal locations according to personal preference and previous surgical experience). Accordingly, systems and methods according to the present disclosure may be configured to determine the preferred portal location for a particular surgeon and automatically indicate where to perform the incisions in subsequent surgeries.

[0096] In other examples, for a given portal location, surgeons can be provided with information that indicates which regions of the anatomy are accessible and visible for a particular set of instruments, the scope, etc., and whether a surgical plan can be executed.

[0097] In other examples, opening of new portals can be guided / informed based on locations of existing portals and / or locations of portals in previous procedures.

[0098] In other example, estimates of instrument / tool and camera poses can be refined based on estimated portal locations.

[0099] In still other examples, some types of surgical instruments (e.g., surgical instruments with moving parts) can be automatically adjusted based on portal location. For example, some procedures (e.g., drilling of the ACL tibial tunnel performed during ACLR) require the surgeon to define a particular angle of the aimer that will be used for placing the guidewire. By knowing the location of the portals, this angle may be determined automatically such that a planned tunnel can be achieved with minimal effort.

[0100] Portal estimation techniques according to the present disclosure will now be described in more detail. Portal estimation techniques according to the present disclosure include estimating portal locations using, as input, standard arthroscopic footage that is acquired during VBSN. In VBSN, the anatomy and surgical tools include visual fiducials / markers that can be detected by the arthroscopic camera. AlthoughPT-6210-WO-PCTdescribed with respect to tools including fiducial markers, the principles of the present disclosure can also be implemented using tools that do not include fiducial markers. For example, geometric entities of tools as described herein can be represented in any coordinate system associated with a particular tool (e.g., in a coordinate system of a CAD model used to model the tool).

[0101] FIGS. 6A, 6B, and 6C illustrate example image frames 600, 604, and 608, respectively, from arthroscopic footage obtained during VBSN for use in a medical procedure (e.g., for ACL reconstruction). Each of the image frames 600, 604, and 608 includes a different surgical instrument (e.g., a touch probe 612, a tibial aimer 616, and a femoral aimer 620, respectively), each including a respective fiducial marker 624. By tracking the poses of these instruments at every frame time instant (e.g., based on detection of the fiducial marker 624), the instrument can be represented in the coordinate system of a base marker (e.g., a fiducial marker 628 fixed in patient anatomy). Further, in accordance with the principles of the present disclosure, a set of geometric entities relevant for portal estimation can also be represented in the coordinate system of the base marker as described below in more detail.

[0102] FIGS. 7A, 7B, and 7C illustrate various example surgical tools / instruments (e.g., a touch probe 700, a tibial aimer 704, and a femoral aimer 708, respectively, corresponding to the touch probe 612, the tibial aimer 616, and the femoral aimer 620 of FIGS. 6A, 6B, and 6C). Each of the instruments has one or more geometric entities (e.g., objects, characteristics, features, etc.) and / or sets of entities. In particular, each of the instruments has one or more geometric entities (e.g., corresponding to physical structures of the instruments) that pass through a portal in the skin of the patient when the instrument is inserted through the portal into a surgical site / environment (e.g., a joint). For example, the touch probe 700 includes an axis 712 (e.g., a longitudinal axis) and one or more planes or planar surfaces (e.g., a plane that includes / is defined by the longitudinal axis and the tip of the touch probe 700). In some examples, the touch probe 700 may include one or more fiducial markers 716. Similarly, the tibial aimer 704 includes an axis 720 (e.g., a longitudinal axis of an arm 722) and one or more planes (e.g., a plane that includes / is defined by the axis 720 and a tip of the aimer 704. The femoral aimer 708 includes an axis 728.PT-6210-WO-PCT

[0103] At each frame-time instant (e.g., for each frame of an image feed of the surgical environment that includes a base marker, such as the base marker 628, and a surgical instrument), positions of the base marker and markers on the instruments can be tracked by the surgical navigation system as described herein. Further, the position and orientation of geometric entities corresponding to the instruments can also be determined. In other words, since the position / location of the base marker and the instrument is known, corresponding positions, orientations, etc. of various geometric entities associated with the instrument can be calculated (e.g., since a position of a fiducial marker on an end of the instrument is known, a location / position of a plane, an axis of the instrument, etc. represented in the coordinate system associated with the fiducial marker can also be calculated). As one example, for each instrument, data correlating various geometric entities of an instrument with a location of fiducial markers on the instrument can be determined, stored, retrieved, etc. and used to calculate the geometric entities based on the determined fiducial marker location. As an example, fiducial marker and geometric entity location can be calculated based on a reference frame of the base marker.

[0104] Similarly, an arthroscopic camera is inserted into through the same (or different) portal as the instrument to provide an image feed of the surgical environment. At each frame-time instant, a mechanical axis of the camera can be estimated (e.g., based on the current lens position) and the coordinates of the axis and / or other features of the camera can be represented in the reference frame of the base marker.

[0105] In examples where patient anatomy is previously registered (e.g., using a preoperative model as described above), the geometric entities can be represented in the same reference system as the pre-operative model. A visualization of the geometric entities can be generated and displayed / provided, such as via a display of a surgical navigation system, tablet computer, etc.

[0106] For example, FIGS. 8A and 8B show an example sagittal view 800 and coronal view 804, respectively, of a femur 808. In this example, geometric entities of a touch probe may include a set of lines 812 corresponding to a longitudinal axis of the touch probe, as calculated and represented in accordance with a coordinate system of patient anatomy. For example, the set of lines 812 may correspond toPT-6210-WO-PCTtracked positions of the touch probe during a surgical procedure (e.g., as calculated based on determined positions of a base marker, fiducial markers on the instrument, etc.).

[0107] A location of a portal 816 can be determined based on the tracked positions of the touch probe and the set of lines 812. For example, since the set of lines 812 corresponds to the axis of the touch probe, an intersection point or region 820 of the set of lines 812 can be assumed to correspond to an approximate location (or set of possible locations) of the portal 816. In other words, since the instrument passes through the portal 816, the portal 816 functions as a fulcrum or other convergence point for possible orientations of the instrument. Although shown in FIGS. 8A and 8B as a single location, the portal 816 can be calculated and visually represented as one or more locations, a cylinder or other shape representing multiple possible locations, etc. In this manner, the location of the portal 816 can be calculated / estimated based on tracked positions of the instrument, and can be visually displayed, used for subsequent surgical procedures and planning, etc.

[0108] While described above with respect to the set of lines 812 passing through a single portal 816, in some examples the surgeon may insert surgical instruments through multiple portals during a surgical procedure and / or a mechanical axis of the camera may be used to estimate portal location. In these examples, multiple sets of lines, planes, and other geometric entities corresponding to more than one portal may be defined and may not be considered simultaneously in the estimation process for a single portal. Accordingly, prior to applying the portal location estimation techniques described herein, multiple sets of geometric entities may be divided into different subsets, each corresponding to a different portal (e.g., using a clustering scheme).

[0109] In an example where the geometric entities are sets of lines 900 and 904 as shown in FIG. 9, clustering may be performed by considering a 3D surface that encloses the targeted anatomy, such as a sphere or a cylinder (e.g., a cylinder 908, as shown), and intersecting each line with the 3D surface of the cylinder 908. Intersection points between the sets of lines 900 and 904 define respective sets of 3D points 912 and 916 that can be clustered using various techniques such as mean shifting techniques (e.g., in examples where the number of clusters is not known), k-means techniques, etc.PT-6210-WO-PCT

[0110] In the example shown in FIG. 9, the targeted anatomy is a tibia 920. The lines 900 correspond to positions of a camera while the lines 904 correspond to positions of an instrument. The respective sets of points 912 and 916 are determined and provided to, for example, a clustering algorithm. To perform these clustering techniques, geometric characteristics of the 3D surface (e.g., a diameter of the cylinder 908) is known / predetermined (e.g., based on the size, landmarks, etc. of a bone model of the tibia 920) to facilitate accurate estimation of locations of the points 912, 916. In this manner, systems and methods of the present disclosure can identify which lines in sets of lines 900, 904 correspond to which portal.

[0111] In other examples, the 3D space around the bone model can be sampled to generate assumed / putative portal locations. A voting map can be constructed and provided as input to a peak detection algorithm (such as a peak detection algorithm implemented in conjunction with Hough transform or other feature extraction techniques), with each detected peak representing a portal location.

[0112] In one example implementation, portal estimation using a set of determined lines may include the following steps. First, a point (a reference point) in a working volume is considered / identified. For example, the point may correspond to a location of the base marker within the working volume (e.g., a surgical environment). In another example, the point may correspond to a center of a tibial plateau or another landmark of patient anatomy. The identified point may correspond to as the origin of a reference frame where the lines will be represented / displayed.

[0113] For each line, a plane that contains that line and the origin of the reference frame is determined. The working volume is then sampled (e.g., using spherical coordinates theta and phi that represent elevation and azimuth, respectively). Each theta, phi pair defines a 3D direction that, together with the origin of the reference frame, originates a line that contains the assumed / putative portal location. For each plane, the angles between a normal of the plane and the directions corresponding to all theta, phi pairs are then computed.

[0114] For angles near 90 degrees (e.g., within a threshold, such as 3 degrees, of 90 degrees), the corresponding theta, phi pair is identified as corresponding to a direction of the portal relative to the origin. For example, the theta, phi pair accumulates one “vote,” and a voting map can be constructed based on the votes ofPT-6210-WO-PCTeach theta, phi pair. In other words, the geometric interpretation is that if a line goes through the point representing the portal, then the plane containing the line and the origin also contains the portal, and thus the normal of that plane is perpendicular to the direction the portal makes with the origin of the reference frame. Accordingly, all lines that go through the same portal originate planes that belong to a pencil of planes whose axis of rotation is a direction between the portal and the origin.

[0115] Peaks of the voting map are determined to retrieve directions of each existing portal. Each line is assigned to a cluster using a metric such as the angle calculated between the normal of the plane and the represented direction as described above. With the clustered data calculated in this manner, each portal can be estimated using one or more techniques.

[0116] In one example, a portal can be modeled as a 3D point. For example, it can be assumed that there is a point in space about which the instruments and / or camera rotate. The 3D point can be estimated as a point that intersects all identified lines (or corresponding planes). However, since the portal itself is not a single point and has elasticity, in practice, there is not a single point of intersection. Accordingly, a point that is simultaneously closest to all lines (or planes) can be estimated. This technique can be combined with a random sample consensus (RANSAC)-based technique to limit the effects of outliers.

[0117] In an example, the first clustering technique described above may further be used to intersect the lines with cylinders having radii that vary in a plausible range (i.e. , a plausible range of radius values for a portal size, location, etc.). For each radius value, a spread of the obtained intersection points calculated and the set that yields the smallest spread is considered for estimating the portal location (e.g., by considering a centroid of the intersection points).

[0118] In an example, instead of directly intersecting all lines in the set, whether or not a robust scheme line RANSAC is used, all possible pairs of lines can be determined and the intersection points of each pair of lines are determined to build a graph with nodes corresponding to the respective pairs of lines. Nodes can be connected by an edge if the distance between the intersections of the lines in each pair is below a threshold. A maximum clique can then be determined and all pairs ofPT-6210-WO-PCTlines corresponding to that clique are retrieved. An estimate for the portal location is obtained by intersecting the retrieved pairs of lines.

[0119] In another example, a portal can be modeled as a 3D line. For example, because of the thickness of the skin, a portal is not a single point but rather a 3D line segment (e.g., a line segment extending through a corresponding opening in the skin of the patient, which may be referred to as “portal direction”). Portal estimation / modeling techniques according to the present disclosure can be used to take into account the portal direction when guiding surgical instruments.

[0120] Estimating the direction of the portal may be performed by calculating a median or average direction of all the lines. By also estimating a 3D point as previously described, a 3D line or line segment intersecting the 3D point can be defined. As another example, this 3D line, which represents portal direction, may be estimated in a single step by finding a line that is simultaneously closest to all identified lines (e.g., using a representation with Plucker coordinates or other line representation techniques).

[0121] In another example, a portal can be modeled as a 3D surface. For example, other than skin thickness, portals are also defined by the size and shape of the incision and the elasticity of the skin. Accordingly, the portal can be further represented as a 3D entity, such as a 3D surface, having an associated area and / or volume.

[0122] As one example, a sphere centered around a 3D point (e.g., a 3D point determined as described above) and having a radius dependent on a spread of the intersections of the lines with a 3D surface that encloses the targeted bone can be calculated. As another example, if it is desired to only model the portal at the skin surface, a 3D circle or ellipse that best fits the intersection points can be estimated. For example, by modeling the portal location as an ellipse (i.e. , rather than a sphere), horizontal and vertical portals can be distinguished. In still another example, a general polygon can be defined in 3D to model an outer shape of the portal using the intersection points. By combining polygonal models with the estimation of a 3D line as described above, the portal may be represented in a more complex manner by considering a 3D surface such as cylinder, an ellipsoidal cylinder, etc.

[0123] FIG. 10 shows an example method 1000 for performing portal estimation in accordance with the principles of the present disclosure. As described, the methodPT-6210-WO-PCT1000 may be performed by one or more processing devices or processors, computing devices, etc., such as the system 100 or another computing device executing instructions stored in memory. One or more steps of the method 1100 may be omitted in various examples, and / or may be performed in a different sequence than shown in FIG. 10. The steps may be performed sequentially or non-sequentially, two or more steps may be performed concurrently, etc.

[0124] At 1004, the method 1000 includes obtaining images (e.g., real-time or near real-time images) of a surgical environment including patient anatomy with one or more visual markers (e.g., a bone fiducial marker, which may be referred to as a base marker) fixed to patient anatomy as well as visual / fiducial markers associated with one or more surgical instruments. Obtaining the images may include obtaining images using an arthroscopic camera or other imaging device configured to provide an image feed. In some examples, prior to obtaining the images, an image scan of patient anatomy may be performed, such as by performing a pre-operative imaging scan (e.g., a CT scan), retrieving the stored image scan (as data) from memory, etc. In other examples, other imaging techniques may be used. The images obtained from the image feed may be aligned with a model that is generated based on the image scan to provide visual guidance as described herein. The obtained images may include a sparse or dense set of images of the surgical environment including the one or more visual markers (e.g., at least one visual marker).

[0125] At 1008, the method 1000 includes tracking, and obtaining data indicating, positions of at least one of a surgical instrument and a camera in the surgical environment. For example, locations of geometric entities associated with various surgical instruments, relative to fiducial markers arranged on the instruments, may be calibrated / known as described herein. As one example, relationships between the fiducial markers and locations, positions, orientations, etc. of one or more geometric entities (e.g., lines or axes, points, planes, etc.) of a given surgical instrument are determined and data indicative of these relationships is stored in memory. This data can be retrieved and positions of the surgical instrument within the surgical environment can be tracked based on respective locations of the base marker and the fiducial markers on the instrument. In other words, since the position of the fiducial markers relative to the base marker can be determined from the obtained images andPT-6210-WO-PCTthe positions of various geometric entities of the surgical instrument relative to the fiducial markers are known (e.g., based on store data), positions of the surgical instrument within the surgical environment can be tracked / determined based on the relative positions of the base marker and the fiducial markers of the instrument. Positions of the camera can be tracked in a similar manner.

[0126] At 1012, the method 1000 includes defining or identifying a set of geometric entities of the surgical instrument (and / or the arthroscopic camera) based on the tracked positions of the surgical instrument. The set of geometric entities may be identified for positions of the surgical instrument tracked during a predetermined period of time, all or portion of a surgical procedure, etc. As one example, the set of geometric entities corresponds to a set of lines indicating an axis of the surgical instrument as described above in FIGS. 8A and 8B.

[0127] At 1016, the method 1000 includes determ ining / calculating, based on the set of geometric entities, a location of at least one portal (e.g., a portal through the skin of the patient providing access for one or more instruments, a camera, etc. to the surgical site). As one example, for geometric entities that include a set of lines indicating an axis of the surgical instrument, the location of the portal may be determined based on one or more points of intersection of the set of lines as described herein (e.g., one or more 3D points, such as an estimated point that is closest to all of the lines). In some examples, determining the location of the portal may include determining locations of two or more portals. Determining the locations of two or more portals may include performing clustering to identify two or more sets of geometric entities and determining the locations of respective portals based on the clustering as described above.

[0128] At 1020, the method 1000 includes modeling the portal based on the determined location. Modeling the portal may include, but is not limited to, generating and storing data indicating the location of the portal or displaying visual and / or other guidance indicating the location of the portal. Displaying visual guidance may include displaying one or more visual indicators of the portal, displaying a position of the surgical instrument and / or camera, etc. during any of the steps of the method 1000. For example, while tracking the positions of the instrument during the surgical procedure, the position of the instrument and the location of the portal may be displayed within a model of the surgical environment. As described herein, the portalPT-6210-WO-PCTcan be modeled as a 3D point, line, surface, etc. In some examples, the data indicating the location of the portal may be used for various other functions. As one example, the data may be used to determine various surgical parameters, tool settings, etc., such as setting angles for tibial aimers.

[0129] Although described with respect to ACL reconstruction, portal estimation techniques according to the present disclosure can be used for portal estimation in other types of arthroscopic or laparoscopic procedures. Further, data acquisition is described herein as being performed using visual tracking of instruments, but other sensing modalities can be used, such as optical or electromagnetic tracking. In examples where the instruments are not augmented with a marker, tracking without fiducials using deep learning methods (e.g., via image analysis) can be performed.

[0130] With respect to the clustering of a set of lines / planes as described herein, these techniques may be performed using the maximal cliques method described for the purpose of estimating the portal location. For example, a similar graph can be constructed (e.g., by considering pairs of lines, triplets of planes, etc.), and multiple cliques can be determined, each corresponding to a different portal.

[0131] Clustering may be performed using other suitable techniques, including deep learning techniques (e.g., by considering the arthroscopic video to analyze the direction of view with respect to the anatomy and the surgical instruments) or hypothesize-and-test schemes such as RANSAC, wherein portal location hypoteses are generated by sampling sets of entities and inliers are entities that are close to an hypothesis according to some metric.

[0132] Although the portal estimation techniques described herein consider a single type of geometric entity for modeling the portal (e.g., a 3D point, 3D line, or 3D surface), multiple estimation techniques / geometric entities may be combined (e.g., for visualization purposes or performing different guidance tasks). Further, any of the techniques described herein may be combined with RANSAC-based techniques to ensure robustness to outliers.

[0133] In some example, calibration of non-perspective image systems may be facilitated by considering raxels, or ray pixels, which are virtual photo-sensitive elements that measure light in a single direction. Similarly to the lines used for portal estimation, each raxel has a position and orientation and a set of them can be used toPT-6210-WO-PCTmodel an imaging system. Determining the caustic, i.e., finding the 3D surface to which the raxels are tangent, is relevant in the literature of camera calibration

[0015] , In an analogous manner, determining the caustic of the 3D lines, i.e., the 3D surface to which all lines are tangent, can be employed in portal estimation.

[0134] FIG. 11 shows an example computer system or computing device 1100 configured to implement the various systems and methods of the present disclosure. In one example, the computer system 1100 may correspond to one or more computing devices of the system 100, the surgical controller 118, a device that creates a patientspecific instrument, a tablet device within the surgical room, or any other system that implements any or all the various methods discussed in this specification. For example, the computer system 1100 may be configured to implement all or portions of the method 1000. The computer system 1100 may be connected (e.g., networked) to other computer systems in a local-area network (LAN), an intranet, and / or an extranet (e.g., device cart 102 network), or at certain times the Internet (e.g., when not in use in a surgical procedure). The computer system 1100 may be a server, a personal computer (PC), a tablet computer or any device capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that device. Further, while only a single computer system is illustrated, the term “computer” shall also be taken to include any collection of computers that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein.

[0135] The computer system 1100 includes a processing device 1102, a main memory 1104 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM)), a static memory 1106 (e.g., flash memory, static random access memory (SRAM)), and a data storage device 1108, which communicate with each other via a bus 1110.

[0136] Processing device 1102 represents one or more general-purpose processing devices such as a microprocessor, central processing unit, or the like. More particularly, the processing device 1102 may be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or processors implementing a combination of instruction sets. ThePT-6210-WO-PCTprocessing device 1102 may also be one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. The processing device 1102 is configured to execute instructions for performing any of the operations and steps discussed herein. Once programmed with specific instructions, the processing device 1102, and thus the entire computer system 1100, becomes a special-purpose device, such as the surgical controller 118.

[0137] The computer system 1100 may further include a network interface device 1112 for communicating with any suitable network (e.g., the device cart 102 network). The computer system 1100 also may include a video display 1114 (e.g., the display device 114), one or more input devices 1116 (e.g., a microphone, a keyboard, and / or a mouse), and one or more speakers 1118. In one illustrative example, the video display 1114 and the input device(s) 1116 may be combined into a single component or device (e.g., an LCD touch screen).

[0138] The data storage device 1108 may include a computer-readable storage medium 1120 on which the instructions 1122 (e.g., implementing any methods and any functions performed by any device and / or component depicted described herein) embodying any one or more of the methodologies or functions described herein is stored. The instructions 1122 may also reside, completely or at least partially, within the main memory 1104 and / or within the processing device 1102 during execution thereof by the computer system 1100. As such, the main memory 1104 and the processing device 1102 also constitute computer-readable media. In certain cases, the instructions 1122 may further be transmitted or received over a network via the network interface device 1112.

[0139] While the computer-readable storage medium 1120 is shown in the illustrative examples to be a single medium, the term “computer-readable storage medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store the one or more sets of instructions. The term “computer-readable storage medium” shall also be taken to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present disclosure. The term “computer-PT-6210-WO-PCTreadable storage medium” shall accordingly be taken to include, but not be limited to, solid-state memories, optical media, and magnetic media.

[0140] The foregoing description is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses. The broad teachings of the disclosure can be implemented in a variety of forms. Therefore, while this disclosure includes particular examples, the true scope of the disclosure should not be so limited since other modifications will become apparent upon a study of the drawings, the specification, and the following claims. It should be understood that one or more steps within a method may be executed in different order (or concurrently) without altering the principles of the present disclosure. Further, although each of the embodiments is described above as having certain features, any one or more of those features described with respect to any embodiment of the disclosure can be implemented in and / or combined with features of any of the other embodiments, even if that combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and permutations of one or more embodiments with one another remain within the scope of this disclosure.

Claims

PT-6210-WO-PCTCLAIMSWhat is claimed is:

1. A system for determining a location of a portal in an anatomical structure or surface, the system comprising:memory storing instructions; andone or more processing devices configured to execute the instructions, wherein executing the instructions causes the one or more processing devices to determine a position of at least one of a camera and an instrument within a surgical environment,identify a set of geometric entities associated with the at least one of the camera and the instrument,determine the location of the portal based on the identified set of geometric entities, andat least one of (i) present, on a display, a visualization of the location of the portal and (ii) store data indicating the determined location of the portal.

2. The system of claim 1 , wherein the set of geometric entities includes a set of lines corresponding to an axis of the instrument.

3. The system of claim 1 , wherein the set of geometric entities includes at least one of a set of lines, a set of points, and a set of planes.

4. The system of claim 1 , wherein determining the position of the instrument includes:obtaining an image feed of the surgical environment;detecting a fiducial marker associated with the instrument; anddetermining the position of the instrument based on the fiducial marker.PT-6210-WO-PCT5. The system of claim 4, wherein determining the position of the instrument includes obtaining and using data indicating a relationship between a location of the fiducial marker and the position of the instrument.

6. The system of claim 4, wherein determining the position of the instrument includes determining the position of the instrument further based on a relationship between a location of the fiducial marker and a base marker fixed to patient anatomy.

7. The system of claim 1, wherein determining the location of the portal further comprises performing clustering to identify the set of geometric entities.

8. The system of claim 7, wherein performing clustering includes defining a three-dimensional surface containing the surgical environment and identifying a cluster of points based on an intersection between the set of geometric entities and the three-dimensional surface.

9. The system of claim 1, wherein determining the location of the portal includes modeling the portal as a point on the anatomical structure or surface.

10. The system of claim 1, wherein determining the location of the portal includes modeling the portal as a line segment passing through the anatomical structure or surface.

11. The system of claim 1, wherein determining the location of the portal includes modeling the portal as a three-dimensional surface.

12. The system of claim 1 , wherein executing the instructions causes the one or more processing devices to obtain, from the camera, an image feed of the surgical environment and determine the position of the instrument based on the image feed.

13. The system of claim 12, wherein the camera corresponds to an arthroscopic camera.PT-6210-WO-PCT14. A method comprising, using one or more processing devices: obtaining an image feed of a surgical environment;determining, based on the image feed, a position of at least one of a camera and an instrument within the surgical environment;identifying a set of geometric entities associated with the at least one of the camera and the instrument;determining a location of a portal through an anatomical structure or surface based on the identified set of geometric entities; andat least one of (i) presenting, on a display, a visualization of the location of the portal and (ii) storing data indicative of the determined location of the portal.

15. The method of claim 14, wherein the set of geometric entities includes at least one of a set of lines, a set of points, and a set of planes.

16. The method of claim 14, wherein determining the position of the instrument includes:detecting, based on the image feed, a fiducial marker associated with the instrument; anddetermining the position of the instrument based on the fiducial marker.

17. The method of claim 16, wherein determining the position of the instrument includes at least one of:obtaining and using data indicating a relationship between a location of the fiducial marker and the position of the instrument; anddetermining the position of the instrument further based on a relationship between the location of the fiducial marker and a base marker fixed to patient anatomy.

18. The method of claim 14, wherein determining the location of the portal further comprises performing clustering to identify the set of geometric entities.PT-6210-WO-PCT19. The method of claim 18, wherein performing clustering includes defining a three-dimensional surface containing the surgical environment and identifying a cluster of points based on an intersection between the set of geometric entities and the three-dimensional surface.

20. The method of claim 14, wherein determining the location of the portal includes at least one of modeling the portal as a point on the anatomical structure or surface, modeling the portal as a line segment passing through the anatomical structure or surface, and modeling the portal as a three-dimensional surface.

Images