Video imaging augmentation with a digital stamp

Abstract

Systems and methods of the present disclosure relate to generation and placement of a digital template in a display of patient anatomy. An example system detects a visual marker of a surgical tool in an image feed of the patient anatomy, determines, based on the visual marker, a first location or first orientation of one or more features of the surgical tool relative to the patient anatomy, and generates the digital template based on the first location or the first orientation of the one or more features of the surgical tool. Generating the digital template includes rendering, in the display, the digital template in a second location and a second orientation relative to the surgical tool.

Description

Attorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201VIDEO IMAGING AUGMENTATION WITH A DIGITAL STAMPCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No.63 / 755,422, filed on February 7, 2025. The entire disclosure of the application referenced above is incorporated herein by reference.FIELD

[0002] The present disclosure relates to probes and other instruments for surgical navigation systems, and more particularly to video imaging augmentation of instruments used in surgical navigation systems.BACKGROUND

[0003] The background description provided here is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.

[0004] Arthroscopic surgical procedures are minimally invasive surgical procedures in which access to the surgical site within the body is by way of small keyholes or ports through the patient’s skin. The various tissues within the surgical site are visualized by way of an arthroscope placed through a port, and the internal scene is shown on an external display device. The tissue may be repaired or replaced through the same or additional ports. In computer-assisted surgical procedures (e.g., replacement of the anterior cruciate ligament (ACL), reduction of femora-acetabular impingement), the location of various objects within the surgical site may be determined relative to the bone by way of images captured by an arthroscope and a three-dimensional model of the bone.SUMMARY

[0005] A processor is configured to execute instructions stored in memory to control generation and placement of a digital template in a display of patient anatomy.Attorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 Executing the instructions causes the processor to detect a visual marker of a surgical tool in an image feed of the patient anatomy, determine, based on the visual marker, at least one of a first location and a first orientation of one or more features of the surgical tool relative to the patient anatomy, and generate the digital template based on the at least one of the first location and the first orientation of the one or more features of the surgical tool. Generating the digital template includes rendering, in the display, the digital template in a second location and a second orientation relative to the surgical tool. A first longitudinal axis of the digital template as rendered in the display is at least one of parallel but not colinear with a second longitudinal axis of the surgical tool and not parallel with the second longitudinal axis.

[0006] In other features, the digital template includes numerical measurements indicating distances from the surgical tool in at least one direction not parallel to the second longitudinal axis of the surgical tool. The digital template includes two or more concentric rings indicating radial distances from the surgical tool. The first longitudinal axis is parallel to the second longitudinal axis and offset from the second longitudinal axis in a lateral direction. The digital template includes one of a linear ruler and a full-circle protractor ruler.

[0007] In other features, the digital template includes a cylinder. The digital template includes one of a rectangular box, a circular shape, and an elliptical shape. The digital template has a crescent shape. The image feed includes a fixed fiducial marker arranged in the patient anatomy. Executing the instructions further causes the processor to transfer the digital template from a first coordinate space of the surgical tool to a second coordinate space of the fixed fiducial marker. Executing the instructions further causes the processor to cause the digital template to be displayed independent of the surgical tool.

[0008] A method for controlling generation and placement of a digital template in a display of patient anatomy includes detecting a visual marker of a surgical tool in an image feed of the patient anatomy, determining, based on the visual marker, at least one of a first location and a first orientation of one or more features of the surgical tool relative to the patient anatomy, and generating the digital template based on the at least one of the first location and the first orientation of the one or more features of the surgical tool. Generating the digital template includes rendering, in the display,Attorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 the digital template in a second location and a second orientation relative to the surgical tool. A first longitudinal axis of the digital template as rendered in the display is at least one of parallel but not colinear with a second longitudinal axis of the surgical tool and not parallel with the second longitudinal axis.

[0009] In other features, the digital template includes numerical measurements indicating distances from the surgical tool in at least one direction not parallel to the second longitudinal axis of the surgical tool. The digital template includes two or more concentric rings indicating radial distances from the surgical tool. The first longitudinal axis is parallel to the second longitudinal axis and offset from the second longitudinal axis in a lateral direction. The digital template includes one of a linear ruler and a full-circle protractor ruler.

[0010] In other features, the digital template includes a cylinder. The digital template includes a rectangular box. The digital template has a crescent shape. The image feed includes a fixed fiducial marker arranged in the patient anatomy. The method further includes transferring the digital template from a first coordinate space of the surgical tool to a second coordinate space of the fixed fiducial marker. The method further includes causing the digital template to be displayed independent of the surgical tool.

[0011] In other features, one or more methods include steps corresponding to the functions performed by systems, processors or processing devices, controllers, circuits or circuitry, etc. as described herein.

[0012] Further areas of applicability of the present disclosure will become apparent from the detailed description, the claims and the drawings. The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the disclosure.BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The present disclosure will become more fully understood from the detailed description and the accompanying drawings, wherein:

[0014] FIG. 1 shows a surgical system in accordance with at least some embodiments;Attorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201

[0015] FIG. 2 shows a conceptual drawing of a surgical site with various objects within the surgical site tracked, in accordance with at least some embodiments;

[0016] FIG. 3 shows a method in accordance with at least some embodiments;

[0017] FIG. 4 is an example video display showing portions of a femur and a bone fiducial during a registration procedure, in accordance with at least some embodiments;

[0018] FIG. 5 shows a method in accordance with at least some embodiments;

[0019] FIGS. 6A and 6B show example probes used in surgical navigation systems;

[0020] FIGS. 7A and 7B show probes that include measurement features;

[0021] FIGS. 8A, 8B, 8C, and 8D show example digital templates of rulers displayed based on a detected position of a tool or a fiducial marker in accordance with at least some embodiments;

[0022] FIGS. 9A, 9B, and 9C show example shapes or patterns of a digital template in accordance with at least some embodiments;

[0023] FIGS. 10A and 10B an example display of digital templates for use with an awl in accordance with at least some embodiments;

[0024] FIGS. 11 A, 11 B, 11C, 11D, 11 E, and 11 F show example digital template patterns in accordance with at least some embodiments;

[0025] FIG. 12 shows an example computing device including a touchscreen interface 1202 in accordance with at least some embodiments;

[0026] FIG. 13 illustrates steps of an example method for performing digital template techniques in accordance with at least some embodiments; and

[0027] FIG. 14 shows an example computer system in accordance with at least some embodiments.

[0028] In the drawings, reference numbers may be reused to identify similar and / or identical elements.Attorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 DEFINITIONS

[0029] Various terms are used to refer to particular system components. Different companies may refer to a component by different names - this document does not intend to distinguish between components that differ in name but not function. In the following discussion and in the claims, the terms “including” and “comprising” are used in an open-ended fashion, and thus should be interpreted to mean “including, but not limited to... .” Also, the term “couple” or “couples” is intended to mean either an indirect or direct connection. Thus, if a first device couples to a second device, that connection may be through a direct connection or through an indirect connection via other devices and connections.

[0030] An endoscope having “a single optical path” through an endoscope shall mean that the endoscope is not a stereoscopic endoscope having two distinct optical paths separated by an interocular distance at the light collecting end of the endoscope. The fact that an endoscope has two or more optical members (e.g., glass rods, optical fibers) forming a single optical path shall not obviate the status as a single optical path.

[0031] Similarly, spatial and functional relationships between elements (for example, between device, modules, circuit elements, etc.) are described using various terms, including “connected,” “engaged,” “coupled,” “adjacent,” “next to,” “on top of,” “above,” “below,” and “disposed.” Unless explicitly described as being “direct,” when a relationship between first and second elements is described in the above disclosure, that relationship can be a direct relationship where no other intervening elements are present between the first and second elements, but can also be an indirect relationship where one or more intervening elements are present (either spatially or functionally) between the first and second elements. Nevertheless, this paragraph shall serve as antecedent basis in the claims for referencing any electrical connection as “directly coupled” for electrical connections shown in the drawing with no intervening element(s).

[0032] Terms of degree, such as “substantially” or “approximately,” are understood by those skilled in the art to refer to reasonable ranges around and including the given value and ranges outside the given value, for example, general tolerances associated with manufacturing, assembly, and use of the embodiments. The termAttorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 “substantially,” when referring to a structure or characteristic, includes the characteristic that is mostly or entirely present in the characteristic or structure. As one example, numerical values that are described as “approximate” or “approximately” as used herein may refer to a value within + / - 5% of the stated value.

[0033] “A”, “an”, and “the” as used herein refers to both singular and plural referents unless the context clearly dictates otherwise. By way of example, “a processor” programmed to perform various functions refers to one processor programmed to perform each and every function, or more than one processor collectively programmed to perform each of the various functions. To be clear, an initial reference to “a [referent]”, and then a later reference for antecedent basis purposes to “the [referent]”, shall not obviate the fact the recited referent may be plural.

[0034] In general, terminology may be understood at least in part from usage in context. For example, terms, such as “and”, “or”, or “and / or,” as used herein may include a variety of meanings that may depend at least in part upon the context in which such terms are used. Typically, “or” if used to associate a list, such as A, B or C, is intended to mean A, B, and C, here used in the inclusive sense, as well as A, B or C, here used in the exclusive sense. In addition, the term “one or more” as used herein, depending at least in part upon context, may be used to describe any feature, structure, or characteristic in a singular sense or may be used to describe combinations of features, structures or characteristics in a plural sense. Similarly, terms, such as “a,” “an,” or “the,” again, may be understood to convey a singular usage or to convey a plural usage, depending at least in part upon context. In addition, the term “based on” may be understood as not necessarily intended to convey an exclusive set of factors and may, instead, allow for existence of additional factors not necessarily expressly described, again, depending at least in part on context. As used herein, the phrase at least one of A, B, and C should be construed to mean a logical (A OR B OR C), using a non-exclusive logical OR, and should not be construed to mean “at least one of A, at least one of B, and at least one of C.”

[0035] The terms “input” and “output” when used as nouns refer to connections (e.g., electrical, software) and / or signals, and shall not be read as verbs requiring action. For example, a timer circuit may define a clock output. The example timer circuit may create or drive a clock signal on the clock output. In systemsAttorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 implemented directly in hardware (e.g., on a semiconductor substrate), these “inputs” and “outputs” define electrical connections and / or signals transmitted or received by those connections. In systems implemented in software, these “inputs” and “outputs” define parameters read by or written by, respectively, the instructions implementing the function. In examples where used in the context of user input, “input” may refer to actions of a user, interactions with input devices or interfaces by the user, etc.

[0036] “Controller,” “module,” or “circuitry” shall mean, alone or in combination, individual circuit components, an application specific integrated circuit (ASIC), a microcontroller with controlling software, a reduced-instruction-set computer (RISC) with controlling software, a digital signal processor (DSP), a processor with controlling software, a programmable logic device (PLD), a field programmable gate array (FPGA), or a programmable system-on-a-chip (PSOC), configured to read inputs and drive outputs responsive to the inputs.

[0037] As used to describe various surgical instruments or devices, such as a probe, the term “proximal” refers to a point or direction nearest a handle of the probe (e.g., a direction opposite the probe tip). Conversely, the term “distal” refers to a point or direction nearest the probe tip (e.g., a direction opposite the handle).

[0038] For the purposes of this disclosure, a non-transitory computer readable medium (or computer-readable storage medium / media) stores computer data, which data can include computer program code (or computer-executable instructions) that is executable by a computer, in machine-readable form. By way of example, and not limitation, a computer readable medium may comprise computer readable storage media, for tangible or fixed storage of data, or communication media for transient interpretation of code-containing signals. Computer readable storage media, as used herein, refers to physical or tangible storage (as opposed to signals) and includes without limitation volatile and non-volatile, removable and non-removable media implemented in any method or technology for the tangible storage of information such as computer-readable instructions, data structures, program modules or other data. Computer readable storage media includes, but is not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, optical storage, cloud storage, magnetic storage devices, or any other physical or material medium which can be used to tangibly store the desiredAttorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 information or data or instructions and which can be accessed by a computer or processor.

[0039] For the purposes of this disclosure, the term “server” should be understood to refer to a service point that provides processing, database, and communication facilities. By way of example, and not limitation, the term “server” can refer to a single, physical processor with associated communications and data storage and database facilities, or it can refer to a networked or clustered complex of processors and associated network and storage devices, as well as operating software and one or more database systems and application software that support the services provided by the server. Cloud servers are examples.

[0040] For the purposes of this disclosure, a “network” should be understood to refer to a network that may couple devices so that communications may be exchanged, such as between a server and a client device or other types of devices, including between wireless devices coupled via a wireless network, for example. A network may also include mass storage, such as network attached storage (NAS), a storage area network (SAN), a content delivery network (CDN) or other forms of computer or machine-readable media, for example. A network may include the Internet, one or more local area networks (LANs), one or more wide area networks (WANs), wire-line type connections, wireless type connections, cellular or any combination thereof. Likewise, sub-networks, which may employ differing architectures or may be compliant or compatible with differing protocols, may interoperate within a larger network.

[0041] For purposes of this disclosure, a “wireless network” should be understood to couple client devices with a network. A wireless network may employ stand-alone ad-hoc networks, mesh networks, Wireless LAN (WLAN) networks, cellular networks, or the like. A wireless network may further employ a plurality of network access technologies, including Wi-Fi, Long Term Evolution (LTE), WLAN, Wireless Router (WR) mesh, or 2nd, 3rd, 4th or 5th generation (2G, 3G, 4G or 5G) cellular technology, mobile edge computing (MEC), Bluetooth, 802.11b / g / n, or the like. Network access technologies may enable wide area coverage for devices, such as client devices with varying degrees of mobility, for example. In short, a wireless network may include virtually any type of wireless communication mechanism byAttorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 which signals may be communicated between devices, such as a client device or a computing device, between or within a network, or the like.

[0042] A computing device may be capable of sending or receiving signals, such as via a wired or wireless network, or may be capable of processing or storing signals, such as in memory as physical memory states, and may, therefore, operate as a server. Thus, devices capable of operating as a server may include, as examples, dedicated rack-mounted servers, desktop computers, laptop computers, set top boxes, integrated devices combining various features, such as two or more features of the foregoing devices, or the like.

[0043] For purposes of this disclosure, a client (or consumer or user) device, referred to as user equipment (UE)), may include a computing device capable of sending or receiving signals, such as via a wired or a wireless network. A client device may, for example, include a desktop computer or a portable device, such as a cellular telephone, a smart phone, a display pager, a radio frequency (RF) device, an infrared (IR) device a Near Field Communication (NFC) device, a Personal Digital Assistant (PDA), a handheld computer, a tablet computer, a phablet, a laptop computer, a set top box, a wearable computer, smart watch, an integrated or distributed device combining various features, such as features of the forgoing devices, or the like.

[0044] In some embodiments, as discussed below, the client device can also be, or can communicatively be coupled to, any type of known or to be known medical device (e.g., any type of Class I, II or III medical device), such as, but not limited to, a MRI machine, CT scanner, Electrocardiogram (ECG or EKG) device, photopletismograph (PPG), Doppler and transmit-time flow meter, laser Doppler, an endoscopic device neuromodulation device, a neurostimulation device, and the like, or some combination thereof.DETAILED DESCRIPTION

[0045] The present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, which form a part hereof, and which show, by way of non-limiting illustration, certain example embodiments. Subject matter may, however, be embodied in a variety of different forms and, therefore, covered orAttorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 claimed subject matter is intended to be construed as not being limited to any example embodiments set forth herein; example embodiments are provided merely to be illustrative. Likewise, a reasonably broad scope for claimed or covered subject matter is intended. Among other things, for example, subject matter may be embodied as methods, devices, components, or systems. Accordingly, embodiments may, for example, take the form of hardware, software, firmware or any combination thereof (other than software per se). The following detailed description is, therefore, not intended to be taken in a limiting sense.

[0046] Throughout the specification and claims, terms may have nuanced meanings suggested or implied in context beyond an explicitly stated meaning. Likewise, the phrase “in one embodiment” as used herein does not necessarily refer to the same embodiment and the phrase “in another embodiment” as used herein does not necessarily refer to a different embodiment. It is intended, for example, that claimed subject matter include combinations of example embodiments in whole or in part.

[0047] The present disclosure is described below with reference to block diagrams and operational illustrations of methods and devices. It is understood that each block of the block diagrams or operational illustrations, and combinations of blocks in the block diagrams or operational illustrations, can be implemented by means of analog or digital hardware and computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer to alter its function as detailed herein, a special purpose computer, ASIC, or other programmable data processing apparatus, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, implement the functions / acts specified in the block diagrams or operational block or blocks. In some alternate implementations, the functions / acts noted in the blocks can occur out of the order noted in the operational illustrations. For example, two blocks shown in succession can in fact be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functional ity / acts involved.

[0048] Various examples are directed to methods and systems of registering a three-dimensional (3D) model of a rigid structure, such as bone. More particularly,Attorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 various examples are directed to methods and related systems for identifying surface features of a rigid structure visible in a video stream, and using the surface features to register a three-dimensional model for use in computer-assisted navigation of the surgical procedure (e.g., Computer-Aided Surgery, or CAS). In some examples, the surface features are determined using touchless techniques based on a known or calculated motion of the camera. In other examples, the surface features are gathered using a touch probe that is not itself directly tracked; rather, the pose of the touch probe, and thus the locations of the distal tip of the touch probe touching the bone, may be determined by segmenting the frames of the video stream and pose estimation. In yet still further examples, the three-dimensional model may be registered by use of a patient-specific instrument that couples to the rigid structure in only one orientation; thus, a fiducial coupled to the patient-specific instrument, or in some cases the patient-specific instrument itself without a fiducial, may be used to register a three-dimensional anatomical (e.g., bone) model.

[0049] Video-based surgical navigation (VBSN) techniques that use patient-specific instruments may include using visual fiducials or fiducial markers (also called visual markers) attached to patient anatomy to guide the surgeon throughout the medical procedure. The video-based navigation process requires precise registration of a pre-operative anatomical model with data acquired intra-operatively. The registration process or procedure requires the surgeon to digitize the surface of interest that corresponds to the pre-operative model. The fiducial markers with known visual patterns that are attached to the anatomies define reference frames to which the preoperative model and the intra-operative acquired data are aligned. The fiducial markers may be attached both to the targeted anatomy and to the instruments and subsequently tracked such that their relative poses can be accurately estimated (e.g., by applying 3D computer vision methods on the images / video acquired by a camera). These relative poses allow the instruments to be located with respect to the anatomy at every frame time instant. For example, VBSN facilitates the tracking of instruments with respect to the targeted anatomy to which a fiducial is rigidly attached (which may be referred to as a “base marker”).

[0050] In some examples of a registration procedure, a user probes an anatomical surface using a handheld probe. Collected points (e.g., a point cloud) are processedAttorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 (e.g., using a machine learning algorithm) and matched to a bone model, such as a bone model created via a scan (e.g., a CT or MRI scan) or other technique. For example, the bone model is overlaid on top of a live arthroscopic video feed to provide an augmented or mixed reality visual representation of a surgical or anatomical site.

[0051] In this manner, by using a probe with a calibrated tip (i.e., a probe that is calibrated such that the 3D position of the tip of the probe relative to the reference frame of the tip is known), 3D points can be intra-operatively reconstructed in the reference frame of the fiducial marker. Once the tip is in contact with the surface of patient anatomy, the set of reconstructed 3D points is a representation of the anatomy of interest that is provided as input to the registration procedure.

[0052] Some procedures, such as arthroscopy procedures, are limited by techniques and tooling that fit within the environment of the joint. Typically, joint spaces / environments are small, which limits what methods and hardware can be used for registration and other VBSN techniques. Limitations on procedures and instrumentation are determined by introductory portals into the joint spaces (i.e., portals located / positioned and formed for receiving probes, arthroscopes, etc.). Accordingly, existing tools / instruments are restricted in size by the dimensions and other characteristics of the portals and joints, resulting in the use of rulers and probes that are narrowly shaped and that can fit into the joint space. Further, these tools have units of measurements that are limited by the length, width, and / or other dimensions of the tool.

[0053] Systems and methods according to the principles of the present disclosure use VBSN and alternate / augmented reality (AR) techniques to render, display, or project (which may be used interchangeably in the present disclosure) objects in joint or other surgical spaces that would be challenging or not possible with a physical tool. In an example, a surgical tool includes one or more visual tracking mechanisms such as one or more fiducials, a QR code or pattern, etc. In some examples, a fixed fiducial (e.g., a bone fiducial) can also be used.

[0054] The visual tracking mechanism is used to provide a virtual, digital rendering or overlay of a digital template or object (e.g., a virtual model, element, etc.) on a live video feed. The digital template is positioned in accordance with a tip, axis (e.g.,Attorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 longitudinal axis), and / or other geometric feature of the tool. For example, markers or markings on the tool, such as fiducial markers (with or without additional positioning data that can be obtained from bone or other fixed-location fiducials), can be used to locate the tip, longitudinal axis, or other geometric feature of the tool and digital overlays (e.g., ruler elements, shapes, or other virtual models) can be projected / overlaid on a live video feed accordingly, with or without numerical measurements.

[0055] For example, the digital template may include a ruler, an object including a ruler or ruler / measurement markings, bone plugs or anchors, etc. As one example, the digital template or overlay may include one or more types of rulers or ruler elements, such as a linear (e.g., “standard”) ruler, a protractor ruler, a full-circle protractor ruler (e.g., a “dartboard” ruler), etc. As another example, the digital template or overlay may include other projected shapes, patterns, or objects, such as a cylinder or tube, sphere, planar cross-section of a cylinder (e.g., a circular or elliptical shape), etc. The projected shapes can be aligned with various geometric features of the tool (e.g. aligned with an axis of the tool) and / or offset from geometric features of the tool. The ruler elements and projected shapes can have different possible projection positions, such as colinear with the long axis of the tool, offset colinear (e.g., offset but parallel) with the long axis, offset and non-colinear with the long axis, and so on based on type of procedure or specific surgical steps, user selections or inputs, and so on.

[0056] Various examples described herein relate to anterior-cruciate ligament (ACL) repair (e.g., for placing femoral and / or tibial tunnels during ACL reconstruction), and thus the discussion below is based on the developmental context. In this context, the rigid structure is bone and / or cartilage, and the three-dimensional model is a three-dimensional bone model. However, the techniques described herein are applicable to any suitable rigid anatomical structure, such as teeth. Moreover, the various techniques may be applicable to many types of surgical procedures, such as repairs associated with the knee, the hip, the shoulder, the wrist, or the ankle. The techniques may be applicable not only to ligament repair (e.g., medial collateral ligament repair, lateral collateral ligament repair, and posterior cruciate ligament repair), but also for planning and placing anchors to reattach soft tissue (e.g.,Attorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 reattaching the labrum of the hip, the rotator cuff, or the meniscal root), and surgical procedures to address femoroacetabular impingement. Thus, the description and developmental context shall not be read as a limitation of the applicability of the teachings.

[0057] FIG. 1 shows a surgical system (not to scale) 100 in accordance with at least some embodiments. In particular, the example surgical system 100 comprises a tower or device cart 102, an example mechanical resection instrument 104, an example plasma-based ablation instrument (hereafter just ablation instrument 106), and an endoscope in the example form of an arthroscope 108 and attached camera head or camera 110. In the example systems, the arthroscope 108 is a rigid device, unlike endoscopes for other procedures, such as upper-endoscopies. The device cart 102 may comprise a display device 414, a resection controller 116, and a camera control unit (CCU) together with an endoscopic light source and video controller 118. In example cases the combined CCU and video controller 118 not only provides light to the arthroscope 108 and displays images received from the camera 110, but also implements various additional aspects, such as registering a three-dimensional bone model with the bone visible in the video images, and providing computer-assisted navigation during the surgery. Thus, the combined CCU and video controller are hereafter referred to as surgical controller 118. In other cases, however, the CCU and video controller may be a separate and distinct system from the controller that handles registration and computer-assisted navigation, yet the separate devices would nevertheless be operationally coupled.

[0058] The example device cart 102 further includes a pump controller 122 (e.g., single or dual peristaltic pump). Fluidic connections of the mechanical resection instrument 104 and ablation instrument 106 to the pump controller 122 are not shown so as not to unduly complicate the figure. Similarly, fluidic connections between the pump controller 122 and the patient are not shown so as not to unduly complicate the figure. In the example system, both the mechanical resection instrument 104 and the ablation instrument 106 are coupled to the resection controller 116 being a dual-function controller. In other cases, however, there may be a mechanical resection controller separate and distinct from an ablation controller. The example devices and controllers associated with the device cart 102Attorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 are merely examples, and other examples include vacuum pumps, patientpositioning systems, robotic arms holding various instruments, ultrasonic cutting devices and related controllers, patient-positioning controllers, and robotic surgical systems.

[0059] FIG. 1 further shows additional instruments that may be present during an arthroscopic surgical procedure. In particular, FIG. 2 shows an example touch probe 124, a drill guide or aimer 126, and a bone fiducial 128. The touch probe 124 may be used during the surgical procedure to provide information to the surgical controller 118, such as information to register a three-dimensional bone model to an underlying bone visible in images captured by the arthroscope 108 and camera head 110. The aimer 126 may be used as a guide for placement and drilling with a drill wire to create an initial or pilot tunnel through the bone. The bone fiducial 128 may be affixed or rigidly attached to the bone and serve as an anchor location for the surgical controller 118 to know the orientation of the bone (e.g., after registration of a three-dimensional bone model). Additional tools and instruments will be present, such as the drill wire, various reamers for creating the throughbore and counterbore aspects of a tunnel through the bone, and various tools, such as for suturing and anchoring a graft. These additional tools and instruments are not shown so as not to further complicate the figure. The specification now turns to a workflow for an example anterior cruciate ligament repair.

[0060] A surgical procedure may begin with a planning phase. The example anterior cruciate ligament repair may start with imaging (e.g., X-ray imaging, computed tomography (CT), magnetic resonance imaging (MRI)) of the knee of the patient, including the relevant anatomy like the lower portion of the femur, the upper portion of the tibia, and the articular cartilage. The imaging may be preoperative imaging, hours or days before the intraoperative repair, or the imaging may take place within the surgical setting just prior to the intraoperative repair. The discussion that follows assumes MRI imaging, but again many different types of imaging may be used. The image slices from the MRI imaging can be segmented such that a volumetric model or three-dimensional model of the anatomy is created. Any suitable currently available, or after developed, segmentation technology may be used to create the three-dimensional model. More specifically to the example ofAttorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 anterior cruciate ligament repair, a three-dimensional bone model of the lower portion of the femur, including the femoral condyles, is created.

[0061] Using the three-dimensional bone model, an operative plan is created that comprises choosing a planned-tunnel path through the femur, including locations of the apertures of the bone that define the ends of the tunnel. For an example inside-out repair, the aperture within the femoral notch is the entry location for the drilling, and the aperture on the lateral surface of the femur is the exit location. For an outside-in repair, the entry and exit locations for drilling are swapped. Still assuming an inside-out repair, the entry location may be selected to be the same as, or close to, the attachment location of the native anterior cruciate ligament to the femur within the femoral notch. In some cases, selecting the entry location within the femoral notch may involve use of a Bernard & Hertel Quadrant or grid placed on a fluoroscopic image, or by placing the Bernard & Hertel Quadrant on a simulated fluoroscopic image created from the three-dimensional bone model. Based on use of the Bernard & Hertel Quadrant, an entry location for the tunnel is selected. For an inside-out repair, selection of the exit location is less restrictive, not only because the portion of the tunnel proximate to the exit location is used for placement of the anchor for the graft, but also because the exit location is approximately centered in the femur (considered anteriorly to posteriorly), and thus issues of bone wall thickness at the exit location are of less concern. In some cases, a three-dimensional bone model of the proximal end of the tibia is also created, and the surgeon may likewise choose planned-tunnel path(s) through the tibia.

[0062] The results of the planning may include: a three-dimensional bone model of the distal end of the femur; a three-dimensional bone model for a proximal end of the tibia; an entry location and exit location through the femur and thus a planned-tunnel path for the femur; and an entry location and exit location through the tibia and thus a planned-tunnel path through the tibia. Other surgical parameters may also be selected during the planning, such as tunnel throughbore diameters, tunnel counterbore diameters and depth, desired post-repair flexion, and the like, but those additional surgical parameters are omitted so as not to unduly complicate the specification.Attorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201

[0063] The specification now turns to intraoperative aspects. The intraoperative aspects include steps and procedures for setting up the surgical system to perform the various repairs. It is noted, however, that some of the intraoperative aspects (e.g., optical system calibration), may take place before any ports or incisions are made through the patient’s skin, and in fact before the patient is wheeled into the surgical room. Nevertheless, such steps and procedures may be considered intraoperative as they take place in the surgical setting and with the surgical equipment and instruments used to perform the actual repair.

[0064] The example ACL repair is conducted arthroscopically and is computer-assisted in the sense the surgical controller 118 is used for arthroscopic navigation within the surgical site. More particularly, in example systems the surgical controller 118 provides computer-assisted navigation during the ligament repair by tracking location of various objects within the surgical site, such as the location of the bone within the three-dimensional coordinate space of the view of the arthroscope, and location of the various instruments (e.g., a drill wire) within the three-dimensional coordinate space of the view of the arthroscope. The specification turns to brief description of such tracking techniques.

[0065] FIG. 2 shows a conceptual drawing of a surgical site with various objects within the surgical site. In particular, visible in FIG. 2 is a distal end of the arthroscope 108, a portion of a bone 200 (e.g., femur), the bone fiducial 128 within the surgical site, and the touch probe 124. Each is addressed in turn.

[0066] The arthroscope 108 illuminates the surgical site with visible light. In the example of FIG. 2, the illumination is illustrated by arrows 208. The illumination provided to the surgical site is reflected by various objects and tissues within the surgical site, and the reflected light that returns to the distal end enters the arthroscope 108, propagates along an optical channel within the arthroscope 108, and is eventually incident upon a capture array within the camera 110 (FIG. 1). The images detected by the capture array within the camera 110 are sent electronically to the surgical controller 118 (FIG. 1) and displayed on the display device 114 (FIG.1). In one example, the arthroscope 108 is monocular or has a single optical path through the arthroscope for capturing images of the surgical site, notwithstanding that the single optical path may be constructed of two or more optical members (e.g.,Attorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 glass rods, optical fibers, etc.). That is to say, in example systems and methods the computer-assisted navigation provided by the arthroscope 108, the camera 110, and the surgical controller 118 is provided with the arthroscope 108 that is not a stereoscopic endoscope having two distinct optical paths separated by an interocular distance at the distal end endoscope.

[0067] During a surgical procedure, a surgeon selects an arthroscope with a viewing direction beneficial for the planned surgical procedure. Viewing direction refers to a line residing at the center of an angle subtended by the outside edges or peripheral edges of the view of an endoscope. The viewing direction for some arthroscopes is aligned with the longitudinal central axis of the arthroscope, and such arthroscopes are referred to as “zero degree” arthroscopes (e.g., the angle between the viewing direction and the longitudinal central axis of the arthroscope is zero degrees). The viewing direction of other arthroscopes forms a non-zero angle with the longitudinal central axis of the arthroscope. For example, for a 30° arthroscope the viewing direction forms a 30° angle to the longitudinal central axis of the arthroscope, the angle measured as an obtuse angle beyond the distal end of the arthroscope. In many cases for ACL repair, the surgeon selects a 30° arthroscope or a 45° arthroscope based on location the port created through the skin of the patient. In the example of FIG. 2, the view angle 210 of the arthroscope 108 forms a non-zero angle to the longitudinal central axis 212 of the arthroscope 108.

[0068] Still referring to FIG. 2, within the view of the arthroscope 108 is a portion of the bone 200 (e.g., within the intercondylar notch), along with the example bone fiducial 128 and the example touch probe 124. The example bone fiducial 128 is a multi-faceted element, with each face or facet having a fiducial disposed or created thereon. However, the bone fiducial need not have multiple faces, and in fact may take any shape so long as that shape, a fiducial or other visual marker, etc. can be tracked within the video images. The bone fiducial, such as bone fiducial 128, may be attached to the bone 200 in any suitable form. In this example, the bone fiducial is 128 fastened by a screw portion (not visible in FIG. 2, but visible in FIG. 1). The patterns of the fiducials on each facet are designed to provide information regarding the position and orientation of the bone fiducial 128 in the three-dimensional coordinate space of the view of the arthroscope 108. More particularly, the pattern isAttorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 selected such that the position and orientation of the bone fiducial 128 may be determined from images captured by the arthroscope 108 and attached camera (FIG. 1).

[0069] The touch probe 124 is also shown as partially visible within the view of the arthroscope 108. The touch probe 124 may be used, as discussed more below, to identify a plurality of surface features on the bone 200 as part of the registration of the bone 200 to the three-dimensional bone model. Alternatively, though not specifically shown, the aimer 126 (FIG. 1) may be used as the device to assist with the registration process. In some cases, the touch probe 124 and / or the aimer 126 may carry their own, unique fiducials, such that their respective poses may be calculated from the one or more fiducials present in the video stream. However, in other cases, and as shown, the medical instrument used to help with registration of the three-dimensional bone model, be it the touch probe 124, the aimer 126, or any other suitable medical device, may omit carrying fiducials. Stated otherwise, in such examples the medical instrument has no fiducial markings. In such cases, the pose of the medical instrument may be determined by a machine learning model, discussed in more detail below.

[0070] The images captured by the arthroscope 108 and attached camera are subject to optical distortion in many forms. For example, the visual field between a distal end of the arthroscope 108 and the bone 200 within the surgical site is filled with fluid, such as bodily fluids and saline used to distend the joint. Many arthroscopes have one or more lenses at the distal end that widen the field of view, and the wider field of view causes a “fish eye” effect in the captured images. Further, the optical elements within the arthroscope (e.g., rod lenses) may have optical aberrations inherent to the manufacturing and / or assembly process. Further still, the camera may have various optical elements for focusing the images received onto the capture array, and the various optical elements may have aberrations inherent to the manufacturing and / or assembly process. In example systems, prior to use within each surgical procedure, the endoscopic optical system is calibrated to account for the various optical distortions. The calibration creates a characterization function that characterizes the optical distortion, and further analysis of the frames ofAttorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 the video stream may be, prior to further analysis, compensated using the characterization function.

[0071] The next example step in the intraoperative procedure is the registration of the bone model created during the planning stage. During the intraoperative repair, the three-dimensional bone model is obtained by or provided to the surgical controller 118. Again using the example of anterior cruciate ligament repair, and specifically computer-assisted navigation for tunnel paths through the femur, the three-dimensional bone model of the lower portion of the femur is obtained by or provided to the surgical controller 118. Thus, the surgical controller 118 receives the three-dimensional bone model, and assuming the arthroscope 108 is inserted into the knee by way of a port through the patient’s skin, the surgical controller 118 also receives video images of a portion of the lower end of the femur. In order to relate the three-dimensional bone model to the images received by way of the arthroscope 108 and camera 110, the surgical controller 118 registers the three-dimensional bone model to the images of the femur received by way of the arthroscope 108 and camera 110.

[0072] In order to perform the registration, and in accordance with example methods, the bone fiducial 128 is attached to the femur. The bone fiducial placement is such that the bone fiducial is within the field of view of the arthroscope 108, but in a location spaced apart from the expected tunnel entry / exit point through the lateral condyle. More particularly, in example cases the bone fiducial 128 is placed within the intercondylar notch superior to the expected location of the tunnel through lateral condyle. To relate or register bone visible in the video images to the three-dimensional bone model, the surgical controller 118 (FIG. 1) is provided or determines a plurality of surface features of an outer surface of the bone. Identifying the surface features may take several forms, including a touch-based registration using the touch probe 124 without a carried fiducial, a touchless registration technique in which the surface features are identified after resolving the motion of the arthroscope 108 and camera relative to the bone fiducial 128, and a third technique in which uses a patient-specific instrument.

[0073] In the example touch-based registration, the surgeon may touch a plurality of locations using the touch probe 124 (FIG. 1). In some cases, particularly whenAttorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 portions of the outer surface of the bone are exposed to view, receiving the plurality of surface features of the outer surface of the bone may involve the surgeon “painting” the outer surface of the bone. “Painting” is a term of art that does not involve application of color or pigment, but instead implies motion of the touch probe 124 when the distal end of the touch probe 124 is touching bone. In this example, the touch probe 124 does not carry or have a fiducial visible to the arthroscope 108 and the camera 110. It follows that the pose of the touch probe 124 and the location of the distal tip of the touch probe 124 needs to be determined in order to gather the surface features for purposes of registering the three-dimensional bone model.

[0074] FIG. 3 shows a method 300 in accordance with at least some embodiments. The example method 300 may be implemented in software within a computer system, such as the surgical controller 118. In particular, the example method 300 comprises obtaining a three-dimensional bone model (block 302). That is to say, in the example method 300, what is obtained is the three-dimensional bone model that may be created by segmenting a plurality of non-invasive images (e.g., CT, MRI) taken preoperatively or intraoperatively. With the bone segmented from or within the images, the three-dimensional bone model may be created. The three-dimensional bone may take any suitable form, such as a computer-aided design (CAD) model, a point cloud of data points with respect to an arbitrary origin, or a parametric representation of a surface expressed using analytical mathematical equations. Thus, the three-dimensional bone model is defined with respect to the origin and in any suitable an orthogonal basis.

[0075] The next step in the example method 300 is capturing video images of the bone fiducial attached to the bone (block 304). The capturing is performed intraoperatively. In the example case of an arthroscopic anterior cruciate ligament repair, the capturing of video images is by way of the arthroscope 108 and camera 110. Other endoscopes may be used, such as endoscopes in which the capture array resides at the distal end of the device (e.g., chip-on-the-tip devices). However, in open procedures where the skin is cut and pulled away, exposing the bone to the open air, the capturing may be by any suitable camera device, such as one or both cameras of a stereoscopic camera system, or a portable computing device, such asAttorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 a tablet or smart-phone device. The video images may be provided to the surgical controller 118 in any suitable form.

[0076] The next step in the example method 300 is determining locations of a distal tip of the medical instrument visible within the video images (block 306), where the distal tip is touching the bone in at least some of the frames of the video images, and the medical instrument does not have a fiducial. Determining the locations of the distal tip of the medical instrument may take any suitable form. In one example, determining the locations may include segmenting the medical instrument in the frames of the video images (block 308). The segmenting may take any suitable form, such as applying the video images to a segmentation machine learning algorithm. The segmentation machine learning algorithm may take any suitable form, such as neural network or convolution neural network trained with a training data set showing the medical instrument in a plurality of known orientations. The segmentation machine learning algorithm may produce segmented video images where the medical instrument is identified or highlighted in some way (e.g., box, brightness increased, a highlighted outline, other objects removed, etc.).

[0077] With the segmented video images, the example method 300 may estimate a plurality of poses of the medical instrument within a respective plurality of frames of the video images (block 310). The estimating the poses may take any suitable form, such as applying the video images to a pose machine learning algorithm. The pose machine learning algorithm may take any suitable form, such as neural network or convolution neural network trained to perform six-dimensional pose estimation. The resultant of the pose machine learning algorithm may be, for at least some of the frames of the video image, an estimated pose of the medical instrument in the reference frame of the video images and / or in the reference frame provided by the bone fiducial. That is, the result of the pose machine learning algorithm may be a plurality of poses, one pose each for at least some of the frames of the segmented video images. While in many cases a pose may be determined for each frame, in other cases it may not be possible to make a pose estimation for at least some frame because of video quality issues, such as motion blur caused by electronic shutter operation.Attorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201

[0078] The next step in the example method 300 is determining the locations based on the plurality of poses (block 312). In particular, for each frame for which a pose can be estimated, based on a model of the medical device the location of the distal tip can be determined in the reference frame of the video images and / or the bone fiducial. Thus, the resultant is a set of locations that, at least some of which, represent locations of the outer surface of the bone.

[0079] FIG. 3 shows an example three-step process for determining the locations of the distal tip of the medical instrument. However, the method 300 is merely an example, and many variations are possible. For example, a single machine learning model, such as a convolution neural network, may be set up and trained to perform all three steps as a single overall process, though there may be many hidden layers of the convolution neural network. That is, the convolution neural network may segment the medical instrument, perform the six-dimensional pose estimation, and determine the location of the distal tip in each frame. The training data set in such a situation would include a data set in which each frame has the medical device segmented, the six-dimensional pose identified, and the location of the distal tip identified. The output of the determining step 306 may be a segmented video stream distinct from the video images captured at step 304. In such cases, the later method steps may use both segmented video stream and the video images to perform the further tasks. In other cases, the location information may be combined with the video images, such as being embedded in the video images, or added as metadata to each frame of the video images.

[0080] FIG. 4 is an example video display showing portions of a femur and a bone fiducial during a registration procedure. The display may be shown, for example, on the display device 114 associated with the device cart 102, or any other suitable location. In particular, visible in the main part of the display of FIG. 4 is an intercondylar notch 400, a portion of the lateral condyle 402, a portion the medial condyle 404, and the example bone fiducial 128. Shown in the upper right corner of the example display is a depiction of the bone, which may be a rendering 406 of the bone created from the three-dimensional bone model. Shown on the rendering 406 is a recommended area 408, the recommended area 408 being portions of the surface of the bone to be “painted” as part of the registration process. Shown in theAttorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 lower right corner of the example display is a depiction of the bone, which again may be a rendering 412 of the bone created from the three-dimensional bone model. Shown on the rendering 412 are a plurality of surface features 416 on the bone model that have been identified as part of the registration process. Further shown in the lower right corner of the example display is progress indicator 418, showing the progress of providing and receiving of locations on the bone. The example progress indicator 418 is a horizontal bar having a length that is proportional to a level of confidence that the registration is accurate, but any suitable graphic or numerical display showing progress may be used (e.g., 0% to 100%).

[0081] Referring to both the main display and the lower right rendering, as the surgeon touches the outer surface of the bone within the images captured by the arthroscope 108 and camera 110, the surgical controller 118 receives the surface features on the bone, and may display each location both within the main display as dots or locations 416, and within the rendering shown in the lower right corner. More specifically, the example surgical controller 118 overlays indications of identified surface features 416 on the display of the images captured by the arthroscope 108 and camera 110, and in the example case shown, also overlays indications of identified surface features 416 on the rendering 412 of the bone model. Moreover, as the number of identified locations 416 increases, the surgical controller 118 also updates the progress indicator 418.

[0082] Returning to FIG. 3, the plurality of surface features 416 may be, or the example surgical controller 118 may generate, a registration model relative to the bone fiducial 128 (block 314). The registration model may take any suitable form, such as a computer-aided design (CAD) model or point cloud of data points in any suitable orthogonal basis. The registration model, regardless of the form, may have fewer overall data points or less “structure” than the bone model created by the non-invasive computer imaging (e.g., MRI). However, the goal of the registration model is to provide the basis for the coordinate transforms and scaling used to correlate the bone model to the registration model and relative to the bone fiducial 128. Thus, the next step in the example method 300 is registering the bone model relative to the location of the bone fiducial based on the registration model (block 316). Registration may conceptually involve testing a plurality of coordinateAttorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 transformations and scaling values to find a correlation that has a sufficiently high correlation or confidence factor. Once a correlation is found with the sufficiently high confidence factor, the bone model is said to be registered to the location of the bone fiducial. Thereafter, the example registration method 300 may end; however, the surgical controller 118 may then use the registered bone model to provide computer-assisted navigation regarding a procedure involving the bone.

[0083] In the examples discussed to this point, registration of the bone model involves a touch-based registration technique using the touch probe 124 without a carried fiducial. However, other registration techniques are possible, such as a touchless registration technique. The example touchless registration technique again relies on placement of the bone fiducial 128. As before, when the viewing direction of the arthroscope 108 is relatively constant, the bone fiducial may have fewer faces with respective fiducials. Once placed, the bone fiducial 128 represents a fixed location on the outer surface of the bone in the view of the arthroscope 108, even as the position of the arthroscope 108 is moved and changed relative to the bone fiducial 128. Again, in order to relate or register the bone visible in the video images to the three-dimensional bone model, the surgical controller 118 (FIG. 1) determines a plurality of surface features of an outer surface of the bone, and in this example determining the plurality of surface features is based on a touchless registration technique in which the surface features are identified based on motion of the arthroscope 108 and camera 110 relative to the bone fiducial 128.

[0084] Another technique for registering the bone model to the bone uses a patientspecific instrument. In both touch-based and touchless registration techniques, a registration model is created, and the registration model is used to register the bone model to the bone visible in the video images. Conceptually, the registration model is used to determine a coordinate transformation and scaling to align the bone model to the actual bone. However, if the orientation of the bone in the video images is known or can be determined, use of the registration model may be omitted, and instead the coordinate transformations and scaling may be calculated directly.

[0085] FIG. 5 shows a method 500 in accordance with at least some embodiments. The example method may be implemented in software within one or more computer systems, such as, in part, the surgical controller 118. In particular, the exampleAttorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 method 500 starts and comprises obtaining a three-dimensional bone model (block 502). Much like the prior techniques, in the patient-specific instrument registration technique what is obtained is the three-dimensional bone model that may be created by segmenting a plurality of non-invasive images (e.g., MRI) taken preoperatively or intraoperatively.

[0086] The next step in the example method 500 is generating a patient-specific instrument that has a feature designed to couple to the bone represented in the bone model in only one orientation (block 504). Generating the patient-specific instrument may first involve selecting a location at which the patient-specific instrument will attach. For example, a device or computer system may analyze the bone model and select the attachment location. In various examples, the attachment location may be a unique location in the sense that, if a patient-specific instrument is made to couple to the unique location, the patient-specific instrument will not couple to the bone at any other location. In the example case of an anterior cruciate ligament repair, the location selected may be at or near the upper or superior portion on the intercondylar notch. If the bone model shows another location with a unique feature, such as a bone spur or other raised or sunken surface anomaly, such a unique location may be selected as the attachment location for the patient-specific instrument.

[0087] Moreover, forming the patient-specific instrument may take any suitable form. In one example, a device or computer system may directly print, such as using a 3D printer, the patient-specific instrument. In other cases, the device or computer system may print a model of the attachment location, and the model may then become the mold for creating the patient-specific instrument. For example, the model may be the mold for an injection-molded plastic or casting technique. In some examples, the patient-specific instrument carries one or more fiducials, but as mentioned above, in other cases the patient-specific instrument may itself be tracked and thus carry no fiducials.

[0088] The next step in the example method 500 is coupling the patient-specific instrument to the bone, in some cases the patient-specific instrument having the fiducial coupled to an exterior surface (block 506). As previously mentioned, the attachment location for the patient-specific instrument is selected to be unique such that the patient-specific instrument couples to the bone in only one location and inAttorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 only one orientation. In the example case of an arthroscopic ACL repair, the patientspecific instrument may be inserted arthroscopically. That is, the attachment location may be selected such that a physical size of the patient-specific instrument enables insertion through the ports in the patient’s skin. In other case, the patientspecific instrument may be made or constructed of a flexible material that enables the patient-specific instrument to deform for insertion in the surgical site, yet return to the predetermined shape for coupling to the attachment location. However, in open procedures where the skin is cut and pulled away, exposing the bone to the open air, the patient-specific instrument may be a rigid device with fewer size restrictions.

[0089] The next step in the example method 500 is capturing video images of the patient-specific instrument (block 508). Here again, the capturing may be performed intraoperatively. In the example case of an arthroscopic anterior cruciate ligament repair, the capturing of video images is by the surgical controller 118 by way of arthroscope 108 and camera 110. However, in open procedures where the skin is cut and pulled away, exposing the bone to the open air, the capturing may be by any suitable camera device, such as one or both cameras of a stereoscopic camera systems, or a portable computing device, such as a tablet or smart-phone device. In such cases, the video images may be provided to the surgical controller 118 in any suitable form.

[0090] The next step in the example method 500 is registering the bone model based on the location of the patient-specific instrument (block 510). That is, given that the patient-specific instrument couples to the bone at only one location and in only one orientation, the location and orientation of the patient-specific instrument is directly related to the location and origination of the bone, and thus the coordinate transformations and scaling for the registration may be calculated directly. Thereafter, the example method 500 may end; however, the surgical controller 118 may then use the registered bone model to provide computer-assisted navigation regarding a surgical task or surgical procedure involving the bone.

[0091] For example, with the registered bone model the surgical controller 118 may provide guidance regarding a surgical task of a surgical procedure. The specific guidance is dependent upon the surgical procedure being performed and the stage of the surgical procedure. A non-exhaustive list of guidance comprises: changing aAttorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 drill path entry point; changing a drill path exit point; aligning an aimer along a planned drill path; showing location at which to cut and / or resect the bone; reaming the bone by a certain depth along a certain direction; placing a device (suture, anchor or other) at a certain location; placing a suture at a certain location; placing an anchor at a certain location; showing regions of the bone to touch and / or avoid; and identifying regions and / or landmarks of the anatomy. In yet still other cases, the guidance may include highlighting within a version of the video images displayed on a display device, which can be the arthroscopic display or a see-through display, or by communicating to a virtual reality device or a robotic tool.

[0092] In these and other examples of a registration procedure, a user probes an anatomical surface using a handheld probe. FIGS. 6A and 6B show example probes 600 and 602, respectively. In these examples, the probes 600, 602 include one or more fiducials or fiducial markers 604. Tips 608 of the probes 600, 602 can be calibrated such that locations of the tips 608 relative to the fiducials 604 are known. For example, the probes 600, 602 are calibrated, in a pre-operative calibration procedure, to obtain geometrical and dimensional data such that position or location, orientation, etc. of various features of the probes 600, 602, such as the tips 608, relative to the fiducials 604 are known. In this manner, 3D points can be intra-operatively reconstructed in the reference frame of the fiducial marker. For example, points can be obtained when the tip is in contact with the surface of patient anatomy (by determining the location of the tip based on the detected fiducials 604). Accordingly, the set of reconstructed 3D points is a representation of the anatomy of interest that is provided as input to the registration procedure.

[0093] FIGS. 7A and 7B show other example tools 700 and 702 (e.g., probes) that include measurement features. In these examples, the tools 700 and 702 include linear measurement markings (such as a linear scale 704 with labeled units of measurement, spaced markings 706, etc.). Accordingly, when inserted through a portal into a joint space and viewed by an arthroscopic camera, the tools 700 and 702 provide a visual indication of dimensions, relative distances, etc. within a surgical environment. However, tools 700 and 702 are restricted in size and configuration by the size of the portals and joint spaces, which may vary by procedure, patients. Accordingly, the tools 700 and 702 are narrowly shaped toAttorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 allow the tools 700 and 702 to fit into existing cannulas and joint spaces. The configuration and arrangement of the measurement markings, units of measurement, etc. are further limited by the dimensions (e.g., length and width) of the tools 700 and 702.

[0094] Surgical navigation systems and methods according to the principles of the present disclosure are configured to render / display digital templates or objects in joint or other surgical spaces that would be challenging or not possible with a physical tool (e.g., spaces that would be challenging or not possible for tools such as the tools 700 and 702). In an example, a surgical tool includes one or more visual tracking mechanisms such as one or more fiducials, a QR code or pattern, etc., such as the probes 600 and 602 shown in FIGS. 6A and 6B. In some examples, a fixed fiducial (e.g., a bone fiducial) can also be used. The tool can be tracked (e.g., using visual tracking techniques as described above in FIGS. 1-6) and a virtual, digital rendering or overlay of a digital template or object can be provided on a live video feed. In other words, the live video feed may include an image feed of the surgical environment, tool, etc., and the digital template is provided as an overlay on the video feed.

[0095] For example, locations of one or more geometric features of the tool (e.g., a tip, longitudinal axis, etc.) can be determined and tracked based on the determined locations of the fiducial markers on the tool (e.g., based on calibration data correlating locations and orientations of tool features relative to the fiducial markers). The virtual digital template or object can then be projected / displayed based on the determined locations of the geometric features of the tool. For example, the location of the digital template is mated / paired to the tool (e.g., in a coordinate space of the tool). Accordingly, as the tool is moved within the joint space / surgical environment, the projected digital template / object is also automatically moved / adjusted to provide visual guidance to the surgeon.

[0096] Although described herein with respect to fiducial markers on the tool, in some examples, rather than detecting a fiducial marker on the tool, the tool itself can be detected based on features of the tool (e.g., a shape or outline of the tool, distinct geometrical features of the tool, etc.). In some examples, a shape or outline of the tool can be displayed / projected as an overlay. In examples where the shape orAttorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 other geometric feature of the tool is used to detect the position / orientation of the tool, the shape or geometric feature can be used to determine the projected position of the digital template.

[0097] When used in combination with a fixed fiducial maker (e.g., a bone fiducial), the location of the digital template and / or the tool can be transferred to a coordinate space of the fixed fiducial marker. In other words, the digital template can be “stamped” into a fixed location relative to the fiducial marker and patient anatomy and displayed / overlayed independently of subsequent movement or removal of the tool. In some examples, the digital template can be stamped relative to another fixed feature (rather than to a fixed bone fiducial or other fiducial marker), such as an identified anatomical feature or landmark. In some examples, the fixed feature may correspond to a unique configuration of image-based features (e.g., defined by patient anatomy) obtained / visible when the arthroscopic camera views the anatomy / surgical environment from a specific angle.

[0098] In other examples, additional / secondary digital templates can be displayed relative to an initial or primary digital template. For example, a first digital template / model (e.g., a plane) may be defined and displayed based on the detected tool position as described herein, and a second digital template may be defined and displayed based on the first digital template (e.g., an object arranged on the plane). In other words, the second digital template is paired with or stamped to the first digital template. In still other examples, multiple digital templates can be displayed based on the detected tool (e.g., digital templates of both a standard ruler and a dartboard ruler, both a ruler and an implant, etc.).

[0099] While physical measurement markings (i.e., measurement markings on the actual tool) can be obscured or difficult to read during a procedure, the digital template according to the present disclosure can be provided at high resolution regardless of the condition of the image feed quality and surgical environment. Other objects that can be displayed include, but are not limited to, bone plugs, anchors, surgical patches (e.g., bio-inductive patches or implants, which can be represented as a three-dimensional rectangular shape / tem plate or box, a two-dimensional plane or other two-dimensional polygonal shape, a generally flat polyhedron, etc.), or other implants, which can be displayed / projected to assist sizingAttorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 and placement of hardware within patient anatomy. Such objects can be displayed along with other relevant features, such as suggested anchor locations (e.g., as markings on or adjacent to a displayed implant), measurement markings, etc.

[0100] FIGS. 8A and 8B show example digital templates of rulers displayed / projected based on a detected position of a tool (e.g., a probe 800) according to the present disclosure. As shown in FIG. 8A, the digital template includes a standard (e.g., linear) ruler 802 configured to provide linear measurements. As shown in FIG. 8B, the digital template includes a full-circle protractor ruler (e.g., a “dartboard” ruler) 804 configured to provide circumferential and / or radial measurements. In both examples, the probe 800 includes one or more fiducial markings 808. The position of the probe 800 is determined and tracked based on the detected fiducial markings 808 using the surgical navigation techniques described herein. For example, stored calibration data correlates the position / orientation of the fiducial markings 808 to one or more features of the probe 800, such as a longitudinal axis 812, a tip or pointer 814, etc.

[0101] In turn, display of the digital templates are correlated to the position of one or more features of the probe 800. For example, the projection of the standard ruler 802 and / or the dartboard ruler 804 may be displayed based on the determined position / orientation of the axis 812, the tip 814, etc. Accordingly, as the probe 800 is moved within the surgical environment (as captured by a camera and provided in an image feed), the display of the digital template is automatically adjusted.

[0102] As shown in these examples, the display of the digital template is not limited to a specific orientation relative to the probe 800 or the features of the probe 800. For example, as shown in FIG. 8A, the standard ruler 802 is not coaxial or otherwise aligned with the axis 812 of the probe 800. In other words, a longitudinal axis 818 of the standard ruler 802 is not parallel or coaxial with the axis 812. Accordingly, as described herein, ruler elements and other projected shapes can have different possible projection positions relative to the probe 800, such as colinear with the long or longitudinal axis 812 of the probe 800, offset colinear (e.g., offset but parallel) with the axis 812, offset and non-colinear with the axis 812, etc. based on type of procedure or specific surgical steps, user selections or inputs, and so on.Attorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201

[0103] In some examples, a fixed fiducial marker (e.g., a bone fiducial 820) can be installed in patient anatomy. As shown in FIGS. 8A and 8B, the digital template is mated / paired with the probe 800 (e.g., displayed / positioned within a coordinate space of the probe 800). When used in combination with the bone fiducial 820, the location of the digital template can be transferred to a coordinate space of the bone fiducial 820 (e.g., responsive to user input). In other words, the digital template can be “stamped” into a fixed location relative to the bone fiducial 802 and displayed / overlayed independently of subsequent movement or removal of the probe 800 as shown in FIGS. 8C and 8D.

[0104] In another example as shown in FIGS. 9A, 9B, and 9C, a digital overlay or template may include other projected shapes, patterns, or objects, such as a cylinder 900, that are aligned with or offset from various geometric features of a tool (e.g. a probe 902). Although shown as the cylinder 900, in other examples the digital template may include a tube, sphere, disc, cube, or other symmetrical or asymmetrical (irregular) shape. In various examples, the projected shape (e.g., the cylinder 900) may represent a bone or other tissue plug, an anchor or other implant, a projected socket or tunnel, etc. As shown in FIG. 9A, the cylinder 900 is colinear with the probe 902. As shown in FIG. 9B, the cylinder 900 is offset colinear (e.g., offset but parallel) with the probe 902. As shown in FIG. 9C, the cylinder 900 is offset from and non-colinear with the probe 902.

[0105] FIGS. 10A and 10B show another example implementation of the principles of the present disclosure. In this example, a cylinder 1000 is displayed / projected relative to an awl or punch 1002. The awl 1002 can be used to mark tissue, anchor points, etc. at specific radial distances from the awl 1002. For example, force can be applied to the awl 1002 to form indentations or other markings in tissue (e.g., to measure out an arc, such as an arc corresponding to a labral repair in a joint pace along a fixed set of distances radially from an awl tip 1004 as shown at 1008 in FIG.10B). In other examples, the awl 1002 or another tool can be used to form holes (e.g., for anchors or other implants).

[0106] In other examples, the digital template techniques of the present disclosure can be applied to other types of tools and instruments to display various patterns, shapes, etc. for different types of surgical procedures (e.g., an arthroscopic wand, aAttorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 coblation wand, etc. that includes one or more fiducial markers). For example, digital templates such as a radial or other pattern may be displayed relative to a coblation or ablation wand to facilitate marking of coblation or ablation locations on patient anatomy. The displayed or projected pattern can be varied based on type of procedure or use case.

[0107] FIGS. 11 A, 11 B, 11C, 11D, 11 E, and 11 F show example digital template patterns that can be displayed (e.g., relative to a detected position of a surgical wand or other instrument tool) in accordance with the principles of the present disclosure. FIG. 11A shows a linear pattern 1100 (e.g., a linear pattern with known distance markings, with or without unit of measurement labels). FIG. 11 B shows a cylinder or cylindrical projection 1104 (e.g., a cylinder with known diameter, depth, etc.). FIG.110 shows a radial circular pattern 1108. In FIG. 11C, the radial circular pattern 1108 has rings 1110 having regular or fixed spacing and known diameters. Conversely, in FIG. 11 D, the rings 1110 have varied spacing.

[0108] FIG. 11E shows a polygon shape or projection 1116 (in this example, a crescent or “C”-shaped projection). The polygon shape 1116 can be symmetrical or asymmetrical.

[0109] FIG. 11F shows an adjustable projected shape or patten, such as an adjustable linear pattern 1120. For example, the pattern 1120 may be adjustable, responsive to user inputs, context cues, etc., from a first configuration as shown at 1122 (e.g., a linear pattern centered on a point 1124, which may correspond to a feature of a tool, such as a tip of a probe) to a second configuration as shown at 1126 (e.g., a linear pattern extending in only one direction form the point 1124). Although shown with respect to the linear patter 1120, other types of patterns may be adjustable in a similar manner.

[0110] For example, any tool or instrument described herein, a tablet computer or other computing device, any component of the system 100, a display, touchscreen, or other user interface, etc. may include buttons or other input mechanisms or devices for receiving inputs (e.g., user inputs, software, context, or process cues or inputs, etc.) for adjusting the digital template. Adjusting the digital template may include, but is not limited to: changing a type of the digital template (e.g., changing between types of rulers, changing from a ruler to a cylinder or other shape, etc.);Attorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 changing parameters or dimensions of the digital template (e.g., changing a length or width of a linear model, changing a diameter or length / depth of a cylinder, changing ring or measurement marking spacing, etc.); changing displayed units of measurement; toggling measurement markings or labels on and off; changing a position or orientation of the digital template (e.g., rotating the digital template about an axis or pin point in various directions, changing the axis or pin point, changing an offset distance of the digital template relative to the tool, etc.); and so on.

[0111] In some examples, a tablet or other computing device can be used to input user-generated or customized digital templates or shapes. For example, FIG. 12 shows an example computing device 1200 including a touchscreen interface 1202. The touchscreen interface 1202 is configured to receive user inputs (e.g., via a finger, stylus, etc.) to draw / create a customized digital template 1208. The touchscreen interface 1202 may include a drawing grid 1210 or other reference elements to indicate dimensions. In some examples, stored drawings or templates can be retrieved or imported and further modified via the touchscreen interface 1202. The digital template 1208 can be displayed / drawn relative to a virtual tool 1212, a reference point (e.g., a tip 1214 of the tool 1212), etc.

[0112] As another example, customized digital templates can be drawn / generated within the patient anatomy or joint space. For example, a tip of an instrument, such as a probe, can be used to trace a shape of a desired digital template (e.g., by pressing a button or providing another input to start and stop recording of the location of the tip) and the digital template is automatically generated in accordance with the traced shape.

[0113] As another example, the principles of the present disclosure can be implemented with a second tracked tool or instrument within the surgical environment. For example, the digital template can be displayed relative to a first tool as described above. A second tool (e.g., a probe) can be used to manipulate / adjust the digital template (e.g., by inserting the second tool into the surgical environment, virtually contacting / engaging with the digital template (e.g., as provided on the display), and providing user inputs to adjust the digital template (e.g., to change the position of the digital template relative to the first tool, rotate the digital template, change the size of the digital template, etc.).Attorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201

[0114] FIG. 13 illustrates steps of an example method 1300 for performing digital template techniques in accordance with at least some embodiments. In an example, the method 1300 may include, or be performed subsequent to, performing a touchbased registration procedure using a probe. The method 1300 can be performed in conjunction with any of the other methods described herein. In an example, a computing device, one or more processors or processors, etc. are configured to perform the steps or functions of the method 1300.

[0115] At 1304, the method 1300 (optionally) includes receiving one or more inputs associated with a digital template to be displayed on an image feed. For example, the one or more inputs may indicate or include, but are not limited to, a type of digital template to be displayed, parameters or characteristics of the digital template, etc.

[0116] At 1308, the method 1300 includes receiving one or more images (e.g., an image feed) of a surgical environment. For example, the images may correspond to images obtained from an arthroscopic camera arranged within the surgical environment. The images may include patient anatomy.

[0117] At 1312, the method 1300 includes detecting, in the images, a surgical instrument arranged within the surgical environment. For example, detecting the surgical instrument may include detecting and identifying one or more fiducial markers located on the surgical instrument.

[0118] At 1316, the method 1300 includes determining a position of the surgical instrument relative to the surgical environment (e.g., patient anatomy). For example, the position of the surgical instrument may be determined based on a position / orientation of the detected fiducial markers, calibration data associating the position of the fiducial markers to features of the surgical instrument (e.g., an axis, tip, etc. of the surgical instrument), and so on.

[0119] At 1320, the method 1300 includes displaying (e.g., as an overlay) a digital template on one or more images, an image feed, etc. provided on a display. For example, displaying the digital template includes displaying the digital template in a position / orientation within the surgical environment based on the detected position of the surgical instrument, the one or more inputs received at 1304, etc. In various examples, the digital template may include a ruler, shape, pattern, etc. as described herein.Attorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201

[0120] At 1324, the method 1300 (optionally) includes receiving adjustment inputs. The adjustment inputs may include, but are not limited to, inputs associated with a type of digital template, parameters or characteristics (e.g., dimensions) of the digital template, and so on.

[0121] At 1328, the method 1300 includes adjust the display of the digital template. Adjusting the display of the digital template may include, but is not limited to, adjusting the position / orientation of the digital template responsive to movement of the surgical instrument, adjusting the digital template response to inputs received at 1324, etc.

[0122] In some examples, the method 1300 may include receiving an input to cause the digital template to be stamped or otherwise transferred to a fixed coordinate space, such as stamping the digital template to a position or location in a coordinate space of a bone fiducial or other fixed feature.

[0123] FIG. 14 shows an example computer system 1400. In one example, the computer system 1400 may correspond to the surgical controller 118, a tablet device within the surgical room, or any other system that implements any or all the various methods discussed in this specification. The computer system 1400 may be connected (e.g., networked) to other computer systems in a local-area network (LAN), an intranet, and / or an extranet (e.g., device cart 102 network), or at certain times the Internet (e.g., when not in use in a surgical procedure). The computer system 1400 may be a server, a personal computer (PC), a tablet computer or any device capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that device. Further, while only a single computer system is illustrated, the term “computer” shall also be taken to include any collection of computers that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein.

[0124] The computer system 1400 includes a processing device 1402, a main memory 1404 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM)), a static memory 1406 (e.g., flash memory, static random access memory (SRAM)), and a data storage device 1408, which communicate with each other via a bus 1410.Attorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201

[0125] Processing device 1402 represents one or more general-purpose processing devices such as a microprocessor, central processing unit, or the like. More particularly, the processing device 1402 may be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or processors implementing a combination of instruction sets. The processing device 1402 may also be one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. The processing device 1402 is configured to execute instructions for performing any of the operations and steps discussed herein. Once programmed with specific instructions, the processing device 1402, and thus the entire computer system 1400, becomes a special-purpose device, such as the surgical controller 118.

[0126] The computer system 1400 may further include a network interface device 1412 for communicating with any suitable network (e.g., the device cart 102 network). The computer system 1400 also may include a video display 1414 (e.g., display device 414), one or more input devices 1416 (e.g., a microphone, a keyboard, and / or a mouse), and one or more speakers 1418. In one illustrative example, the video display 1414 and the input device(s) 1416 may be combined into a single component or device (e.g., an LCD touch screen).

[0127] The data storage device 1408 may include a computer-readable storage medium 1420 on which the instructions 1422 (e.g., implementing any methods and any functions performed by any device and / or component depicted described herein) embodying any one or more of the methodologies or functions described herein is stored. The instructions 1422 may also reside, completely or at least partially, within the main memory 1404 and / or within the processing device 1402 during execution thereof by the computer system 1400. As such, the main memory 1404 and the processing device 1402 also constitute computer-readable media. In certain cases, the instructions 1422 may further be transmitted or received over a network via the network interface device 1412.Attorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201

[0128] While the computer-readable storage medium 1420 is shown in the illustrative examples to be a single medium, the term “computer-readable storage medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store the one or more sets of instructions. The term “computer-readable storage medium” shall also be taken to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present disclosure. The term “computer-readable storage medium” shall accordingly be taken to include, but not be limited to, solid-state memories, optical media, and magnetic media.

[0129] The computer system 1400 or one or more computing or processing devices may be configured to perform functions of the procedures described herein, including functions related to communication and / or control of any of the probes, functions, steps, etc. described in FIGS. 7-13, and / or functions of the methods described herein.

[0130] The foregoing description is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses. The broad teachings of the disclosure can be implemented in a variety of forms. Therefore, while this disclosure includes particular examples, the true scope of the disclosure should not be so limited since other modifications will become apparent upon a study of the drawings, the specification, and the following claims. It should be understood that one or more steps within a method may be executed in different order (or concurrently) without altering the principles of the present disclosure. Further, although each of the embodiments is described above as having certain features, any one or more of those features described with respect to any embodiment of the disclosure can be implemented in and / or combined with features of any of the other embodiments, even if that combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and permutations of one or more embodiments with one another remain within the scope of this disclosure.

Claims

Attorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201CLAIMSWhat is claimed is:

1. A processor configured to execute instructions stored in memory to control generation and placement of a digital template in a display of patient anatomy, wherein executing the instructions causes the processor to:detect a visual marker of a surgical tool in an image feed of the patient anatomy;determine, based on the visual marker, at least one of a first location and a first orientation of one or more features of the surgical tool relative to the patient anatomy; andgenerate the digital template based on the at least one of the first location and the first orientation of the one or more features of the surgical tool, wherein generating the digital template includes rendering, in the display, the digital template in a second location and a second orientation relative to the surgical tool, and wherein a first longitudinal axis of the digital template as rendered in the display is at least one of (i) parallel but not colinear with a second longitudinal axis of the surgical tool and (ii) not parallel with the second longitudinal axis.

2. The processor of claim 1, wherein the digital template includes numerical measurements indicating distances from the surgical tool in at least one direction not parallel to the second longitudinal axis of the surgical tool.

3. The processor of claim 1, wherein the digital template includes two or more concentric rings indicating radial distances from the surgical tool.

4. The processor of claim 1 , wherein the first longitudinal axis is parallel to the second longitudinal axis and offset from the second longitudinal axis in a lateral direction.

5. The processor of claim 1 , wherein the digital template includes one of a linear ruler and a full-circle protractor ruler.Attorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-262016. The processor of claim 1, wherein the digital template includes a cylinder.

7. The processor of claim 1 , wherein the digital template includes one of a rectangular box, a circular shape, and an elliptical shape.

8. The processor of claim 1, wherein the digital template has a crescent shape.

9. The processor of claim 1, wherein the image feed includes a fixed fiducial marker arranged in the patient anatomy.

10. The processor of claim 9, wherein executing the instructions further causes the processor to transfer the digital template from a first coordinate space of the surgical tool to a second coordinate space of the fixed fiducial marker.

11. The processor of claim 10, wherein executing the instructions further causes the processor to cause the digital template to be displayed independent of the surgical tool.

12. A method for controlling generation and placement of a digital template in a display of patient anatomy, the method comprising:detecting a visual marker of a surgical tool in an image feed of the patient anatomy;determining, based on the visual marker, at least one of a first location and a first orientation of one or more features of the surgical tool relative to the patient anatomy; andAttorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-26201 generating the digital template based on the at least one of the first location and the first orientation of the one or more features of the surgical tool, wherein generating the digital template includes rendering, in the display, the digital template in a second location and a second orientation relative to the surgical tool, and wherein a first longitudinal axis of the digital template as rendered in the display is at least one of (i) parallel but not colinear with a second longitudinal axis of the surgical tool and (ii) not parallel with the second longitudinal axis.

13. The method of claim 12, wherein the digital template includes numerical measurements indicating distances from the surgical tool in at least one direction not parallel to the second longitudinal axis of the surgical tool.

14. The method of claim 12, wherein the digital template includes two or more concentric rings indicating radial distances from the surgical tool.

15. The method of claim 12, wherein the first longitudinal axis is parallel to the second longitudinal axis and offset from the second longitudinal axis in a lateral direction.

16. The method of claim 12, wherein the digital template includes one of a linear ruler and a full-circle protractor ruler.

17. The method of claim 12, wherein the digital template includes a cylinder.

18. The method of claim 12, wherein the digital template includes a rectangular box.

19. The method of claim 12, wherein the digital template has a crescent shape.

20. The method of claim 12, wherein the image feed includes a fixed fiducial marker arranged in the patient anatomy.Attorney Docket No. PT-6213-WO-PCT DW Ref. No. 73888-2620121. The method of claim 20, further comprising transferring the digital template from a first coordinate space of the surgical tool to a second coordinate space of the fixed fiducial marker.

22. The method of claim 21 , further comprising causing the digital template to be displayed independent of the surgical tool.

Images