Instrument energy adjustment in surgical environments

Abstract

The present disclosure describes a computer system and method for adjusting the energy output of medical instruments in surgical environments. The computer system includes one or more memories and one or more processors communicatively coupled to the one or more memories. The one or more processors, individually or collectively, perform an operation that includes applying a first amount of energy to an anatomical structure using a medical instrument and determining, based on a first set of images of the anatomical structure, a treatment level of the anatomical structure caused by applying the first amount of energy to the anatomical structure using the medical instrument. The operation also includes adjusting, based on the treatment level, an energy output of the medical instrument such that the medical instrument applies a second amount of energy to the anatomical structure.

Description

Attorney Docket No. P07031-WO (160618)INSTRUMENT ENERGY ADJUSTMENT IN SURGICAL ENVIRONMENTSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims benefit of co-pending United States provisional patent application Serial No. 63 / 750,435 filed January 28, 2025. The aforementioned related patent application is herein incorporated by reference in its entirety.TECHNICAL FIELD

[0002] The present disclosure relates generally to medical systems (e.g., digital fiducial systems, anatomy detection systems, clinical guidance systems, and surgical systems). Specifically, the present disclosure relates to adjusting the energy output of medical instruments in surgical environments.BACKGROUND

[0003] Doctors use computer assisted medical systems to perform different medical tasks. For example, doctors may use computer assisted surgical systems to perform operations on patients, even remotely. During the operations, doctors may use medical instruments that apply energy (e.g., electrical energy, heat energy, ultrasound energy, light energy, radiofrequency energy, plasma energy, etc.) to tissue. There is wide variability in the medical instruments. For example, different medical instruments may output different energy types and levels. Some medical instruments output energy for cutting, while other medical instruments output energy for sealing or coagulation.

[0004] The medical instruments may also be controlled to vary the amount of energy output by the medical instruments. During operations, the energy output of a medical instrument may need to be adjusted to address or handle variabilities encountered in the surgical environment. As an example, the medical instrument may need to apply energy to different types of tissue with different consistencies, thicknesses, or densities. Tissue with lower density (e.g., fascia, omentum, tubular vessels, ureters, etc.) may be more susceptible to damage and so lower energy should be applied to such tissue, while tissue with higher density (e.g., solid organs) are less susceptible to damage and may need more energy application to cut or seal. As another example, the medical instrument may need to apply different amounts ofAttorney Docket No. P07031-WO (160618)energy during different surgical tasks. Some tasks (e.g., dissection in hernia cases) may require slow and careful energy application, while other tasks (e.g., dissection of omentum in colorectal cases) may not require as slow or careful of energy application. Different movements or strokes of the medical instrument may also benefit from different amounts of energy output. For example, it may be beneficial to use different energy levels for short strokes as opposed to long strokes and for sideways strokes as opposed to vertical strokes.

[0005] Moreover, it may be difficult for doctors, especially newer doctors, to understand what energy levels to use when using new or unfamiliar medical instruments or to address different surgical conditions (e.g., different BMI, different organ shapes, different organ sizes, etc.). As a result, the doctors may set the medical instruments to apply too much or too little energy for the medical procedure, which leads to suboptimal results and risks to patient health.SUMMARY

[0006] The present disclosure describes a computer system and method for adjusting the energy output of medical instruments in surgical environments. According to an embodiment, the computer system includes one or more memories and one or more processors communicatively coupled to the one or more memories. The one or more processors, individually or collectively, perform an operation that includes applying a first amount of energy to an anatomical structure using a medical instrument and determining, based on a first set of images of the anatomical structure, a treatment level of the anatomical structure caused by applying the first amount of energy to the anatomical structure using the medical instrument. The operation also includes adjusting, based on the treatment level, an energy output of the medical instrument such that the medical instrument applies a second amount of energy to the anatomical structure.

[0007] According to another embodiment, the method includes applying a first amount of energy to an anatomical structure using a medical instrument and determining, based on a first set of images of the anatomical structure, a treatment level of the anatomical structure caused by applying the first amount of energy to the anatomical structure using the medical instrument. The method also includes adjusting, based on the treatment level, an energy output of the medical instrumentAttorney Docket No. P07031-WO (160618)such that the medical instrument applies a second amount of energy to the anatomical structure. Other embodiments include a non-transitory machine-readable medium storing instructions for adjusting an energy output of a medical instrument that, when executed by one or more processors, cause the one or more processors to, individually or collectively, perform the method.

[0008] The foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In that regard, additional aspects, features, and advantages of the present disclosure will be apparent to one skilled in the art from the following detailed description.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figures 1A through 1C illustrate an example medical system.

[0010] Figure 2A illustrates an example medical system.

[0011] Figure 2B illustrates an example medical instrument system in the medical system of Figure 2A.

[0012] Figure 2C illustrates an example portion of the medical instrument system of Figure 2B.

[0013] Figure 3 illustrates an example operation for adjusting an energy output of a medical instrument.

[0014] Figure 4 illustrates an example operation for generating a recommendation.

[0015] Figure 5 illustrates an example operation for determining an energy level.

[0016] Figures 6A, 6B, 6C, and 6D illustrate example operations for adjusting an energy level.

[0017] Figure 7 illustrates an example operation for adjusting an energy level.

[0018] Figure 8 illustrates an example operation for adjusting an energy level.

[0019] Figure 9 illustrates an example operation for adjusting an energy level.Attorney Docket No. P07031-WO (160618)

[0020] Figure 10 illustrates an example operation for adjusting an energy level.

[0021] Figure 11 illustrates an example operation for generating a recommendation.

[0022] Figure 12 illustrates an example operation for controlling an instrument.

[0023] Figures 13A and 13B illustrate example operations for controlling an instrument.

[0024] Figure 14 is a flowchart of an example method for adjusting an energy level.DETAILED DESCRIPTION

[0025] Doctors may use computer assisted surgical systems to perform operations on patients, even remotely. During the operations, doctors may use medical instruments that apply energy (e.g., electrical energy, heat energy, ultrasound energy, light energy, radiofrequency energy, plasma energy, etc.) to tissue. There is wide variability in the medical instruments. For example, different medical instruments may output different energy types and levels. Some medical instruments output energy for cutting, while other medical instruments output energy for sealing or coagulation.

[0026] The present disclosure describes a computer system that sets and adjusts the energy output of a medical instrument. The computer system may receive multiple inputs, such as images of the tissue, the type of medical procedure to be performed (e.g., dissection, sealing, repair, ablation, etc.), the type of medical instrument to be used (e.g., bipolar, monopolar, etc.), a pose of the medical instrument (e.g., a position or an angle of the medical instrument, which may indicate a thickness or an amount of tissue grabbed by the medical instrument), an amount of force applied by the medical instrument on the tissue, or information about the tissue (e.g., a model of the tissue, depth, etc.). The computer system may use some or all of this information to determine an amount of energy to be output by the medical instrument to perform the medical procedure on the tissue.

[0027] While the medical instrument is applying energy to the tissue, the computer system may analyze images of the tissue (e.g., camera images, hyperspectral / dragonfly / firefly images, ultrasound images, etc.) to assess damage caused by the medical instrument to the tissue. For example, the computer systemAttorney Docket No. P07031-WO (160618)may determine a treatment level of the tissue (e.g., temperature of the tissue, size of a burned area on the tissue, etc.), oxygenation level of the tissue, or an amount of smoke. The computer system may then adjust the amount of energy output by the medical instrument according to the determined damage and according to any of the information discussed above (e.g., a movement of the medical instrument, a change in force applied by the medical instrument on the tissue, etc.). For example, the computer system may provide guidance to an operator about the energy output of the medical instrument, and the operator may adjust the energy output manually. As another example, the computer system may request confirmation from the operator prior to automatically adjusting the energy output.

[0028] In certain embodiments, the computer system provides several technical advantages. For example, the computer system may determine or set the energy level of a medical instrument to safely perform a determined procedure. As another example, the computer system may adjust the energy level of the medical instrument according to a number of inputs or factors during an operation. In this manner, the computer system reduces damage to tissue, which improves patient health.

[0029] In some examples, one or more components of the medical system may be implemented as a computer-assisted surgical system. It is understood, however, that the medical system may be implemented in any type of medical system (e.g., digital fiducial systems, anatomy detection systems, and clinical guidance systems). Figure 1A shows an example computer-assisted surgical system 100 that may implement some of the features described herein.

[0030] The surgical system 100 may include a manipulator assembly 102, a user control apparatus 104, and an auxiliary apparatus 106, all of which are communicatively coupled to each other. The surgical system 100 may be utilized by a medical team to perform a computer-assisted medical procedure or other similar operation on a body of a patient 108 or on any other body as may serve a particular implementation. The medical team may include a first user 110-1 (such as a surgeon for a surgical procedure), a second user 110-2 (such as a patient-side assistant), a third user 110-3 (such as another assistant, a nurse, a trainee, etc.), and a fourth user 110-4 (such as an anesthesiologist for a surgical procedure), all of whom may be collectively referred to as users 110, and each of whom may control, interact with, orAttorney Docket No. P07031-WO (160618)otherwise be a user of the surgical system 100. More, fewer, or alternative users may be present during a medical procedure as may serve a particular implementation. For example, team composition for different medical procedures, or for non-medical procedures, may differ and include users with different roles.

[0031] Although Figure 1A illustrates an ongoing minimally invasive medical procedure such as a minimally invasive surgical procedure, it will be understood that the surgical system 100 may similarly be used to perform open medical procedures or other types of operations. For example, operations such as exploratory imaging operations, mock medical procedures used for training purposes, or other operations may also be performed.

[0032] The manipulator assembly 102 may include one or more manipulator arms 112 (e.g., manipulator arms 112-1 through 112-4) to which one or more instruments may be coupled. The instruments may be used for a computer-assisted surgical procedure on the patient 108 (e.g., by being at least partially inserted into the patient 108 and manipulated within the patient 108). While the manipulator assembly 102 is depicted and described herein as including four manipulator arms 112, the manipulator assembly 102 may include a single manipulator arm 112 or any other number of manipulator arms as may serve a particular implementation. Although the example of Figure 1A illustrates the manipulator arms 112 as robotic manipulator arms, in some examples, one or more instruments may be partially or entirely manually controlled, such as by being handheld and controlled manually by a person. These partially or entirely manually controlled instruments may be used in conjunction with, or as an alternative to, computer-assisted instrumentation that is coupled to the manipulator arms 112.

[0033] During the medical operation, the user control apparatus 104 may facilitate teleoperational control by the user 110-1 of the manipulator arms 112 and instruments attached to the manipulator arms 112. To this end, the user control apparatus 104 may provide the user 110-1 with imagery of an operational area associated with the patient 108 as captured by an imaging device. The manipulator arms 112 or any instruments coupled to the manipulator arms 112 may mimic the dexterity of the hand, wrist, and fingers of the user 110-1 across multiple degrees of freedom of motion. In this manner, the user 110-1 may intuitively perform a procedure (e.g., an incision procedure, aAttorney Docket No. P07031-WO (160618)suturing procedure, etc.) using one or more of the manipulator arms 112 or any instruments coupled to the manipulator arms 112.

[0034] The auxiliary apparatus 106 may include one or more computing devices that perform auxiliary functions in support of the procedure, such as providing insufflation, electrocautery energy, illumination or other energy for imaging devices, image processing, or coordinating components of the surgical system 100. In some examples, the auxiliary apparatus 106 may include a display monitor 114 that displays one or more user interfaces, or graphical or textual information in support of the procedure. In some instances, the display monitor 114 may be a touchscreen display that provides user input functionality. Augmented content provided by a region-based augmentation system may be similar, or differ from, content associated with the display monitor 114 or one or more display devices in the operation area (not shown).

[0035] The manipulator assembly 102, user control apparatus 104, and auxiliary apparatus 106 may be communicatively coupled one to another in any suitable manner. For example, the manipulator assembly 102, user control apparatus 104, and auxiliary apparatus 106 may be communicatively coupled by way of control lines 116, which may represent any wired or wireless communication link as may serve a particular implementation. To this end, manipulator assembly 102, user control apparatus 104, and auxiliary apparatus 106 may each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, and so forth.

[0036] Figure 1B illustrates an example manipulator assembly 102. As seen in Figure 1 B, the manipulator assembly 102 includes a base 118, a manipulator arm 112-1, a manipulator arm 112-2, a manipulator arm 112-3, and a manipulator arm 112-4. Each manipulator arm 112-1, 112-2, 112-3, and 112-4 is pivotably coupled to the base 118. Although the base 118 may include casters to allow ease of mobility, in some embodiments, the manipulator assembly 102 is fixedly mounted to a floor, ceiling, operating table, structural framework, or the like.

[0037] In a typical procedure, two of the manipulator arms 112-1, 112-2, 112-3, or 112-4 hold surgical instruments and a third holds a stereo endoscope. The remaining manipulator arms are available so that other instruments may be introduced at theAttorney Docket No. P07031-WO (160618)work site. Alternatively, the remaining manipulator arms may be used for introducing another endoscope or another image capturing device, such as an ultrasound transducer, to the work site.

[0038] Each of the manipulator arms 112-1, 112-2, 112-3, and 112-4 may be formed of links that are coupled together and manipulated through actuatable joints. Each of the manipulator arms 112-1, 112-2, 112-3, and 112-4 may include a setup arm and a device manipulator. The setup arm positions its held device so that a pivot point occurs at its entry aperture into the patient. The device manipulator may then manipulate its held device so that the held device may be pivoted about the pivot point, inserted into and retracted out of the entry aperture, and rotated about its shaft axis. Each of the manipulator arms 112-1, 112-2, 112-3, and 112-4 may include sensors (e.g., kinematics sensors, position sensors, accelerometers, etc.) that detect or track movement of the manipulator arms 112-1, 112-2, 112-3, and 112-4. For example, these sensors may detect how far or how quickly a manipulator arm 112-1, 112-2, 112-3, or 112-4 moves in a certain direction.

[0039] Figure 1C illustrates an example user control apparatus 104. The user control apparatus 104 includes a stereo vision display 120 so that the user may view the surgical work site in stereo vision from images captured by the stereoscopic camera of the manipulator assembly 102. Left and right eyepieces 122 and 124 are provided in the stereo vision display 120 so that the user may view left and right display screens inside the display 120 respectively with the user's left and right eyes. While viewing typically an image of the surgical site on a suitable viewer or display, the surgeon performs the surgical procedures on the patient by manipulating master control input devices, which in turn control the motion of robotic instruments.

[0040] The user control apparatus 104 also includes left and right input devices 126 and 128 that the user may grasp respectively with his / her left and right hands to manipulate devices (e.g., surgical instruments) being held by the manipulator arms 112-1, 112-2, 112-3, and 112-3 of the manipulator assembly 102 in preferably six or more degrees of freedom (“DOF”). Foot pedals 130 with toe and heel controls are provided on the user control apparatus 104 so the user may control movement or actuation of devices associated with the foot pedals.Attorney Docket No. P07031-WO (160618)

[0041] A processing device 132 is provided in the user control apparatus 104 for control and other purposes. The processing device 132 performs various functions in the surgical system 100. One function performed by processing device 132 may be to translate and transfer the mechanical motion of input devices 126 and 128 to actuate their corresponding joints in their associated manipulator arms 112-1, 112-2, 112-3, and 112-4 so that the surgeon can effectively manipulate devices, such as the surgical instruments. Another function of the processing device 132 may be to implement the methods, crosscoupling control logic, and controllers or processors described herein. The auxiliary apparatus 106 may include a processing device 132 that performs the functions or actions described herein. The processing device 132 may include a processor and a memory that perform the functions described herein.

[0042] The processor may include any electronic circuitry, including, but not limited to one or a combination of microprocessors, microcontrollers, application specific integrated circuits (ASIC), application specific instruction set processor (ASIP), or state machines, that communicatively couples to a memory and controls the operation of the user control apparatus 104 or the auxiliary apparatus 106. The processor may be 8-bit, 16-bit, 32-bit, 64-bit or of any other suitable architecture. The processor may include an arithmetic logic unit (ALU) for performing arithmetic and logic operations, processor registers that supply operands to the ALU and store the results of ALU operations, and a control unit that fetches instructions from memory and executes them by directing the coordinated operations of the ALU, registers and other components. The processor may include other hardware that operates software to control and process information. The processor executes software stored on a memory to perform any of the functions described herein. The processor controls the operation and administration of the user control apparatus 104 or the auxiliary apparatus 106 by processing information (e.g., information received from the user control apparatus 104, the manipulator assembly 102, the auxiliary apparatus 106, or a memory). The processor is not limited to a single processing device and may encompass multiple processing devices contained in the same device or computer or distributed across multiple devices or computers. The processor is considered to perform a set of functions or actions if the multiple processing devices collectively perform the set of functions or actions, even if different processing devices perform different functions or actions in the set.Attorney Docket No. P07031-WO (160618)

[0043] Figure 2A illustrates an example computer-assisted surgical system 200 that may implement some of the features described herein. The surgical system 200 can be used, for example, in surgical, diagnostic, therapeutic, biopsy, or non-medical procedures. As shown in Figure 2A, the surgical system 200 (which may be a robotically-assisted surgical system) includes one or more manipulator assemblies 202 for operating one or more medical instrument systems 204 in performing various procedures on a patient P positioned on a table T in a medical environment. For example, the manipulator assembly 202 can drive catheter or end effector motion, can apply treatment to target tissue, or can manipulate control members. The manipulator assembly 202 can be teleoperated, non-teleoperated, or a hybrid teleoperated and non-teleoperated assembly with select degrees of freedom of motion that can be motorized or teleoperated and select degrees of freedom of motion that can be nonmotorized or non-teleoperated. An operator input system 206, which can be inside or outside of the medical environment, generally includes one or more control devices for controlling the manipulator assembly 202. The manipulator assembly 202 supports a medical instrument system 204 and can optionally include a plurality of actuators or motors that drive inputs on the medical instrument system 204 in response to commands from a control system 212. The actuators can optionally include drive systems that when coupled to the medical instrument system 204 can advance the medical instrument system 204 into a natural or surgically created anatomic orifice. Other drive systems can move the distal end of the medical instrument in multiple degrees of freedom, which can include three degrees of linear motion (e.g., linear motion along the x, y, and z Cartesian axes) and in three degrees of rotational motion (e.g., rotation about the x, y, and z Cartesian axes). The manipulator assembly 202 can support various other systems for irrigation, treatment, or other purposes. Such systems can include fluid systems (e.g., reservoirs, heating / cooling elements, pumps, and valves), generators, lasers, interrogators, and ablation components.

[0044] The surgical system 200 also includes a display system 210 for displaying an image or representation of the surgical site and a medical instrument system 204. The image or representation is generated by an imaging system 209, which may include an endoscopic imaging system. The display system 210 and operator input system 206 may be oriented so that an operator O can control the medical instrument system 204 and the operator input system 206 with the perception of telepresence. AAttorney Docket No. P07031-WO (160618)graphical user interface can be displayable on the display system 210 or a display system of an independent planning workstation.

[0045] In some examples, the imaging system 209 includes an endoscopic imaging system with components that are integrally or removably coupled to the medical instrument system 204. However, in some examples, a separate imaging device, such as an endoscope, attached to a separate manipulator assembly can be used with the medical instrument system 204 to image the surgical site. The imaging system 209 can be implemented as hardware, firmware, software, or a combination thereof, which interact with or are otherwise executed by one or more computer processors, which can include the processors 214 of the control system 212.

[0046] The surgical system 200 also includes a sensor system 208. The sensor system 208 may include a position / location sensor system (e.g., an actuator encoder or an electromagnetic (EM) sensor system) or a shape sensor system (e.g., an optical fiber shape sensor) for determining the position, orientation, speed, velocity, pose, or shape of the medical instrument system 204. These sensors may also detect a position, orientation, or pose of the patient P on the table T. For example, the sensors may detect whether the patient P is face-down or face-up. As another example, the sensors may detect a direction in which the head of the patient P is directed. The sensor system 208 can also include temperature, pressure, force, or contact sensors, or the like.

[0047] The surgical system 200 can also include a control system 212, which includes at least one memory 216 and at least one computer processor 214 for effecting control between the medical instrument system 204, the operator input system 206, the sensor system 208, and the display system 210. The control system 212 includes programmed instructions (e.g., a non-transitory machine-readable medium storing the instructions) to implement a procedure using the surgical system 200, including for navigation, steering, imaging, engagement feature deployment or retraction, applying treatment to target tissue (e.g., via the application of energy), or the like.

[0048] The control system 212 may further include a virtual visualization system to provide navigation assistance to the operator O when controlling medical instrumentAttorney Docket No. P07031-WO (160618)system 204 during an image-guided surgical procedure. Virtual navigation using the virtual visualization system can be based upon reference to an acquired pre-operative or intra-operative dataset of anatomic passageways. The virtual visualization system processes images of the surgical site imaged using imaging technology, such as computerized tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, or the like. The control system 212 uses a pre-operative image to locate the target tissue (using vision imaging techniques or by receiving user input) and create a pre-operative plan, including an optimal first location for performing treatment. The pre-operative plan can include, for example, a planned size to expand an expandable device, a treatment duration, a treatment temperature, or multiple deployment locations.

[0049] The processor 214 is any electronic circuitry, including, but not limited to one or a combination of microprocessors, microcontrollers, application specific integrated circuits (ASIC), application specific instruction set processor (ASIP), or state machines, that communicatively couples to the memory 216 and controls the operation of the control system 212. The processor 214 may be 8-bit, 16-bit, 32-bit, 64-bit or of any other suitable architecture. The processor 214 may include an arithmetic logic unit (ALU) for performing arithmetic and logic operations, processor registers that supply operands to the ALU and store the results of ALU operations, and a control unit that fetches instructions from memory and executes them by directing the coordinated operations of the ALU, registers and other components. The processor 214 may include other hardware that operates software to control and process information. The processor 214 executes software stored on the memory 216 to perform any of the functions described herein. The processor 214 controls the operation and administration of the control system 212 by processing information (e.g., information received from the manipulator assembly 202, the operator input system 206, and the memory 216). The processor 214 is not limited to a single processing device and may encompass multiple processing devices contained in the same device or computer or distributed across multiple devices or computers. The processor 214 is considered to perform a set of functions or actions if the multiple processing devices collectively perform the set of functions or actions, even if different processing devices perform different functions or actions in the set.Attorney Docket No. P07031-WO (160618)

[0050] The memory 216 may store, either permanently or temporarily, data, operational software, or other information for the processor 214. The memory 216 may include any one or a combination of volatile or non-volatile local or remote devices suitable for storing information. For example, the memory 216 may include random access memory (RAM), read only memory (ROM), magnetic storage devices, optical storage devices, or any other suitable information storage device or a combination of these devices. The software represents any suitable set of instructions, logic, or code embodied in a computer-readable storage medium. For example, the software may be embodied in the memory 216, a disk, a CD, or a flash drive. In particular embodiments, the software may include an application executable by the processor 214 to perform one or more of the functions described herein. The memory 216 is not limited to a single memory and may encompass multiple memories contained in the same device or computer or distributed across multiple devices or computers. The memory 216 is considered to store a set of data, operational software, or information if the multiple memories collectively store the set of data, operational software, or information, even if different memories store different portions of the data, operational software, or information in the set.

[0051] Figure 2B illustrates an example medical instrument system 204 in the surgical system 200. In some embodiments, the medical instrument system 204 is used in an image-guided medical procedure. For example, the medical instrument system 204 may be used for non-teleoperational exploratory procedures or in procedures involving traditional manually operated medical instruments, such as endoscopy.

[0052] The medical instrument system 204 includes an elongate flexible device 220, such as a flexible catheter or endoscope (e.g., gastroscope, bronchoscope), coupled to a drive unit 222. The elongate flexible device 220 includes a flexible body 224 having a proximal end 226 and a distal end, or tip portion, 228. In some embodiments, the flexible body 224 has an approximately 14-20 millimeter outer diameter. Other flexible body outer diameters may be larger or smaller. The flexible body 224 has an appropriate length to reach certain portions of the anatomy, such as the lungs, sinuses, throat, or the upper or lower gastrointestional region, when the flexible body 224 is inserted into a patient’s oral or nasal cavity.Attorney Docket No. P07031-WO (160618)

[0053] The medical instrument system 204 includes a tracking system 230 for determining the position, orientation, speed, velocity, pose, or shape of the distal end 228 or of one or more segments 232 along the flexible body 224 using one or more sensors or imaging devices. The entire length of the flexible body 224, between the distal end 228 and the proximal end 226, is effectively divided into the segments 232. The tracking system 230 is implemented as hardware, firmware, software, or a combination thereof, which interact with or are otherwise executed by one or more computer processors, which may include the processors 214 of control system 212.

[0054] The tracking system 230 tracks distal the end 228 or one or more of the segments 232 using a shape sensor 234. In some embodiments, the tracking system 230 tracks the distal end 228 using a position sensor system 236, such as an electromagnetic (EM) sensor system. In some examples, the position sensor system 236 measures six degrees of freedom (e.g., three position coordinates x, y, and z and three orientation angles indicating pitch, yaw, and roll of a base point) or five degrees of freedom (e.g., three position coordinates x, y, and z and two orientation angles indicating pitch and yaw of a base point).

[0055] The flexible body 224 includes one or more channels 238 sized and shaped to receive one or more medical instruments 240. In some embodiments, the flexible body 224 includes two channels 238 for separate instruments 240, however, a different number of channels 238 can be provided. Figure 2C illustrates an example portion of the medical instrument system 204 of Figure 2B. As seen in Figure 2C, the medical instrument 240 extends through the flexible body 224. In some embodiments, the medical instrument 240 can be used for procedures and aspects of procedures, such as surgery, biopsy, ablation, mapping, imaging, illumination, irrigation, or suction. The medical instrument 240 is deployed through the channel 238 of the flexible body 224 and is used at a target location within the anatomy. The medical instrument 240 includes, for example, image capture devices, biopsy instruments, ablation instruments, catheters, laser ablation fibers, or other surgical, diagnostic, or therapeutic tools. The medical tools include end effectors having a single working member such as a scalpel, a blunt blade, a lens, an optical fiber, an electrode, or the like. Other end effectors include, for example, forceps, graspers, balloons, needles, scissors, clip appliers, or the like. Other end effectors further include electricallyAttorney Docket No. P07031-WO (160618)activated end effectors such as electrosurgical electrodes, transducers, sensors, imaging devices, or the like. The medical instrument 240 is advanced from the opening of the channel 238 to perform the procedure and then retracted back into the channel when the procedure is complete. The medical instrument 240 is removed from the proximal end 226 of the flexible body 224 or from another optional instrument port (not shown) along the flexible body 224. The medical instrument 240 may be used with an image capture device (e.g., an endoscopic camera) also within the elongate flexible device 220. Alternatively, the medical instrument 240 may itself be the image capture device.

[0056] The medical instrument 240 additionally houses cables, linkages, or other actuation controls (not shown) that extend between the proximal and distal ends to controllably bend the distal end of the medical instrument 240. The flexible body 224 also houses cables, linkages, or other steering controls (not shown) that extend between the drive unit 222 and the distal end 228 to controllably bend the distal end 228 as shown, for example, by the broken dashed line depictions 242 of the distal end 228. In some examples, at least four cables are used to provide independent “up-down” steering to control a pitch motion of the distal end 228 and “left-right” steering to control a yaw motion of the distal end 228. In embodiments in which the medical instrument system 204 is actuated by a robotically-assisted assembly, the drive unit 222 can include drive inputs that removably couple to and receive power from drive elements, such as actuators, of the teleoperational assembly. In some embodiments, the medical instrument system 204 includes gripping features, manual actuators, or other components for manually controlling the motion of the medical instrument system 204. The information from the tracking system 230 can be sent to a navigation system 244, where the information is combined with information from the visualization system 246 or the preoperatively obtained models to provide the physician or other operator with real-time position information.

[0057] Figures 3 through 14 illustrate example operations performed by a computer system in a medical system (e.g., the surgical system 100 of Figure 1 A or the surgical system 200 of Figure 2A). Generally, the computer system (which may be implemented in the user control apparatus 104 or the auxiliary apparatus 106 of the surgical system 100 using the processing device 132 or in the control system 212 ofAttorney Docket No. P07031-WO (160618)the surgical system 200 using the processor 214 and the memory 216) recommends, sets, or adjusts an energy level of a medical instrument.

[0058] The computer system may be described as performing certain actions (e.g., capturing video, moving cameras, detecting objects, providing feedback, etc.) that may involve other components, such as cameras, motors, etc. In these instances, it is understood that the controller performs these actions by communicating signals to the other components that causes those components to perform the actions.

[0059] Each of these figures may describe the computer system using or analyzing different types of inputs. Although depicted separately, it is understood that the computer system may consider any number of these different types of inputs together or separately. Additionally, it is understood that the computer system may perform any number of the example operations together or separately.

[0060] Figure 3 illustrates an example operation 300 for adjusting an energy output of a medical instrument. The computer system performs the operation 300. As seen in Figure 3, the computer system receives inputs 302. Generally, the computer system uses the inputs 302 to determine an energy level 304 to be output by an instrument 306. The computer system may consider many different types of inputs 302. For example, the inputs 302 may include an image or video of a surgical environment. The computer system may use machine vision techniques to analyze the image or video to determine various aspects of the surgical environment, such as a tissue type involved in the operation.

[0061] As another example, the inputs 302 may include output of a haptic sensor on a medical instrument. This output may indicate an amount of force that the medical instrument is applying to the tissue. A higher amount of force may lead to faster dissection and potential damage, while a lower amount of force may lead to slower and safer dissection.

[0062] The inputs 302 may include information about a medical instrument used to apply energy to the tissue. For example, the information may indicate an instrument type for the medical instrument. The type of medical instrument may indicate how the medical instrument applies energy to tissue. For example, the medical instrument may be a bipolar instrument (e.g., a force bipolar instrument) or a monopolar instrumentAttorney Docket No. P07031-WO (160618)(e.g., monopolar curve scissors (MCS), hook, spatula, etc.). The information may also indicate an energy type and level produced by the medical instrument. Different energy types and levels may be needed depending on the surgical task, which may also be indicated by the information. For example, different energy types and levels may be needed to cut tissue as opposed coagulating tissue. The information may indicate an instrument pose, which may indicate an angle or yaw of a jaw of the medical instrument. This information may suggest an amount of tissue (e.g., the thickness of tissue) that is being grabbed by the medical instrument. The more tissue that is being grabbed, the more energy that may need to be applied to the tissue.

[0063] The inputs 302 may include other types of images of the tissue. For example, the computer system may receive fluorescent images (e.g., hyperspectral images, dragonfly images, firefly images, etc.) or ultrasound images that provide insight as to the health of the tissue (e.g., the temperature of the tissue, whether the tissue is too burned / damaged, etc.). The fluorescent images may be produced by injecting a dye into the tissue and when particular wavelengths of light are directed towards the tissue, the dye may fluoresce to produce different colors depending on the condition or health of the tissue. The ultrasound images may provide views of areas beneath the surface of the tissue.

[0064] The inputs 302 may also include images of the tissue captured before the procedure (which may also be referred to as pre-op images). These images may include three-dimensional models of the surgical environment or tissue. For example, these models may include segmented models and volume renderings that are generated from magnetic resonance imaging (MRI) scans or computed tomography (CT) scans. These models may show the depth of tissue or target anatomical structures.

[0065] The inputs 302 may also include kinematics data that indicates the movement or pose of the medical instrument in the surgical environment. This information may indicate where the medical instrument is in the surgical environment, and consequently, the type or characteristics of the tissue near the medical instrument. Additionally, this information may indicate how much tissue is being grabbed or engaged by the medical instrument.Attorney Docket No. P07031-WO (160618)

[0066] The inputs 302 may also include depth information, such as depth measurements or a depth map of the tissue. This information may indicate the thickness of the tissue at certain portions of the surgical environment. The computer system may use this information to determine characteristics of the tissue throughout the surgical environment.

[0067] The inputs 302 may also include tissue tracking information. The computer system may use one or more processes to track tissue as a camera moves to capture images or video of the tissue. The computer system may use the images or video to generate a virtual map of the surgical environment (e.g., generated using simultaneous localization and mapping (SLAM)). The computer system may then use the virtual map to track the tissue that is contacting the medical instrument as the medical instrument is moved across the tissue.

[0068] The computer system uses the inputs 302 to determine an energy level 304 for an instrument 306. The instrument 306 may be an energy instrument that applies energy to tissue (e.g., to cut, seal, repair, coagulate, or ablate the tissue). For example, the instrument 306 may apply one or more of electrical energy, heat energy, ultrasound energy, light energy, radiofrequency energy, or plasma energy to tissue. The computer system may determine and set, from the inputs 302 (e.g., the type of procedure being performed and the type of tissue on which the procedure is performed), an initial energy level 304 for the instrument 306.

[0069] As the instrument 306 applies energy to the tissue, the computer system may determine, from the inputs 302, changes that occur to the tissue. For example, the computer system may determine a treatment level, oxygenation level, or temperature of the tissue as energy is applied. The computer system may then determine an adjustment 308 to the energy level 304. The adjustment 308 may increase or decrease the energy level 304. For example, if the computer system determines that the tissue is suffering too much damage is becoming too burned, the computer system may reduce the energy level 304 to reduce or slow the burning of the tissue. As another example, if the computer system determines that the oxygenation level of the tissue is falling too low, the computer system may reduce the energy level 304 to reduce or slow the drop in oxygenation levels. After adjusting theAttorney Docket No. P07031-WO (160618)energy level 304, the computer system may apply the new energy level 304 to the tissue using the instrument 306.

[0070] The computer system may request confirmation or input from the operator prior to making the adjustment 308. For example, the computer system may provide guidance (e.g., a message) to the operator about the adjustment 308 to the energy level 304, and the operator may adjust the energy level 304 manually. As another example, the computer system may request confirmation from the operator prior to automatically adjusting the energy level 304.

[0071] In this manner, the computer system uses various inputs 302 to determine, set, or adjust the energy level 304 of the instrument 306. In certain embodiments, the computer system reduces the amount of damage caused to tissue by energy applications relative to existing systems, which improves patient health and safety.

[0072] Figure 4 illustrates an example operation 400 for generating a recommendation. The computer system performs the operation 400. Generally, prior to applying energy to tissue using the medical instrument, the computer system may use computer vision techniques to analyze an image or video of the tissue to determine a tissue type. Different tissue types have different consistencies, thicknesses, densities, etc., which may affect how much energy should be applied to the tissue. For example, solid organs like the kidney tend to be tougher and be less prone to damage, while fascia, omentum, and tubular structures like vessels and ureters tend to be less dense and be more prone to damage. As a result, the computer system may recommend applying a higher energy level to solid organs and a lower energy level to fascia, omentum, and tubular structures to avoid damage. Additionally, the computer system may use other imaging techniques (e.g., pre-op scans and models and fluorescent imaging) to determine the consistency, thickness, and density of tissue or to supplement these determinations.

[0073] The computer system begins by receiving the inputs 302. As discussed above, the inputs 302 may indicate a procedure 402, which may have been provided by an operator of the computer system to indicate the medical procedure that is being performed by applying energy to tissue. For example, the procedure 402 may indicate whether the tissue is being cut, dissected, sealed, repaired, coagulated, ablated, etc.Attorney Docket No. P07031-WO (160618)Different amounts of energy may need to be applied to the tissue to perform different procedures. The inputs 302 may also include an image 404 of the tissue. The image 404 may be a photograph of the tissue or the image 404 may be a frame of a video showing the tissue. The image 404 may be captured or generated prior to energy application to the tissue.

[0074] The computer system may analyze the image 404 to determine a tissue type 406 of the tissue in the image 404. The computer system may use computer vision techniques to analyze the image 404. For example, the computer system may determine certain aspects of the image (e.g., color, shading, contrast, etc.) and compare these aspects against a database of information for other images. This comparison may produce information about the tissue shown in the image 404, which is encapsulated as the tissue type 406. For example, the tissue type 406 may indicate an anatomical structure for the tissue (e.g., a solid organ like the kidney, fascia, omentum, vessels, ureters, etc.). As another example, the tissue type 406 may indicate a consistency, thickness, or density of the tissue. As a result, the tissue type 406 may indicate any quality or characteristic of the tissue.

[0075] As explained above, different types of tissue may be more or less prone to damage through energy application. For example, tougher tissue with greater consistencies, thicknesses, or densities may be less prone to damage, and a higher energy level may be needed to cut, seal, coagulate, or ablate such tissue. By contrast, fascia, omentum, or tubular structures tend to be less dense and more prone to damage. The computer system uses the procedure 402 to be performed and the determined tissue type 406 to determine a recommendation 408. The recommendation 408 may include a type of medical instrument to use to perform the procedure 402. For example, the recommendation 408 may indicate a monopolar instrument, a bipolar instrument, etc. Additionally, the recommendation 408 may indicate an energy level to use to perform the procedure 402. For example, the recommendation 408 may indicate an absolute energy level, or the recommendation 408 may indicate a range for the energy level. The operator may follow the recommendation 408 by selecting the recommended medical instrument and by setting the energy level of the medical instrument according to the recommendationAttorney Docket No. P07031-WO (160618)408. In this manner, the computer system provides the recommendation 408 that reduces damage to the tissue during the procedure 402.

[0076] Figure 5 illustrates an example operation 500 for determining an energy level. The computer system performs the operation 500. Generally, the operation 500 may be part of the operation 400 shown in Figure 4. The computer system may perform the operation 500 to determine the energy level for the medical instrument prior to applying energy to the tissue. The computer system may provide the energy level in a recommendation (e.g., the recommendation 408 shown in Figure 4) or in a message to the operator.

[0077] The computer system begins with the inputs 302. As seen in Figure 5, the inputs 302 may include an image 502 or a model 503. As discussed previously, the image 502 may be a photograph of the tissue or a frame of a video of the tissue. In some instances, the image 502 may be of an anatomical structure that includes the tissue. The image 502 may be any type of image. For example, the image 502 may be an image captured when directing white light at the tissue or anatomical structure. As another example, the image 502 may be a fluorescent image that is generated by injecting a dye into the tissue or anatomical structure and by directing a light at the tissue or anatomical structure that causes the dye to fluoresce. As a result, the image 502 may show the tissue or anatomical structure in various states of illumination.

[0078] The model 503 may be a three-dimensional model of the anatomical structure, including the tissue. The computer system may have generated the model 503 by stitching together images (e.g., ultrasound images) of the anatomical structure. As a result, the model 503 may be a virtual, three-dimensional representation of the anatomical structure or tissue. The model 503 may show the sizes or relative sizes of different portions of the anatomical structure and their positioning with respect to each other.

[0079] The computer system uses the inputs 302 to determine tissue information 504. For example, the computer system may use the inputs 302 to determine a type 506, consistency 508, thickness 510, density 512, etc. for the tissue shown in the image 502 or the model 503. The computer system may use computer vision techniques to analyze the image 502 to determine characteristics of the image 502Attorney Docket No. P07031-WO (160618)(e.g., color, shading, contrast, etc.). The computer system may then use a database to determine how these characteristics map or link to characteristics of the tissue, such as the type 506, consistency 508, thickness 510, or density 512. In some embodiments, the computer system may also use the model 503 to augment or supplement the determination of the tissue information 504. For example, the computer system may analyze the model 503 to determine the type 506, size, and thickness 510 of the tissue.

[0080] The computer system then uses the tissue information 504 to determine the energy level 304 for the medical instrument. The computer system may use any approach to determine the energy level 304 based on the tissue information 504. For example, the computer system may use rules or thresholds that specify the energy level 304 given certain tissue information 504. As another example, the computer system may use the tissue information 504 to reference into a database that returns the energy level 304. As another example, the computer system may use artificial intelligence or a machine learning model to predict the energy level 304 using the tissue information 504. The computer system may then recommend the energy level 304 to the operator. In some embodiments, the computer system may set the energy level of the medical instrument according to the energy level 304. The medical instrument may then be operated to apply energy to the tissue.

[0081] Figures 6A, 6B, 6C, and 6D illustrate example operations 600, 620, 640, and 660 for adjusting an energy level. The computer system performs the operations 600, 620, 640, and 660 during a procedure or while the medical instrument is applying energy to the tissue. Generally, the computer system may analyze one or more images of the tissue to determine a treatment level or oxygenation level of the tissue, which may change as energy is applied to the tissue. The computer system may then generate a map of the tissue that shows the treatment level or oxygenation level throughout the tissue. The computer system uses this map to adjust the energy level of the medical instrument. In this manner, the computer system adjusts the energy level to accommodate for damage suffered by the tissue.

[0082] The treatment level may be a numerical value that indicates whether tissue is burned and how burned the tissue is. For example, a treatment level of zero may indicate that the tissue is not burned. A higher value of treatment level may indicateAttorney Docket No. P07031-WO (160618)that the tissue is more burned. Additionally, the oxygenation level may be a numerical value that indicates how oxygenated the tissue is. For example, the higher the oxygenation level, the more oxygen that is present in the tissue.

[0083] Figure 6A shows the operation 600 for determining a treatment level of the tissue. Generally, the computer system determines the treatment level by comparing an image of the tissue with one or more reference images of tissue with various treatment levels. By determining a reference image that appears similar to the image of the tissue, the computer system may approximate or determine the treatment level of the tissue.

[0084] As seen in Figure 6A, the computer system begins by receiving the inputs 302. The inputs 302 may include an image 602 of the tissue. Generally, the image 602 may show the tissue during a procedure while energy is or has been applied to the tissue. The image 602 may be any type of image. For example, the image 602 may be an image captured when directing white light at the tissue or anatomical structure. As another example, the image 602 may be a fluorescent image that is generated by injecting a dye into the tissue or anatomical structure and by directing a light at the tissue or anatomical structure that causes the dye to fluoresce. The dye may fluoresce with different colors or tones depending on the damage to the tissue. As a result, the image 602 may show the tissue or anatomical structure in various states of illumination.

[0085] The computer system may compare the image 602 to references images of tissue with particular treatment levels. For example, the computer system may compare the image 602 with the reference images by comparing the color, shading, contrast, etc. of the tissue in the image 602 with the color, shading, contrast, etc. of the tissue in the reference images. Through the comparison, the computer system may determine the reference image that appears most similar to the image 602. The computer system may then determine the treatment level of the tissue in the image 602 as the treatment level of the reference image. In the example of Figure 6B, the computer system determines that the image 604 is most similar to the image 602. The computer system then determine that the tissue in the image 602 has the treatment level 606 of the tissue in the image 604.Attorney Docket No. P07031-WO (160618)

[0086] In some embodiments, the computer system may also use computer vision to analyze fluorescent images 602 of the tissue to determine a treatment level of the tissue at different depths. The fluorescent imaging may reveal how deep the bum extends beneath the surface of the tissue. By analyzing the color or appearance of the bum in the fluorescent image, the computer system may determine depths 608 of the bum beneath the surface of the tissue.

[0087] Figure 6B shows the operation 620 for determining the treatment level of the tissue. The computer system may perform the operation 620 in addition to or as an alternative to the operation 600. Generally, the computer system determines the treatment level by analyzing characteristics of the bum shown in an image of the tissue.

[0088] As seen in Figure 6B, the computer system begins by receiving the inputs 302, which include an image 622 of the tissue. Generally, the image 622 may show the tissue during a procedure while energy is or has been applied to the tissue. The image 622 may be any type of image. For example, the image 622 may be an image captured when directing white light at the tissue or anatomical structure. As another example, the image 622 may be a fluorescent image that is generated by injecting a dye into the tissue or anatomical structure and by directing a light at the tissue or anatomical structure that causes the dye to fluoresce. The dye may fluoresce with different colors or tones depending on the damage to the tissue. As a result, the image 622 may show the tissue or anatomical structure in various states of illumination.

[0089] The computer system analyzes the image 622 to determine various characteristics or aspects of a bum on the tissue in the image 622. For example, the computer system may determine a color 624 of the bum, a texture 626 of the bum, a size 628 of the bum, etc. The computer system may identify the pixels of the image 622 that show the bum on the tissue. These pixels may appear as a different color than the rest of the tissue, indicating that the tissue shown in the pixels is burned. The computer system may analyze the color of these pixels to determine the color 624 of the bum. The computer system may analyze the shading or contrast in the pixels to determine a texture 626 of the bum. The computer system may analyze the positioning of the pixels in the image 622 to determine the size 628 of the bum.Attorney Docket No. P07031-WO (160618)

[0090] The computer system then determine the treatment level 606 for the tissue using the information about the bum shown in the image 622. For example, the computer system may determine the treatment level 606 of the tissue from the color 624 and the texture 626 of the burn in the image 622. More severe bums may appear darker and more textured than less severe bums. As another example, the computer system may determine the treatment level 606 of the tissue from the size 628 of the bum in the image 622. The larger the bum, the more severe the bum may be.

[0091] In some embodiments, the computer system may also use computer vision to analyze fluorescent images 622 of the tissue to determine a treatment level of the tissue at different depths. The fluorescent imaging may reveal how deep the bum extends beneath the surface of the tissue. By analyzing the color or appearance of the bum in the fluorescent image, the computer system may determine depths 608 of the bum beneath the surface of the tissue.

[0092] Figure 6C shows the operation 640 for determining an oxygenation level of tissue. Generally, the computer system determines the oxygenation level analyzing an image of the tissue as energy is applied to the tissue or after energy is applied to the tissue. As more energy is applied to the tissue, perfusion or blood flow through the tissue may decrease, which reduces the oxygenation level of the tissue. As the oxygenation level of the tissue changes, so does the appearance of the tissue in images. Thus, oxygenation level may be an indication of damage to the tissue.

[0093] As seen in Figure 6C, the computer system begins by receiving the inputs 302, which include an image 642 of the tissue. Generally, the image 642 may show the tissue during a procedure while energy is or has been applied to the tissue. The image 642 may be any type of image. For example, the image 642 may be an image captured when directing white light at the tissue or anatomical structure. As another example, the image 642 may be a fluorescent image that is generated by injecting a dye into the tissue or anatomical structure and by directing a light at the tissue or anatomical structure that causes the dye to fluoresce. The dye may fluoresce with different colors or tones depending on the damage to the tissue. As a result, the image 642 may show the tissue or anatomical structure in various states of illumination.Attorney Docket No. P07031-WO (160618)

[0094] The computer system analyzes the image 642 to determine an oxygenation level 644 of the tissue. For example, the computer system may use computer vision techniques to analyze a fluorescent image 642 of the tissue. The fluorescence of the tissue may reveal the amount of blood flow and oxygenation of the tissue. By analyzing the fluorescence of the tissue, the computer system may determine the oxygenation level 644 of the tissue. The computer system may also determine the oxygenation level 644 at different depths 646 of the tissue.

[0095] Figure 6D shows the operation 660 for determining the adjustment to the energy level of the medical instrument. Generally, the computer system uses the determined treatment levels and oxygenation levels of the tissue to generate a three-dimensional map of the tissue. The map may indicate the treatment levels and oxygenation levels of the tissue at different portions or different depths of the tissue. As the medical instrument moves or applies energy to the tissue, the computer system uses the map to determine how to adjust the energy level of the medical instrument.

[0096] As seen in Figure 6D, the computer system begins with the treatment levels 606 (e.g., determined using the operation 600 shown in Figure 6A or the operation 620 shown in Figure 6B) and the oxygenation levels 644 (e.g., determined using the operation 640 shown in Figure 5C). As discussed previously, the determined treatment levels 606 and oxygenation levels 644 may be at different depths of the tissue.

[0097] The computer system generates a map 662 of the tissue using the treatment levels 606 and the oxygenation levels 644. The map 662 may be a three-dimensional map of the tissue. Additionally, the map 662 may indicate the treatment levels 606 and the oxygenation levels 644 of the tissue at different portions of the tissue.

[0098] The computer system may reference the map 662 to determine the treatment level 606 or the oxygenation level 644 at a particular point on or in the tissue. For example, as the medical instrument moves over the tissue, the computer system may determine, from the map 662, a location 664 of the medical instrument relative to the tissue. The computer system may also determine, from the map 662, the treatment level 606 or oxygenation level 644 of the tissue at the location 664.Attorney Docket No. P07031-WO (160618)

[0099] The computer system then uses this information to determine how to adjust the energy level for the medical instrument to reduce or prevent the medical instrument from causing excessive damage to the tissue at the location 664. For example, if the map 662 indicates that the treatment level 606 of the tissue at the location 664 is high or that the oxygenation level 644 of the tissue at the location 664 is low, then the computer system may determine a lower energy level 304 to prevent further damage from being caused to the tissue at the location 664. As another example, if the map 662 indicates that a faster burn or a larger burn area is needed, the computer system may increase the energy level 304 of the medical instrument to cause more bum to the tissue.

[0100] The computer system determines the adjustment 308 to change the energy output of the medical instrument to be the determined energy level 304. The adjustment 308 may be a decrease or an increase to the energy output. The medical instrument may then apply the energy level 304 to the tissue. In some embodiments, the computer system determines whether the treatment level 606 or the oxygenation level 644 of the tissue at the location 664 changes as the medical instrument applies energy. The computer system then generates an update 666 and applies the update 666 to the map 662. The update 666 may indicate the change in treatment level 606 or oxygenation level 644 of the tissue. By applying the update 666 to the map, the computer system keeps the map 662 consistent with the state of the tissue as energy is applied to the tissue.

[0101] As discussed above, the computer system may request confirmation or input from the operator prior to making the adjustment 308 rather than making the adjustment 308 automatically. For example, the computer system may provide guidance (e.g., a message) to the operator about the adjustment 308 to the energy level 304, and the operator may adjust the energy level 304 manually. As another example, the computer system may request confirmation from the operator prior to automatically adjusting the energy level 304.

[0102] In some embodiments, the computer system may also reference the map 662 to determine the location 664 on the tissue where energy should be applied. For example, the location 664 may be a portion of the tissue that is not as damaged as other parts of the tissue. The computer system may determine that energy should beAttorney Docket No. P07031-WO (160618)applied to the location 664 to avoid further damaging other parts of the tissue. The computer system may mark the location 664 on the map 662 to indicate that the medical instrument should be navigated to the indicated location 664. The medical instrument may then apply energy to the location 664 on the tissue. As another example, the location 664 may be a portion of the tissue that is to be ablated. The computer system may determine that energy should be applied to the location 664 to further ablate the portion of the tissue. The computer system may mark the location 664 on the map 662 to indicate that the medical instrument should be navigated to the indicated location 664. The medical instrument may then apply energy to the location 664 on the tissue to ablate the tissue.

[0103] Figure 7 illustrates an example operation 700 for adjusting an energy level. Generally, the computer system performs the operation 700 to determine a smoke level in the surgical environment and to adjust the energy level of the medical instrument accordingly.

[0104] The computer system begins by receiving the inputs 302, which may include an image 702. The image 702 may be a photograph or frame of a video of the surgical environment. The image 702 may show the tissue and the space around the tissue. As the medical instrument applies energy to the tissue, the tissue may produce smoke. The smoke may appear in the image 702 and may provide an indication of damage being caused to the tissue. The computer system analyzes the image to determine a smoke level 704 in the space around the tissue in the image 702. The smoke level 704 may cause a haziness in the image 702, or the smoke level 704 may obstruct certain portions of the tissue from view in the image 702.

[0105] The computer system may determine the adjustment 308 to the energy level 304 of the medical instrument using the smoke level 704. For example, if the smoke level 704 causes the image 702 to be too hazy or to be too obstructed, the computer system may determine that the energy level 304 of the medical instrument should be reduced to reduce the amount of smoke produced by the tissue. The computer system may determine the adjustment 308 that reduces the energy level 304. The computer system may then apply the adjustment 308 to the energy level 304 to reduce the energy level 304 of the medical instrument.Attorney Docket No. P07031-WO (160618)

[0106] The computer system may request confirmation or input from the operator prior to making the adjustment 308 rather than making the adjustment 308 automatically. For example, the computer system may provide guidance (e.g., a message) to the operator about the adjustment 308 to the energy level 304, and the operator may adjust the energy level 304 manually. As another example, the computer system may request confirmation from the operator prior to automatically adjusting the energy level 304.

[0107] Figure 8 illustrates an example operation 800 for adjusting an energy level. Generally, the computer system may determine (e.g., from an image or video of the surgical environment, from kinematics data for the medical instrument, etc.) the type of movement or stroke of the medical instrument when performing a procedure. The computer system may then adjust the energy output of the medical instrument to be suitable for the movement or stroke. For example, if the medical instrument is moving slowly, then the computer system may reduce the energy output to avoid overburning or damaging the tissue. If the medical instrument is moving quickly, then the computer system may increase the energy output to ensure that sufficient energy is applied to the tissue to perform the procedure. As another example, the computer system may adjust the energy output to accommodate long strokes or short strokes / pokes of the medical instrument or sideways strokes or vertical strokes of the medical instrument.

[0108] Moreover, the computer system may determine an amount of force applied by the medical instrument to the tissue. There may be a force or haptic sensor positioned on the medical instrument that provides signals to the computer system to indicate the amount of force applied by the medical instrument on the tissue. The computer system may adjust the energy output of the medical instrument depending on the amount of force applied to the tissue. For example, the more force that is applied by the medical instrument to the tissue, the quicker the energy output will bum or damage the tissue. As a result, the computer system may reduce the energy output of the medical instrument if the medical instrument is applying a large amount of force to the tissue. Conversely, the less force that is applied by the medical instrument to the tissue, the less likely it is for the medical instrument to bum or damage the tissue and the more likely it is for the medical instrument to need to output more energy to perform the procedure. As a result, the computer system may increase the energyAttorney Docket No. P07031-WO (160618)output of the medical instrument if the medical instrument is applying a small amount of force to the tissue.

[0109] The computer system begins by receiving the inputs 302. In the example of Figure 8, the inputs 302 include a force 802 and a movement 804. The force 802 may be measured by a force sensor positioned on or attached to the medical instrument. The force sensor may detect or measure an amount of force applied by the medical instrument on the tissue. The force sensor may then produce an electrical signal that the computer system receives as input and as indicative of the force 802 applied by the medical instrument on the tissue.

[0110] The movement 804 may be a detected movement of the medical instrument. For example, as the medical instrument moves, different sensors positioned on or throughout the medical instrument may produce kinematics data that indicates the movement or pose of the medical instrument. The computer system may use the kinematics data to determine the movement 804 of the medical instrument. In the example of Figure 8, the movement 804 may indicate different aspects of movement. For example, the movement 804 may indicate a length 806 of the movement, a direction 808 of the movement, or a repetition 810 of the movement. The length 806 may indicate whether the movement 804 is a long or short stroke. The direction 808 may indicate whether the stroke is horizontal, vertical, sideways, etc. The repetition 810 may indicate whether the medical instrument is moving over the same are repetitively.

[0111] The computer system may determine tissue information 812 based on the movement 804 of the medical instrument. For example, the computer system may determine, from the movement 804, that the medical instrument has moved to another portion of the tissue. The computer system may then determine or retrieve tissue information 812 for that portion of the tissue. The tissue information 812 may include the consistency, thickness, density, treatment level, oxygenation level, etc. of that portion of the tissue. The computer system may reference a map or database of information previously determined for the tissue to determine the tissue information 812. Additionally or alternatively, the computer system may determine the tissue information 812 using any operation (e.g., the operation 500 shown in Figure 5 or the operations 600, 620, and 640 shown in Figures 6A, 6B, and 6C). In this manner, theAttorney Docket No. P07031-WO (160618)computer system determines or updates the tissue information 812 as the medical instrument moves in the surgical environment.

[0112] The computer system determines the adjustment 308 to the energy level 304 using the inputs 302. For example, the computer system may adjust the energy level 304 depending on the force 802 applied by the medical instrument on the tissue. Generally, as the force 802 increases, the computer system may determine the adjustment 308 that reduces the energy level 304. Conversely, as the force 802 decreases, the computer system may determine the adjustment 308 that increases the energy level 304. In this manner, the computer system reduces the chances that the medical instrument bums or damages the tissue when a higher force 802 is applied to the tissue, and the computer system increases the chances that the medical instrument applies sufficient energy to perform a procedure when less force 802 is applied to the tissue.

[0113] As another example, the computer system may adjust the energy level 304 depending on the movement 804 or the tissue information 812. As the medical instrument moves to operate on different portions of tissue, these portions of tissue may have different consistencies, thicknesses, densities, treatment levels, oxygenation levels, etc. As a result, the computer system may determine the adjustment 308 to the energy level 304 that is suitable for the change in consistency, thickness, density, treatment level, or oxygenation level. For example, if the medical instrument moves to contact a portion of tissue with increased consistency, thickness, or density, the computer system may increase the energy level 304. Conversely, if the medical instrument moves to contact a portion of tissue with reduced consistency, thickness, or density, the computer system may reduce the energy level 304. As another example, if the medical instrument moves to contact a portion of tissue with increased treatment levels or reduced oxygenation levels, the computer system may reduce the energy level 304 to avoid further damaging the tissue. Conversely, if the medical instrument moves to contact a portion of tissue with reduced treatment levels or increased oxygenation levels, the computer system may increase the energy level 304. In this manner, the computer system adjusts the energy level for changing tissue types and conditions as the medical instrument moves.Attorney Docket No. P07031-WO (160618)

[0114] The computer system may also adjust the energy level 304 depending on the type of movement 804. For example, if the movement 804 indicates that the medical instrument is moving slowly, then the computer system may reduce the energy level 304 to avoid overburning or damaging the tissue. If the movement 804 indicates that the medical instrument is moving quickly, then the computer system may increase the energy level 304 to ensure that sufficient energy is applied to the tissue. As another example, the computer system may adjust the energy level 304 to accommodate different lengths 806 or directions 808 of movement. The computer system may adjust the energy level 304 for long strokes and short strokes. The computer system may also adjust the energy level for horizontal, vertical, or sideways strokes. In this manner, the computer system may adjust the energy level to accommodate the directionality and lengths of movement of the medical instrument.

[0115] The computer system may also adjust the energy level 304 depending on the repetition 810 of the movement 804. Generally, if the medical instrument is being moved repetitively over the same area of tissue, the computer system may detect this repetitive movement and determine that insufficient energy is being applied to the tissue, which results in the operator moving the medical instrument repeatedly over the same area of tissue to apply energy repeatedly to that area. In response, the computer system may increase the energy level 304 to avoid or reduce the repetition 810.

[0116] As discussed above, the computer system may request confirmation or input from the operator prior to making the adjustment 308 rather than making the adjustment 308 automatically. For example, the computer system may provide guidance (e.g., a message) to the operator about the adjustment 308 to the energy level 304, and the operator may adjust the energy level 304 manually. As another example, the computer system may request confirmation from the operator prior to automatically adjusting the energy level 304.

[0117] Figure 9 illustrates an example operation 900 for adjusting an energy level. Generally, the computer system may use fluorescent images (e.g., hyperspectral images) of the tissue to determine a temperature of the tissue. The computer system may then adjust the energy level of the medical instrument, which may increase or decrease the rate at which the temperature of the tissue increases. The goal of theAttorney Docket No. P07031-WO (160618)procedure may be to apply energy to the tissue until the tissue reaches a particular temperature (e.g., for ablation or coagulation). By adjusting the energy level based on the tissue temperature, the computer system may avoid or prevent the tissue from excessive bum.

[0118] The computer system begins by receiving the inputs 302. In the example of Figure 9, the inputs 302 include an image 902, which may be a fluorescent image. The image 902 may show the tissue with a dye that fluoresces when the tissue is illuminated with particular wavelengths of light. The manner in which the dye fluoresces may reveal a temperature 903 of the tissue. The computer system may analyze the image 902 to determine the temperature 903.

[0119] The computer system then determines an adjustment 308 to the energy level 304 of the medical instrument according to the temperature 903. For example, the computer system may compare the temperature 903 against a threshold temperature set for the particular procedure. When the temperature 903 begins nearing the threshold temperature, the computer system may begin reducing the energy level 304. When the temperature 903 reaches the threshold temperature, the computer system may stop the medical instrument from outputting energy to prevent the temperature 903 from increasing further. In this manner, the computer system adjusts the energy level 304 to control for the temperature 903 of the tissue, which may prevent the tissue from experiencing excessive burning or damage.

[0120] The computer system may request confirmation or input from the operator prior to making the adjustment 308 rather than making the adjustment 308 automatically. For example, the computer system may provide guidance (e.g., a message) to the operator about the adjustment 308 to the energy level 304, and the operator may adjust the energy level 304 manually. As another example, the computer system may request confirmation from the operator prior to automatically adjusting the energy level 304.

[0121] Figure 10 illustrates an example operation 1000 for adjusting an energy level. Some medical instruments (e.g., vessel sealers) have jaws that grab tissue before applying energy to the tissue. When these instruments are used, the computer system may analyze an image or video of the surgical environment to determine anAttorney Docket No. P07031-WO (160618)angle of the jaw of these instruments, which may serve as a proxy for the thickness of tissue grabbed by the instrument. The computer system may then adjust the energy output of the instrument appropriate for the determined jaw angle or thickness of tissue.

[0122] The computer system begins by receiving the inputs 302, which may include an image 1002. The image 1002 may show the surgical environment, including the tissue and the medical instrument. The computer system may analyze the image 1002 to determine an instrument pose 1004 of the medical instrument. In the example of Figure 10, the pose 1004 may indicate a position 1006, an angle 1008, or an orientation 1010 of the medical instrument. The position 1006 may include coordinates that indicate a location of the medical instrument in three-dimensional coordinate space. The orientation 1010 may include coordinates or vectors that indicate a direction in which the medical instrument is pointed or oriented. The angle 1008 may include coordinates or vectors that indicate an angle of a jaw or the medical instrument. As discussed above, the angle of the jaw may indicate a thickness of tissue that the medical instrument may grab.

[0123] The computer system uses the instrument pose 1004 to determine a thickness 1012 of tissue grabbed by the medical instrument. For example, the position 1006 and the orientation 1010 may indicate the location of the medical instrument. The computer system may use this location to determine the tissue near the medical instrument. The computer system may then use tissue information for the tissue near the medical instrument along with the angle 1008 to determine the thickness 1012 of the tissue grabbed by the medical instrument.

[0124] The computer system then determines the adjustment 308 to the energy level 304 that is suitable for the thickness 1012 of the tissue grabbed by the medical instrument. For example, if the grabbed tissue is thick, then the computer system may increase the energy level 304. If the grabbed tissue is thin, then the computer system may reduce the energy level 304 (e.g., to avoid damaging or burning the tissue). In this manner, the computer system adjusts the energy level of the medical instrument to accommodate the thickness of tissue that the medical instrument grabs.Attorney Docket No. P07031-WO (160618)

[0125] The computer system may request confirmation or input from the operator prior to making the adjustment 308 rather than making the adjustment 308 automatically. For example, the computer system may provide guidance (e.g., a message) to the operator about the adjustment 308 to the energy level 304, and the operator may adjust the energy level 304 manually. As another example, the computer system may request confirmation from the operator prior to automatically adjusting the energy level 304.

[0126] Figure 11 illustrates an example operation 1100 for generating a recommendation. Generally, the computer system may provide guidance to the operator before and during the procedure. For example, the computer system may assess the surgical environment (e.g., tissue type, treatment level, etc.) and the medical procedure to be performed (e.g., dissection, sealing, repair, ablating, etc.) to recommend a medical instrument and energy level suitable for performing the procedure. As another example, the computer system may recommend certain movements of the medical instrument, such as the speed at which the medical instrument should be moved over the tissue, the angle at which the medical instrument should be moved over the tissue, or the type of stroke (e.g., long stroke, short stroke, sideways stroke, vertical stroke) that should be performed using the medical instrument. As another example, the computer system may recommend a certain amount of force to be applied by the medical instrument to the tissue during the procedure. In some instances, the computer system may also recommend the energy output of the medical instrument. The operator may need to confirm the energy output before the computer system changes the energy output.

[0127] The computer system begins by receiving the inputs 302, which may include one or more images of the surgical environment. The computer system uses the inputs 302 to determine information about the surgical environment and the medical instrument. For example, the computer system may use any operation (e.g., the operations shown in Figures 4 through 10) to determine tissue information 1102, a treatment level 1104, an oxygenation level 1106, or a procedure 1108. The tissue information 1102 may indicate a type, consistency, thickness, or density of tissue on which the medical instrument is operating. The treatment level 1104 and the oxygenation level 1106 may indicate a health of the tissue. For example, the treatmentAttorney Docket No. P07031-WO (160618)level 1104 or oxygenation level 1106 may indicate how burned or damaged the tissue is. The procedure 1108 may indicate the type of medical procedure to be performed or being performed. For example, the procedure 1108 may indicate the medical instrument is being used to cut, repair, seal, coagulate, etc.

[0128] The computer system uses the information determined from the inputs 302 to determine or generate a recommendation 1110. Generally, the recommendation 1110 may pertain to any aspect of the medical procedure. For example, the computer system may consider the procedure 1108 and the tissue information 1102 to recommend an instrument 1112 that is suitable for performing the procedure 1108 on the tissue type indicated by the tissue information 1102. As another example, the computer system may consider the tissue information 1102, the treatment level 1104, and the oxygenation level 1106 to recommend an energy level 1114 for the medical instrument, which may avoid overburning or damaging the tissue.

[0129] As another example, the computer system may also recommend various movements of the medical instrument based on the tissue information 1102, treatment level 1104, oxygenation level 1106, or procedure 1108. For example, the computer system may recommend a speed 1116 or angle 1118 at which the medical instrument should be moved to perform the procedure 1108 while reducing or minimizing further damage. As another example, the computer system may recommend a type of stroke 1120 (e.g., linear stroke, short poke, etc.) to be performed by the medical instrument. The computer system may also recommend a force 1122 to be applied by the medical instrument to the tissue or a length 1124 and direction 1126 of the movement of the medical instrument.

[0130] The computer system may communicate or display the recommendation 1110 to indicate to the operator how to set or move the medical instrument. Portions of the recommendation 1110 may be presented as a textual message, while portions of the recommendation 1110 may be presented pictorially or with images. In some embodiments, the computer system may automatically set or move the medical instrument according to the recommendation 1110 after receiving a confirmation 1128 of the recommendation 1110 from the operator. For example, the computer system may automatically set the energy level of the medical instrument to be theAttorney Docket No. P07031-WO (160618)recommended energy level 1114 after the operator responds with the confirmation 1128 for the energy level.

[0131] Figure 12 illustrates an example operation 1200 for controlling an instrument. Generally, the computer system may automatically move the medical instrument to perform a procedure. For example, the computer system may use a three-dimensional map of the tissue or a virtual map of the surgical environment along with kinematics data for the medical instrument to automatically guide or move the medical instrument to perform the procedure on the tissue. Additionally, the computer system may set the energy level of the medical instrument using any of the processes described above. The computer system may also adjust the energy level of the medical instrument while performing the procedure to reduce or prevent excessive damage to the tissue. In this manner, the computer system automatically performs the procedure, removing the need for operator input, which may be advantageous for procedures like dissection that are seen as tedious and time consuming but are nevertheless necessary.

[0132] The computer system begins with a map 1202 of the surgical environment or of the tissue. The map 1202 may be a three-dimensional map or virtual map generated using any technique (e.g., SLAM). The map 1202 may indicate the pose (e.g., location and orientation) of the medical instrument in the surgical environment. The map 1202 may also indicate damage to the tissue (e.g., treatment level, oxygenation level, etc.) and where the damage occurs on the tissue. Using the map 1202 the computer system may determine where the medical instrument should be moved to apply energy to the tissue.

[0133] The computer system may determine (e.g., from the map 1202) an energy level 1204 and a movement 1206 of the medical instrument to perform a procedure. For example, the computer system may use the map 1202 to determine a path for the medical instrument to follow to perform a procedure (e.g., cut the tissue). The computer system may also use the map 1202 to determine existing damage to the tissue along the path. The computer system may then determine the energy level 1204 for the medical instrument along the path (which may be a non-constant energy level 1204) to perform the procedure while minimizing damage to the tissue.Attorney Docket No. P07031-WO (160618)

[0134] The computer system then generate an instruction 1208 and communicates the instruction 1208 to the instrument 12010. The instruction 1208 may cause the instrument 1210 to move along the tissue (e.g., along the determined path). Additionally, the instruction 1208 may cause the instrument 1210 to set and adjust the energy level according to the determined energy level 1204. As the instrument 1210 moves along the path, the instrument 1210 may change the energy level to accommodate changing tissue types and damage. In this manner, the computer system automatically moves the instrument 1210 and automatically sets and adjusts the energy level of the instrument 1210.

[0135] Figures 13A and 13B illustrate example operations for controlling an instrument. Generally, the computer system may use artificial intelligence to set the energy output of the medical instrument based on the various inputs described above. For example, a model may be trained using information from previous procedures. The information for a procedure (e.g., tissue type, surgical task, instrument type, treatment level, oxygenation level, force, temperature, etc.) may then be input to the model, and the model may output the energy level for the medical instrument. The computer system may then set the energy level of the medical instrument accordingly.

[0136] Figure 13A shows an example operation 1300 for training a machine learning model. The computer system begins with procedure data 1302, which may have been collected across multiple, previous procedures. The procedure data 1302 may include information about any aspect of the previous procedures. For example, the procedure data 1302 may indicate tissue information 1304 and the procedure 1306 that was performed. The procedure data 1302 may also indicate the instrument 1308 that was used during the procedure along with the treatment level 1310 and the oxygenation level 1312 of the tissue during the procedure. Additionally, the procedure data 1302 may indicate the force 1314 that the medical instrument applied to the tissue during the procedure and the temperature 1316 of the tissue during the procedure. The procedure data 1302 may also indicate the energy level 1318 produced by the instrument during the procedure.

[0137] The computer system uses the procedure data 1302 from previous procedures to train a model 1320. The model 1320 may thus be trained to predict various aspects of a procedure using input data about other aspects of the procedure.Attorney Docket No. P07031-WO (160618)For example, the model 1320 may be trained to predict an energy level to be produced by an instrument given tissue information and the procedure to be performed. As another example, the model 1320 may be trained to predict the energy level to use given the treatment level and oxygenation level of the tissue.

[0138] Figure 13B shows an example operation 1330 for using the machine learning model. Generally, the computer system gathers inputs 302 during or before a procedure. For example, the inputs 302 may include information about the procedure to be performed or being performed and tissue information. The computer system directs the inputs 302 to the model 1320, and the model 1320 predicts aspects of the procedure using the inputs 302. For example, the model 1320 may predict the energy level 304 to be used during the procedure given the inputs 302. The computer system may then set the energy level of the instrument according to the predicted energy level 304. In this manner, the computer system uses artificial intelligence to set or adjust the energy level of the instrument during a procedure as more inputs 302 are provided.

[0139] Figure 14 is a flowchart of an example method 1400 for adjusting an energy level. In particular embodiments, the computer system performs the method 1400. By performing the method 1400, the computer system adjusts an energy level of a medical instrument during a procedure.

[0140] At 1402, the computer system applies a first amount of energy using the instrument. For example, the computer system may be using the instrument to perform a procedure. The computer system may have set the energy level of the instrument to the first amount of energy (e.g., by communicating an instruction to the instrument). The computer system may then communicate instructions to the instrument to move the instrument across tissue to apply the first amount of energy to the tissue.

[0141] At 1404, the computer system receives inputs. The inputs may include any information about the procedure. For example, the inputs may include images of the surgical environment or of the tissue. The images may include fluorescent images that indicate the temperature, treatment level, or oxygenation level of the tissue as the procedure progresses. For example, a dye may be injected into the tissue, and theAttorney Docket No. P07031-WO (160618)dye may fluoresce differently depending on the temperature, treatment level, or oxygenation level of the tissue. At 1406, the computer system analyzes the inputs (e.g., using computer vision techniques to analyze the images) to determine a treatment level of the tissue.

[0142] At 1408, the computer system adjusts an energy level of the instrument according to the determined treatment level. For example, if the treatment level is excessive, the computer system may reduce the energy level of the instrument to reduce or slow down the burning of the tissue. If the treatment level is low, the computer system may increase the energy level of the instrument. In this manner, the computer system changes the energy level of the instrument from the first amount of energy to a second amount of energy. At 1410, the computer system uses the instrument to apply the second amount of energy to the tissue. For example, the computer system may communicate an instruction to the instrument to adjust the energy level of the instrument from the first amount of energy to the second amount of energy. The instruction may also cause the instrument to move over the tissue to apply the second amount of energy to the tissue.

[0143] The computer system may request confirmation or input from the operator prior to making the adjustment rather than making the adjustment automatically. For example, the computer system may provide guidance (e.g., a message) to the operator about the adjustment to the energy level, and the operator may adjust the energy level manually. As another example, the computer system may request confirmation from the operator prior to automatically adjusting the energy level.

[0144] In summary, the computer system sets and adjusts the energy output of a medical instrument. The computer system may receive multiple inputs, such as images of the tissue, the type of medical procedure to be performed (e.g., dissection, sealing, repair, ablation, etc.), the type of medical instrument to be used (e.g., bipolar, monopolar, etc.), a pose of the medical instrument (e.g., a position or an angle of the medical instrument, which may indicate a thickness or an amount of tissue grabbed by the medical instrument), an amount of force applied by the medical instrument on the tissue, or information about the tissue (e.g., a model of the tissue, depth, etc.). The computer system may use some or all of this information to determine an amountAttorney Docket No. P07031-WO (160618)of energy to be output by the medical instrument to perform the medical procedure on the tissue.

[0145] While the medical instrument is applying energy to the tissue, the computer system may analyze images of the tissue (e.g., camera images, hyperspectral / dragonfly / firefly images, ultrasound images, etc.) to assess damage caused by the medical instrument to the tissue. For example, the computer system may determine a treatment level of the tissue (e.g., temperature of the tissue, size of a burned area on the tissue, etc.), oxygenation level of the tissue, or an amount of smoke. The computer system may then adjust the amount of energy output by the medical instrument according to the determined damage and according to any of the information discussed above (e.g., a movement of the medical instrument, a change in force applied by the medical instrument on the tissue, etc.).

[0146] This description and the accompanying drawings that illustrate aspects, embodiments, or modules should not be taken as limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this description and the claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail in order not to obscure other features. Like numbers in two or more figures represent the same or similar elements.

[0147] In this description, specific details are set forth describing some embodiments consistent with the present disclosure. Numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless specifically described otherwise or if the one or more features would make an embodiment non-functional.Attorney Docket No. P07031-WO (160618)

[0148] Further, the terminology in this description is not intended to be limiting. For example, spatially relative terms-such as “beneath”, “below”, “lower”, “above”, “upper”, “proximal”, “distal”, and the like may be used to describe one element’s or feature’s relationship to another element or feature as illustrated in the figures. These spatially relative terms are intended to encompass different positions (i.e., locations) and orientations (i.e., rotational placements) of the elements or their operation in addition to the position and orientation shown in the figures. For example, if the content of one of the figures is turned over, elements described as “below” or “beneath” other elements or features would then be “above” or “over” the other elements or features. Thus, the exemplary term “below” can encompass both positions and orientations of above and below. A device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly. Likewise, descriptions of movement along and around various axes include various special element positions and orientations. In addition, the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context indicates otherwise. And, the terms “comprises”, “comprising”, “includes”, and the like specify the presence of stated features, steps, operations, elements, or components but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, or groups. Components described as coupled may be electrically or mechanically directly coupled, or they may be indirectly coupled via one or more intermediate components.

[0149] Elements described in detail with reference to one embodiment, or module may, whenever practical, be included in other embodiments, or modules in which they are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment and is not described with reference to a second embodiment, the element may nevertheless be claimed as included in the second embodiment. Thus, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, or application may be incorporated into other embodiments, or aspects unless specifically described otherwise, unless the one or more elements would make an embodiment or embodiments non-functional, or unless two or more of the elements provide conflicting functions.Attorney Docket No. P07031-WO (160618)

[0150] In some instances, well known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0151] This disclosure describes various devices, elements, and portions of computer-assisted devices and elements in terms of their state in three-dimensional space. As used herein, the term “position” refers to the location of an element or a portion of an element in a three-dimensional space (e.g., three degrees of translational freedom along Cartesian x-, y-, and z-coordinates). As used herein, the term “orientation” refers to the rotational placement of an element or a portion of an element (three degrees of rotational freedom - e.g., roll, pitch, and yaw). As used herein, the term “shape” refers to a set positions or orientations measured along an element. As used herein, and for a device with repositionable arms, the term “proximal” refers to a direction toward the base of the computer-assisted device along its kinematic chain and “distal” refers to a direction away from the base along the kinematic chain.

[0152] Aspects of this disclosure are described in reference to computer-assisted systems and devices, which may include systems and devices that are teleoperated, remote-controlled, autonomous, semiautonomous, robotic, or the like. Further, aspects of this disclosure are described in terms of an embodiment using a medical system, such as the DA VINCI SURGICAL SYSTEM or ION SYSTEM commercialized by Intuitive Surgical, Inc. of Sunnyvale, California. Knowledgeable persons will understand, however, that aspects disclosed herein may be embodied and implemented in various ways, including robotic and, if applicable, non-robotic embodiments. Techniques described with reference to surgical instruments and surgical methods may be used in other contexts. Thus, the instruments, systems, and methods described herein may be used for humans, animals, portions of human or animal anatomy, industrial systems, general robotic, or teleoperational systems. As further examples, the instruments, systems, and methods described herein may be used for non-medical purposes including industrial uses, general robotic uses, sensing or manipulating non-tissue work pieces, cosmetic improvements, imaging of human or animal anatomy, gathering data from human or animal anatomy, setting up or taking down systems, training medical or non-medical personnel, or the like. Additional example applications include use for procedures on tissue removed from human orAttorney Docket No. P07031-WO (160618)animal anatomies (with or without return to a human or animal anatomy) and for procedures on human or animal cadavers. Further, these techniques can also be used for medical treatment or diagnosis procedures that include, or do not include, surgical aspects.

[0153] Although illustrative embodiments have been shown and described, a wide range of modification, change and substitution is contemplated in the foregoing disclosure and in some instances, some features of the embodiments may be employed without a corresponding use of other features. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. Thus, the scope of the disclosure should be limited only by the following claims, and it is appropriate that the claims be construed broadly and, in a manner, consistent with the scope of the embodiments disclosed herein.

Claims

1. Attorney Docket No. P07031-WO (160618)WHAT IS CLAIMED IS:

1. A computer system for adjusting an energy output of a medical instrument, the computer system comprising:one or more memories; andone or more processors communicatively coupled to the one or more memories, the one or more processors configured to, individually or collectively, perform an operation comprising:applying a first amount of energy to an anatomical structure using a medical instrument;determining, based on a first set of images of the anatomical structure, a treatment level of the anatomical structure caused by applying the first amount of energy to the anatomical structure using the medical instrument; and adjusting, based on the treatment level, an energy output of the medical instrument such that the medical instrument applies a second amount of energy to the anatomical structure.

2. The computer system of Claim 1, wherein the operation further comprises:determining, based on a second image of an anatomical structure, a tissue type for the anatomical structure; andsetting, based on the tissue type, the energy output of the medical instrument such that the medical instrument applies the first amount of energy to the anatomical structure.

3. The computer system of Claim 2, wherein the tissue type indicates at least one of a thickness, a consistency, or a density of the anatomical structure.

4. The computer system of Claim 2, wherein determining the tissue type for the anatomical structure is further based on a three-dimensional model of the anatomical structure or a fluorescent image of the anatomical structure.

5. The computer system of Claim 1, wherein the first set of images comprises a fluorescent image of the anatomical structure.Attorney Docket No. P07031-WO (160618)6. The computer system of Claim 1, wherein determining the treatment level comprises comparing the first set of images of the anatomical structure against a second image of the anatomical structure captured prior to the medical instrument applying the first amount of energy to the anatomical structure.

7. The computer system of Claim 1, wherein determining the treatment level comprises comparing the first set of images of the anatomical structure against an image of burned tissue.

8. The computer system of Claim 1, wherein determining the treatment level is further based on at least one of a color or a texture of the anatomical structure in the first set of images.

9. The computer system of Claim 1, wherein determining the treatment level is further based on a size of a burned area of the anatomical structure in the first set of images.

10. The computer system of Claim 1, wherein the operation further comprises determining, based on the first set of images of the anatomical structure, an oxygenation level of the anatomical structure and wherein adjusting the energy output of the medical instrument is further based on the oxygenation level.

11. The computer system of Claim 10, wherein the operation further comprises generating a map of the anatomical structure, wherein the map indicates at least one of the treatment level or the oxygenation level of the anatomical structure, and wherein adjusting the energy output of the medical instrument is further based on the map.

12. The computer system of Claim 11, wherein the operation further comprises adjusting the treatment level indicated by the map after the medical instrument applies the second amount of energy to the anatomical structure.

13. The computer system of Claim 11, wherein the operation further comprises determining, based on the map, a portion of the anatomical structure and wherein the medical instrument applies the second amount of energy to the portion of the anatomical structure.Attorney Docket No. P07031-WO (160618)14. The computer system of Claim 1, wherein the operation further comprises determining, based on the first set of images, a smoke level and wherein adjusting the energy output of the medical instrument is further based on the smoke level.

15. The computer system of Claim 1, wherein the operation further comprises determining, based on the first set of images, a movement of the medical instrument when applying the first amount of energy to the anatomical structure and wherein adjusting the energy output of the medical instrument is further based on the movement.

16. The computer system of Claim 15, wherein adjusting the energy output of the medical instrument is further based on a length of the movement.

17. The computer system of Claim 15, wherein adjusting the energy output of the medical instrument is further based on a direction of the movement.

18. The computer system of Claim 15, wherein adjusting the energy output of the medical instrument is further based on a repetition of the movement.

19. The computer system of Claim 15, wherein the operation further comprises determining that the movement causes the medical instrument to contact a portion of the anatomical structure with at least one of an updated tissue type, an updated treatment level, or an updated oxygenation level and wherein adjusting the energy output of the medical instrument is further based on at least one of the updated tissue type, the updated treatment level, or the updated oxygenation level.

20. The computer system of Claim 1, wherein the operation further comprises determining an amount of force applied by the medical instrument to the anatomical structure when applying the first amount of energy and wherein adjusting the energy output of the medical instrument is further based on the amount of force.

21. The computer system of Claim 1, wherein the operation further comprises determining, based on the first set of images, a temperature of the anatomical structure when the medical instrument is applying the first amount ofAttorney Docket No. P07031-WO (160618)energy to the anatomical structure and wherein adjusting the energy output of the medical instrument is further based on the temperature.

22. The computer system of Claim 1, wherein the operation further comprises determining, based on the first set of images, an angle of the medical instrument and wherein adjusting the energy output of the medical instrument is further based on the angle.

23. The computer system of Claim 1, wherein the operation further comprises generating a message indicating the second amount of energy and wherein adjusting the energy output of the medical instrument is based on a response to the message.

24. The computer system of Claim 1, wherein the operation further comprises:determining, based on a second image of an anatomical structure, a tissue type for the anatomical structure;determining, based on at least one of the tissue type or a procedure to be performed, that the medical instrument should be used; andbased on determining that the medical instrument should be used, generating a message indicating that the medical instrument should be used.

25. The computer system of Claim 24, wherein the operation further comprises determining, based on at least one of the tissue type or the procedure to be performed, a movement of the medical instrument and wherein the message further indicates the movement of the medical instrument.

26. The computer system of Claim 25, wherein the movement of the medical instrument comprises at least one of a speed at which the medical instrument should be moved, an angle at which the medical instrument should be moved, a length of the movement of the medical instrument, or a direction of the movement of the medical instrument.

27. The computer system of Claim 1, wherein the operation comprises moving the medical instrument over the anatomical structure such that the medical instrument applies the first amount of energy to the anatomical structure.Attorney Docket No. P07031-WO (160618)28. The computer system of Claim 1, wherein the operation further comprises determining, using a machine learning model and based on the treatment level, the second amount of energy.

29. The computer system of Claim 1 , wherein applying the first amount of energy to the anatomical structure dissects the anatomical structure.

30. The computer system of Claim 1 , wherein applying the first amount of energy to the anatomical structure coagulates the anatomical structure.

31. The computer system of Claim 1 , wherein applying the first amount of energy to the anatomical structure ablates the anatomical structure.

32. The computer system of Claim 1, wherein the operation further comprises determining, based on a second set of images, a portion of the anatomical structure to be ablated and wherein the first amount of energy is applied by the medical instrument to the portion of the anatomical structure.

33. A method for adjusting an energy output of a medical instrument, the method comprising:applying a first amount of energy to an anatomical structure using a medical instrument;determining, based on a first set of images of the anatomical structure, a treatment level of the anatomical structure caused by applying the first amount of energy to the anatomical structure using the medical instrument; andadjusting, based on the treatment level, an energy output of the medical instrument such that the medical instrument applies a second amount of energy to the anatomical structure.

34. The method of Claim 33, further comprising:determining, based on a second image of an anatomical structure, a tissue type for the anatomical structure; andsetting, based on the tissue type, the energy output of the medical instrument such that the medical instrument applies the first amount of energy to the anatomical structure.Attorney Docket No. P07031-WO (160618)35. The method of Claim 34, wherein the tissue type indicates at least one of a thickness, a consistency, or a density of the anatomical structure.

36. The method of Claim 34, wherein determining the tissue type for the anatomical structure is further based on a three-dimensional model of the anatomical structure or a fluorescent image of the anatomical structure.

37. The method of Claim 33, wherein the first set of images comprises a fluorescent image of the anatomical structure.

38. The method of Claim 33, wherein determining the treatment level comprises comparing the first set of images of the anatomical structure against a second image of the anatomical structure captured prior to the medical instrument applying the first amount of energy to the anatomical structure.

39. The method of Claim 33, wherein determining the treatment level comprises comparing the first set of images of the anatomical structure against an image of burned tissue.

40. The method of Claim 33, wherein determining the treatment level is further based on at least one of a color or a texture of the anatomical structure in the first set of images.

41. The method of Claim 33, wherein determining the treatment level is further based on a size of a burned area of the anatomical structure in the first set of images.

42. The method of Claim 33, further comprising determining, based on the first set of images of the anatomical structure, an oxygenation level of the anatomical structure and wherein adjusting the energy output of the medical instrument is further based on the oxygenation level.

43. The method of Claim 42, further comprising generating a map of the anatomical structure, wherein the map indicates at least one of the treatment level or the oxygenation level of the anatomical structure, and wherein adjusting the energy output of the medical instrument is further based on the map.Attorney Docket No. P07031-WO (160618)44. The method of Claim 43, further comprising adjusting the treatment level indicated by the map after the medical instrument applies the second amount of energy to the anatomical structure.

45. The method of Claim 43, further comprising determining, based on the map, a portion of the anatomical structure and wherein the medical instrument applies the second amount of energy to the portion of the anatomical structure.

46. The method of Claim 33, further comprising determining, based on the first set of images, a smoke level and wherein adjusting the energy output of the medical instrument is further based on the smoke level.

47. The method of Claim 33, further comprising determining, based on the first set of images, a movement of the medical instrument when applying the first amount of energy to the anatomical structure and wherein adjusting the energy output of the medical instrument is further based on the movement.

48. The method of Claim 47, wherein adjusting the energy output of the medical instrument is further based on a length of the movement.

49. The method of Claim 47, wherein adjusting the energy output of the medical instrument is further based on a direction of the movement.

50. The method of Claim 47, wherein adjusting the energy output of the medical instrument is further based on a repetition of the movement.

51. The method of Claim 47, further comprising determining that the movement causes the medical instrument to contact a portion of the anatomical structure with at least one of an updated tissue type, an updated treatment level, or an updated oxygenation level and wherein adjusting the energy output of the medical instrument is further based on at least one of the updated tissue type, the updated treatment level, or the updated oxygenation level.

52. The method of Claim 33, further comprising determining an amount of force applied by the medical instrument to the anatomical structure when applying the first amount of energy and wherein adjusting the energy output of the medical instrument is further based on the amount of force.Attorney Docket No. P07031-WO (160618)53. The method of Claim 33, further comprising determining, based on the first set of images, a temperature of the anatomical structure when the medical instrument is applying the first amount of energy to the anatomical structure and wherein adjusting the energy output of the medical instrument is further based on the temperature.

54. The method of Claim 33, further comprising determining, based on the first set of images, an angle of the medical instrument and wherein adjusting the energy output of the medical instrument is further based on the angle.

55. The method of Claim 33, further comprising generating a message indicating the second amount of energy and wherein adjusting the energy output of the medical instrument is based on a response to the message.

56. The method of Claim 33, further comprising:determining, based on a second image of an anatomical structure, a tissue type for the anatomical structure;determining, based on at least one of the tissue type or a procedure to be performed, that the medical instrument should be used; andbased on determining that the medical instrument should be used, generating a message indicating that the medical instrument should be used.

57. The method of Claim 56, further comprising determining, based on at least one of the tissue type or the procedure to be performed, a movement of the medical instrument and wherein the message further indicates the movement of the medical instrument.

58. The method of Claim 57, wherein the movement of the medical instrument comprises at least one of a speed at which the medical instrument should be moved, an angle at which the medical instrument should be moved, a length of the movement of the medical instrument, or a direction of the movement of the medical instrument.

59. The method of Claim 33, further comprising moving the medical instrument over the anatomical structure such that the medical instrument applies the first amount of energy to the anatomical structure.Attorney Docket No. P07031-WO (160618)60. The method of Claim 33, further comprising determining, using a machine learning model and based on the treatment level, the second amount of energy.

61. The method of Claim 33, wherein applying the first amount of energy to the anatomical structure dissects the anatomical structure.

62. The method of Claim 33, wherein applying the first amount of energy to the anatomical structure coagulates the anatomical structure.

63. The method of Claim 33, wherein applying the first amount of energy to the anatomical structure ablates the anatomical structure.

64. The method of Claim 33, further comprising determining, based on a second set of images, a portion of the anatomical structure to be ablated and wherein the first amount of energy is applied by the medical instrument to the portion of the anatomical structure.

65. A non-transitory machine-readable medium storing instructions for adjusting an energy output of a medical instrument that, when executed by one or more processors, cause the one or more processors to, individually or collectively: perform the method of any of Claims 33 through 64.

Images