Surgical guidance based on automatic point selection and tracking

Abstract

Systems and methods are described for performing a clinical procedure utilizing sentiment analysis. The system includes an imaging system and a control system. The system is configured to obtain image data of a tissue region of a patient from the imaging system; obtain input data indicative of an assessment to perform on the tissue region; analyze the input data and the obtained image data to select one or more target points in the obtained image data to track for performing the indicated assessment; track, via a point tracking model and across image data obtained via a second imaging modality of the imaging system, the one or more target points for an assessment period to determine one or more tissue parameter values for each tracked target point; provide an output to a user based on (i) the input data, and (ii) the one or more tissue parameter values.

Description

Intuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PCSURGICAL GUIDANCE BASED ON AUTOMATIC POINT SELECTION AND TRACKINGCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of the filing date of provisional U.S. Patent Application No. 63 / 714,409 entitled “SURGICAL GUIDANCE BASED ON AUTOMATIC POINT SELECTION AND TRACKING,” filed on October 31, 2024. The entire contents of the provisional application are hereby expressly incorporated herein by reference.TECHNICAL FIELD

[0002] The present disclosure relates generally to computer-assisted systems for assessing tissue properties, and more particularly, to automatically assessing tissue properties and providing guidance or feedback based on the tissue properties during a clinical procedure.BACKGROUND

[0003] Medical procedures often require various tasks to be performed by individuals or devices. Computer-assisted manipulator systems (“manipulator systems”), sometimes referred to as robotically assisted systems or robotic systems, may include one or more medical devices, equipment, and sensors that can be operated with the assistance of an electronic controller (e.g., computer or control system) to move and control functions of one or more instruments or medical devices alongside individuals and clinicians for performing a procedure.

[0004] Typically, individuals are required to perform tasks along with robotic systems either by controlling manipulators or providing inputs and feedback to the systems. As one example, a trained clinician often performs analysis of images or tissue obtained from an imaging system of the robotic system to determine how to perform a procedure (e.g., by determining a surgical plane to bisect the tissue). This requires that the clinicians stop the procedure to perform the assessment. Additionally, the robotic system may require the robotic system be operated in a different mode to receive the user inputs based on this analysis (e.g., by switching between a leader-device control mode and a user-input mode). For some procedures, this may need to occur multiple times. Thus, the conventional process is prone to delay when receiving clinician assessments of how to perform the procedure. Moreover, this assessment is subjective and cannot be readily reproduced by other, less-skilled clinicians.Intuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC

[0005] Accordingly, there is a need for improved techniques for performing tissue property assessments in clinical environments. Such techniques can allow for improved surgical outcomes, increased task and procedure efficiencies, and may increase the availability of such analyses across a broader set of environments and settings.SUMMARY

[0006] The following presents a simplified summary of various examples described herein and is not intended to identify key or critical elements or to delineate the scope of the claims.

[0007] In some aspects, the techniques described herein relate to a computer-assisted system for assessing tissue properties in a clinical procedure, the system including (i) an imaging system including one or more image sensors, the imaging system configured to operate in two or more imaging modalities to generate image data, wherein a first imaging modality of the one or more imaging modalities is associated with white illumination light, and (ii) a control system operably coupled to the imaging system, wherein the control system is configured to: obtain the image data generated by the imaging system; obtain input data indicative of an assessment to perform on a tissue region of a patient; analyze, using a point selection machine learning model, the input data and the obtained image data to select one or more target points in the obtained image data to track for performing the indicated assessment; track, via a point tracking model and across image data obtained via a second imaging modality of the imaging system, the one or more target points for an assessment period to determine one or more tissue parameter values for each tracked target point; provide an output to a user based on (i) the input data, and (ii) the one or more tissue parameter values.

[0008] In some aspects, the techniques described herein relate to a method for assessing tissue properties in clinical procedures via a computer-assisted system comprising (i) an imaging system including one or more image sensors, wherein the imaging system is configured to operate in two or more imaging modalities to generate image data, wherein a first imaging modality of the one or more imaging modalities is associated with white illumination light, and (ii) a control system operatively coupled to the imaging system, the method including: obtaining the image data generated by the imaging system; obtaining input data indicative of an assessment to perform on a tissue region of a patient; analyzing, using a point selection machine learning model, the input data and the obtained image data to select one or more target points in the obtained image data toIntuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC track for performing the indicated assessment; tracking, via a point tracking model and across image data obtained via a second imaging modality of the imaging system, the one or more target points for an assessment period to determine one or more tissue parameter values for each tracked target point; providing an output to a user based on (i) the input data, and (ii) the one or more tissue parameter values.

[0009] In some aspects, the techniques described herein relate to a computer-readable media storing instructions that, when executed by a control system of a computer-assisted system, causes the control system to perform any of the methods described herein.

[0010] It is to be understood that both the foregoing general description and the following detailed description are illustrative and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In that regard, additional aspects, features, and advantages of the present disclosure will be apparent to one skilled in the art from the following detailed description.BRIEF DESCRIPTION OF THE DRAWINGS

[0011] FIG. 1 is a diagram of a computer-assisted system in accordance with one or more embodiments.

[0012] FIG. 2 is a schematic diagram of a system for assessing tissue properties in clinical procedures in some examples.

[0013] FIG. 3 is an image of a tissue region with selected target points for performing tissue assessment during a clinical procedure.

[0014] FIGs. 4A - 4B depict user interfaces that present a tissue assessment and guidance in view of the same.

[0015] FIG. 5 is a flow diagram of a method for assessing tissue properties for clinical procedures.

[0016] Examples of the present disclosure and their advantages are best understood by referring to the detailed description that follows. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures, wherein showings therein are for purposes of illustrating examples of the present disclosure and not for purposes of limiting the same.Intuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PCDETAILED DESCRIPTION

[0017] In this description, specific details are set forth describing some embodiments consistent with the present disclosure. Numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless specifically described otherwise or if the one or more features would make an embodiment non-functional.

[0018] Further, the terminology in this description is not intended to limit the invention. For example, spatially relative terms-such as “beneath”, “below”, “lower”, “above”, “upper”, “proximal”, “distal”, and the like-may be used to describe the relation of one element or feature to another element or feature as illustrated in the figures. These spatially relative terms are intended to encompass different positions (z.e., locations) and orientations (z.<?., rotational placements) of the elements or their operation in addition to the position and orientation shown in the figures. For example, if the content of one of the figures is turned over, elements described as “below” or “beneath” other elements or features would then be “above” or “over” the other elements or features. A device may be otherwise oriented and the spatially relative descriptors used herein interpreted accordingly. Likewise, descriptions of movement along and around various axes include various special element positions and orientations. In addition, the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context indicates otherwise. Additionally, the terms “comprises”, “comprising”, “includes”, and the like specify the presence of stated features, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups. Components described as coupled may be electrically or mechanically directly coupled, or they may be indirectly coupled via one or more intermediate components.

[0019] Elements described in detail with reference to one embodiment, implementation, system, or module may, whenever practical, be included in other embodiments, implementations, systems, or modules in which they are not specifically shown or described. For example, if an element isIntuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC described in detail with reference to one embodiment and is not described with reference to a second embodiment, the element may nevertheless be claimed as included in the second embodiment. Thus, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, implementation, or application may be incorporated into other embodiments, implementations, or aspects unless specifically described otherwise, unless the one or more elements would make an embodiment or implementation non-functional, or unless two or more of the elements provide conflicting functions.

[0020] In some instances, well known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0021] This disclosure describes various devices, elements, and portions of computer-assisted systems and elements in terms of their state in three-dimensional space. As used herein, the term “position” refers to the location of an element or a portion of an element (e.g.. three degrees of translational freedom in a three-dimensional space, such as along Cartesian x-, y-, and z- coordinates). As used herein, the term “orientation” refers to the rotational placement of an element or a portion of an element (e.g., three degrees of rotational freedom in three-dimensional space, such as about roll, pitch, and yaw axes, represented in angle-axis, rotation matrix, quaternion representation, and / or the like). As used herein, and for a device with a kinematic series, such as with a repositionable structure with a plurality of links coupled by one or more joints, the term “proximal” refers to a direction toward a base of the kinematic series, and “distal” refers to a direction away from the base along the kinematic series.

[0022] As used herein, the term “pose” refers to the multi-degree of freedom (DOF) spatial position and orientation of a coordinate system of interest attached to a rigid body. In general, a pose includes a pose variable for each of the DOFs in the pose. For example, a full 6-DOF pose for a rigid body in three-dimensional space would include 6 pose variables corresponding to the 3 positional DOFs (e.g., x, y, and z) and the 3 orientational DOFs (e.g., roll, pitch, and yaw). A 3- DOF position only pose would include only pose variables for the 3 positional DOFs. Similarly, a 3-DOF orientation only pose would include only pose variables for the 3 rotational DOFs. Further, a velocity of the pose captures the change in pose over time (e.g., a first derivative of the pose). For a full 6-DOF pose of a rigid body in three-dimensional space, the velocity would include 3Intuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC translational velocities and 3 rotational velocities. Poses with other numbers of DOFs would have a corresponding number of velocities translational and / or rotational velocities.

[0023] This disclosure occasionally refers to the disclosed techniques being applied to “patients” undergoing a “medical procedure.” It should be appreciated that these references are not intended to limit the application of the disclosed techniques to applied medicine contexts. For example, the described techniques can be applied to facilitate physician training, equipment testing and / or calibration, and / or other contexts. Accordingly, any reference to the term “patient” is done for ease of explanation and also envisions the application of the described techniques to a generic “subject.”

[0024] The word “task” is used herein to refer to a discrete portion of procedure that may be autonomously, semi-autonomously, or manually implemented in furtherance of a procedure. For example, a task may be to move an endoscope to a particular portion, to advance an instrument to a particular depth, to replace an instrument coupled to a manipulator, and so on. In some embodiments, a task is associated with component tasks to accomplish an overall goal. For example, a task to analyze a worksite may include component tasks related to moving an endoscope to view the worksite, advancing an instrument to predetermined depth, and enabling a functionality supported by the instrument.

[0025] Aspects of this disclosure are described in reference to computer-assisted systems, which can include devices that are teleoperated, externally manipulated, autonomous, semiautonomous, and / or the like. Further, aspects of this disclosure are described in terms of an implementation using a teleoperated surgical system, such as the da Vinci® Surgical System commercialized by Intuitive Surgical. Inc. of Sunnyvale, California. Knowledgeable persons will understand, however, that inventive aspects disclosed herein may be embodied and implemented in various ways, including teleoperated and non-teleoperated, and medical and non-medical embodiments and implementations. Implementations on da Vinci® Surgical Systems are merely exemplary and are not to be considered as limiting the scope of the inventive aspects disclosed herein. For example, techniques described with reference to surgical instruments and surgical methods may be used in other contexts. Thus, the instruments, systems, and methods described herein may be used for humans, animals, portions of human or animal anatomy, industrial systems, general robotic, or teleoperated systems. As further examples, the instruments, systems, and methodsIntuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC described herein may be used for non-medical purposes including industrial uses, general robotic uses, sensing or manipulating non-tissue work pieces, cosmetic improvements, imaging of human or animal anatomy, gathering data from human or animal anatomy, setting up or taking down systems, training medical or non-medical personnel, and / or the like. Additional example applications include use for procedures on tissue removed from human or animal anatomies (with or without return to a human or animal anatomy) and for procedures on human or animal cadavers. Further, these techniques can also be used for medical treatment or diagnosis procedures that include, or do not include, surgical aspects.

[0026] FIG. 1 is a simplified diagram of an example computer-assisted system 100, according to various embodiments. The computer-assisted system 100 may be a computer-assisted medical system for assisting with performing tasks in furtherance of medical procedures. While FIG. 1 depicts a da Vinci® Surgical System, in other embodiments, the disclose techniques may be implemented in other surgical systems (e.g., a laparoscopic surgical system, a surgical system coupled with a tomosynthesis imager, etc.).

[0027] The computer-assisted system 100 may generate tasks that may be potentially performed in furtherance of the medical procedure and corresponding risk values associated therewith. In some examples, the computer-assisted system 100 is a teleoperated system. In medical examples, the computer-assisted system 100 can be a teleoperated medical system such as a surgical system. As shown, the computer-assisted system 100 includes a follower device 104 that can be teleoperated by being controlled by one or more leader devices (also called “leader input devices” when designed to accept external input), described in greater detail below. Systems that include a leader device and a follower device are referred to as leader-follower systems, and also sometimes referred to as master-slave systems. Also shown in FIG. 1 is an input system that includes a workstation 102 (e.g., a console), and in various embodiments the input system can be in any appropriate form and may or may not include the workstation 102.

[0028] In the example of FIG. 1, the workstation 102 includes one or more leader input devices 106 that are designed to be contacted and manipulated by an operator 108. For example, the workstation 102 may comprise one or more leader input devices 106 for use by the hands, the head, or some other body part(s) of operator 108. The leader input devices 106 in this example are supported by the workstation 102 and can be mechanically grounded. In some embodiments,Intuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC an ergonomic support 1 10 (e.g., forearm rest) can be provided on which the operator 108 can rest his or her forearms. In some examples, the operator 108 can perform tasks at a worksite within a workspace near the follower device 104 during a procedure, by commanding the follower device 104 using the leader input devices 106. In a medical example, the worksite may be a surgical worksite associated with a patient.

[0029] A display device 112 is also included in the workstation 102. The display device 112 may be configured to display images for viewing by the operator 108. The display device 112 can be moved in various DOFs to accommodate the viewing position of the operator 108 and / or to provide control functions. In embodiments where the display device 112 provides control functions, the leader input devices 106 may include the display device 112. In the example of the computer- assisted system 100, displayed images may depict a worksite at which the operator 108 is performing various tasks by manipulating the leader input devices 106 and / or the display device 112. In some examples, images displayed by display device 112 may be received by the workstation 102 from one or more imaging devices arranged at a worksite. In other examples, the images displayed by the display device 112 may be generated by the display device 112 (or by a different connected device or system), such as for virtual representations of tools, the worksite, or for user interface components. In some embodiments the display device 112 may display one or more tasks for the operator 108 to perform with respect to any component of the computer-assisted system 100. The display device 112 may display one or more options for a user to provide an input of a user selection such as a preference for a task to be performed manually by the user or another personnel, or for a preference for a task to be performed automatically by or semi- automatically using one or more repositionable structures described further herein.

[0030] As illustrated, the computer-assisted system 100 also includes a follower device 104 that can be commanded by the workstation 102. In a medical example, the follower device 104 can be located near an operating table (e.g., a table, bed, or other support) on which a patient can be positioned. In some medical examples, the workspace is provided on an operating table, e.g., on or in a patient, simulated patient, or model, training dummy, etc. (not shown). As illustrated, the follower device 104 may include a plurality of repositionable structures 120 (sometimes referred to as “manipulator arms” in robotic embodiments). In some embodiments, the repositionable structures 120 may include a plurality of links that are rigid members and joints that can be individually actuated as part of a kinematic series. Additionally, each of the repositionableIntuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC structures 120 is configured to couple to an instrument 122. While FIG. 1 illustrates a follower device 104 that has four repositionable structures 120a- 120d, in other embodiments, the follower device 104 may include one, two, three, four, five, six, or additional or fewer repositionable structures 120a-120d.

[0031] The instrument 122 can include, for example, a working portion 126 and one or more structures for supporting and / or driving the working portion 126. Example working portions 126 include end effectors that physically contact or manipulate material, energy application elements that apply electrical, RF, ultrasonic, or other types of energy, sensors that detect characteristics of the workspace environment (such as temperature sensors, imaging devices, etc.), and the like. In various embodiments, examples of instruments 122 include, without limitation, a sealing instrument, a cutting instrument, a sealing-and-cutting instrument, an energy instrument for applying energy, a gripping instrument (e.g., clamps, jaws), a stapler, an imaging instrument such as one using optical, RF, or ultrasonic imaging modalities, a sensing instrument, an irrigation instrument, a suction instrument, and / or the like. In addition, the instrument 122 may include a transmission mechanism 128 that can be coupled to a drive assembly 130 of the respective repositionable structure 120a- 120d. The drive assembly 130 may include a drive and / or other mechanisms controllable from workstation 102 that transmit forces to the transmission mechanism 128 to articular or otherwise actuate the instrument 122.

[0032] As illustrated, each instalment 122 may be mounted to a poaion of a respective repositionable structure 120a- 120d. In FIG. 1, this is shown with the drive assembly 130 physically coupled to the transmission mechanism 128. The distal portion of each repositionable structure 120a-120d further includes a cannula mount 124 to which a cannula (not shown) is mounted. When a cannula is mounted to the cannula mount 124, a shaft of the instrument 122 passes through the cannula and into a workspace.

[0033] In various embodiments, one or more of the working portions 126 of the instruments 122 may include an imaging device for capturing images for display via the display device 112. The imaging device may include any sensing technology capable of acquiring an image. Example imaging instruments include an optical endoscope, a hyperspectral camera, an ultrasonic sensor, etc. Imaging instruments may comprise monoscopic imagers, stereoscopic imagers, and / or the like. The imaging device may include an illumination source directed at the region being imaged.Intuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PCThe illumination source may be configured to emit light at a particular frequency and / or frequency spectrum to support one or more imaging modalities. For example, the illumination source may emit visible light (also referred to herein as “white light”), infrared illumination, ultraviolet illumination, illumination at a particular frequency that excites a fluorescent probe molecule, and / or the like. The imaging device may include a plurality of sensors or cameras that detect the illumination after reflecting off the region being imaged.

[0034] In some embodiments, the repositionable structures 120a-120d and / or instruments 122 can be controlled to move the working portion 126 in response to manipulation of the leader input devices 106 by the operator 108 which may be used to perform semi-automatic tasks with input from the operator. Accordingly, the repositionable structures 120a-120d and / or instruments 122 may be said to “follow” the leader input devices 106 through teleoperation. This enables the operator 108 to perform tasks at the worksite using the repositionable structures 120a-120d and / or instruments 122. For a surgical example, the operator 108 can direct the repositionable structures 120a-120d of the follower device 104 to move the working portions 126 as part of a surgical procedure performed at an internal surgical site that is entered via one or more minimally invasive apertures or natural orifices. It should be appreciated that, in some embodiments, the follower device 104 may include non-teleoperated components that the operator 108 or other medical professional must manually manipulate to a desired pose.

[0035] In some embodiments, a repositionable structure 120a of the computer-assisted system 100 may be configured to support a working portion 126a that includes an imaging device (also referred to herein as an “imaging device 126a”). For convenience, an instrument 122 that includes an imaging device is also referred to as an “imaging instrument” herein. The control system 140 may be configured to command the repositionable structure 120a and / or the imaging instrument 122 comprising the imaging device 126a to automatically position and / or orient (“pose”) the field of view (FOV) of the imaging device 126a to provide images of the workspace and / or other instruments 122.

[0036] In the illustrated embodiment, a control system 140 is communicatively coupled to the workstation 102. In other embodiments, the control system 140 may be provided as a component of the workstation 102 and / or the follower device 104. During teleoperation, as the operator 108 moves the leader input device(s) 106, one or more sensors configured to detect the leader inputIntuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC device(s) 106 generate spatial and / or orientation movement data that is provided to control system 140. The control system 140 may interpret the spatial and / or orientation information to determine and / or provide control signals to the follower device 104 to control the movement of repositionable structures 120a-120d, instruments 122, and / or working portions 126. In addition to the components of the follower device 104, in some embodiments, the control system 140 is configured to interpret inputs received from the workstation 102 to control operation of one or more auxiliary devices (not depicted) utilized in a procedure. For example, the workstation 102 may be used to control a pose of a surgical bed or operation of an insufflator. The workstation 102 may be used to provide user input for performing one or more manual or semi-automated tasks that require input from users or personnel.

[0037] In one embodiment, the control system 140 supports one or more wired communication protocols, (e.g., Ethernet. USB, and / or the like) and / or one or more wireless communication protocols (e.g., Bluetooth, IrDA, HomeRF, IEEE 1102.11, DECT, Wireless Telemetry, and / or the like) for communications between the control system 140 and the workstation 102 and / or the follower device 104.

[0038] In some embodiments, the control system 140 may be implemented at one or more computing systems. For example, one or more computing systems may be used to control the follower device 104. As another example, one or more computing systems may be used to control components of the workstation 102, such as movement of a display device 1 12. Collectively, these component computing systems may comprise the control system 140.

[0039] As illustrated, the control system 140 includes a processor system 150, a memory 160, a control module 170 and a machine learning module 180. The memory 160 may store the control module 170 and the machine learning module 180. The processor system 150 may include one or more processors having different processing architectures for processing instructions. For example, the one or more processors may be one or more cores or micro-cores of a multi-core processor, a central processing unit (CPU), a microprocessor, a field-programmable gate array (FPGA), an application- specific integrated circuit (ASIC), a digital signal processor (DSP), a graphics processing unit (GPU), a tensor processing unit (TPU), and / or the like.

[0040] In some embodiments, the processor system 150 includes circuity to support one or more communication interfaces (e.g., Bluetooth interface, infrared interface, network interface, opticalIntuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC interface, etc.). Additionally, a communication interface of control system 140 may include an integrated circuit for connecting the control system 140 to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, mobile network, or any other type of network) and / or to another device, such as the workstation 102 and / or the follower device 104.

[0041] Additionally, the memory 160 may include non-persistent storage (e.g. , volatile memory, such as random access memory (RAM), cache memory), persistent storage e.g., a hard disk, an optical drive such as a compact disk (CD) drive or digital versatile disk (DVD) drive, a flash memory, a floppy disk, a flexible disk, a magnetic tape, any other magnetic medium, any other optical medium, programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a FLASH-EPROM, and / or any other memory chip or cartridge. The non- persistent storage and persistent storage are examples of non-transitory, tangible machine-readable media that can store executable code that, when run by one or more processors (e.g., processor system 150), can cause the one or more processors to perform one or more of the techniques and / or methods disclosed herein.

[0042] The machine learning module 180 may implement one or more machine learning models and / or training routines therefor. For example, the machine learning module 180 may implement one or more neural networks, deep learning models, decision trees, support vector machines, linear regression, generative Al models, reinforced learning models, random forests, Naive Bayes models, large language models (LLMs), generative adversarial networks, foundation models, image recognition models, linear discriminant analysis models, creative applications, autoregressive models, supervised or unsupervised learning models, multimodal models, vision language models (VLMs), vision foundation models (VFMs), large multi-modal models (LMMs), Transformer models (including Robotic Transformer models), or another machine learning or Al model for performing the methods described herein. The machine learning module 180 may include dedicated processors and memory for storing and performing machine learning processes, or the machine learning module 180 may utilize resources of the processor system 150 and the memory 160 to store and / or perform any processing or tasks required to perform the methods described herein. In some embodiments, the training routines are executed by another computing system (e.g., a server system) and the trained machine learning models are loaded into the Al assist module 180 prior to performing the medical procedure.Intuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC

[0043] Additionally, the control system 140 may also include one or more input devices (such as a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, or any other type of input device) and / or output devices (such as a display device, a speaker, external storage, a printer, or any other output device). In some embodiments, the control system 140 may be implemented on a particular node of a distributed computing system (e.g., a cloud computing system). As another example, different functionalities associated with the control system 140 may be implemented on different nodes of the distributed computing system. Further, one or more elements of the aforementioned control system 140 may be located at a remote location and connected to the other elements over a network.

[0044] In an endoscopic surgery example, the imaging instrument comprising the imaging device 126a may be inserted into the patient prior to the other instruments 122, including a second instrument 122b comprising a second working portion 126b. The second instrument 122b can include any appropriate working portion 126b, and can even include a second imaging device. Accordingly, the imaging device 126a may be maneuvered to positioned to identify a target to which other instruments may interact with as part of another task. The control system 140 may, for example, automatically command the corresponding repositionable structures 120a and 120b to position respective instruments 122a and 122b to perform one or more tasks in tandem, or sequentially based on the specific task, instruments, and positions of the repositionable structures 120a and 120b.

[0045] In examples, the control system 140 may be configured to assess tissue properties for performing clinical procedures. The control system 140 may obtain images of a region of tissue of a patient and perform image processing on the images to track one or more points or targets in the images. The computer-assisted system 100 may include an imaging system including one or more cameras or imaging sensors may obtain the images over time to provide real-time video, or a time-series of images for performing the points or target tissues. The imaging system may obtain the images using one or more imaging modalities either serially or concurrently by multiple cameras. For example, the various imaging modalities may include a white light modality (e.g., wherein a camera or imaging sensor obtains images of the tissue region using white light illumination), an infrared illumination modality, a fluorescent light modality (e.g., wherein the imaging system is configured to image fluorescent light from the tissue region, such as the Firefly® Imaging System commercialized by Intuitive Surgical, Inc. of Sunnyvale, California), anIntuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC ultraviolet imaging modality, or another imaging modality. Tn some examples, the imaging system may further be configured to operate in an ultrasound imaging modality (e.g., via one or more ultrasound transducers obtain ultrasound images), a speckle imaging modality (e.g., via a modified ultrasound imaging system), an oxygenation detection imaging modality (e.g., via hyperspectral and / or multispectral imaging), x-ray imaging (e.g., via a tomosynthesis imaging sensor coupled to a C-arm), a probe-based confocal laser endomicroscopy (pCLE) imaging modality, a Raman spectroscopy imaging modality, and so on.

[0046] The control system 140 is further configured to obtain input data indicative of a tissue assessment or measurement to perform. For example, a user or clinician may verbally provide an instruction to perform a tissue oxygenation measurement, and the control system may obtain and analyze the verbally provided instructions using an automatic speech recognition (ASR) model. In other examples, the input data indicative of a tissue assessment to be performed may be provided via user input via a keyboard, mouse, touch screen device, or by instructions provided via one or more networks or systems. The control system 140 then analyzes the instructions and determines the target points, or target tissues, within the image data of the tissue region on which the control system 140 performs the tissue assessment. In examples, the input data may be indicative of preferences or preference data associated with how to obtain or assess the images. The preference data may include indications of preferred imaging modalities for capturing images, desired regions of tissue to perform a tissue assessment, a number of target points or control points for performing a tissue assessment, or another preference as may be provided by a user, or retrieved via a memory or other system. In some embodiments, the user may interface with the display device 112 (e.g., via touch, a mouse or other pointing device, a gaze tracking system, etc.) to provide input data indicative of a particular tissue region or point for performing the tissue assessment.

[0047] The input data may further include user input data or information that is indicative of a desired output of a tissue measurement or assessment. For example, the input data may include instructions to provide a visual output such as a reading (e.g., oxygenation levels, an indication of which tissue is health or non-cancerous, etc.) or a visual indication where to perform an interaction with subject anatomy (e.g., to perform a removal, an ablation, a biopsy, apply a treatment, etc). The input data may indicate a desired output to be a report of oxygenation level over time that may be stored in a memory or provided to another system or clinician. The input data may be indicativeIntuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC of a desired output to be an audio output that provides real-time feedback reporting tissue property levels to an operating room during a procedure.

[0048] It should be appreciated that in some embodiments, the image data displayed via the display device 112 and the assessment of tracked tissue may utilize different imaging modalities. For example, the input data may indicate that images obtained via a white light imaging modality should be displayed and overlaid tissue assessment data captured via a fluorescent imaging modality. The control system 140 may then overlay output data derived from the fluorescent imaging modality over the image data obtained via the white light imaging modality to provide a visual representation of the tissue assessment. It should be appreciated that control system 140 may capture data using both imaging modalities concurrently (or at least contemporaneously) such that the displayed image includes real-time data from both imaging modalities.

[0049] In some embodiments, the input data may also be provided by a hospital management system (e.g., a scheduling system, personnel management system, etc.), one or more sensors (e.g., a force sensor, one or more electrodes, a pressure sensor, a camera, etc.), a robotic system (e.g., a repositionable structure, an autonomous or semi-autonomous robotic system, etc.), a network, or a memory. As one example, the scheduling system may provide the control system 140 an indication of a procedure being performed using the computer-assisted system 100 such that the control system 140 is able to determine an expected configuration of imaging modalities to present to the operator. As another example, the control system 140 may determine that sensor data from the robotic system may indicates an anomalous condition and recommend a configuration of imaging modalities to assess the anomalous condition.

[0050] As described herein, many tissue assessment techniques involve monitoring parameter values over time to, for example, distinguish between healthy tissue and tissue to be transected. Because the image data is captured and assessed in real-time, the tissue is not static within the field of view of the imaging device. Accordingly, the control system 140 may implement point tracking techniques when assessing the tissue to ensure that the assessment of an individual point is associated with the same point across the assessment period.

[0051] To set up the tissue assessment, the control system 140 may select a number of points to track when performing the tissue assessment. In some embodiments, the control system 140 implements a point selection machine learning model to select the points to track for performingIntuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC the tissue assessment. The point selection machine learning model may analyze the input data (e.g., a user-indicated point) and one or more obtained images to select target points of tissues in the obtained images. The point selection model may include a vision language model (VLM) to analyze the images and determine the target points in the images. The number of points and location thereof may be determined based on the type of tissue assessment to be performed. For example, the control system 140 may identify a region of the tissue that is likely to include a border between healthy tissue and unhealthy tissue and select a higher resolution of points to track proximate to this border region.

[0052] The automatic selection of points provides several improvements to the tissue assessment process. As described above, the conventional process requires users to manually select points. Because each point must be actively selected and categorized by the user, this limits the number of points that can be tracked. On the other hand, when implementing the instant point selection techniques, the number of points is limited only by the imaging processing compute and image resolution. Thus, the instant techniques are able to more precisely define the border between healthy and unhealthy tissue, resulting in improved patient outcomes, particularly for procedures that transect the unhealthy tissue.

[0053] To implement the automatic point selection techniques, the control system 140 (e.g., via the point selection machine learning model) may determine various regions of tissue for selecting points within given a specific desired tissue assessment. For example, the control system 140 may implement a coarse image analysis technique that segments the image data captured via a single imaging modality into different regions. In some embodiments, the coarse image assessment may be based upon a quantitative value (e.g., based on color, histogram, intensity, reflectivity, and / or any other algorithmic filter that can be applied to the image data) exceeding a threshold value. Additionally or alternatively, the coarse image assessment may implement pattern recognition (e.g., to detect spotting or other patterns indicative of tissue health) and / or feature detection (e.g., to detection lesions, cancerous cells, or other anatomical features indicative of tissue health).

[0054] Based on the coarse image assessment, the control system 140 may define a first region of tissue associated with healthy tissue and a second region of tissue associated with unhealthy tissue. The control system 140 may then select target points within each region. As describedIntuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC above, the control system 140 may select a higher resolution of points for tracking closer to the border between the first and second regions (as determined based on the coarse image analysis).

[0055] For tissue assessments that are based on relative measurements, the control system 140 (e.g., via the point selection machine learning model) may additionally select a control point from the selected points relative to which the relative measurement is performed. The control point may be used to generate assessments representative of healthy tissue (e.g., tissue within the first region described in the example above). In some embodiments, the control system 140 analyzes the coarse measurements associated with the control point to set the threshold that segments the healthy and unhealthy regions of tissue. Additionally the control point may be used to compare tissue parameters to for determining one or more tasks to be completed such as incisions, extractions of tissue, regions of tissues for applying treatments, etc.

[0056] In some embodiments, the control system 140 may present the selected points to the user for approval. In some embodiments, this presentation includes a rationale for why the point selection model selected the presented point based on the coarse image analysis. In these embodiments, the point selection model may further include a text generation layer configured determine and output a natural-language rationale for the selection of the various selected points. In some embodiments, the rationale is responsive to the received input data. For example, the rationale may explain a particular metric is used to segment healthy and unhealthy tissue when performing a procedure type indicated by the input data.

[0057] After selecting the target points for tracking, the control system 140 may register the target points to a three-dimensional model of the tissue region. As described herein, the instruments 122 may be coupled to a repositionable structure 120 via which the control system 140 obtains kinematic data. By analyzing the kinematic data, the control system 140 is able to determine a portion of the model of the tissue region imaged by the imaging device. Accordingly, by registering the target points to the model of the tissue region, the control system 140 is able to correlate and / or track the selected target points across image data of the tissue region captured by the separate imaging devices from different fields of view. Even when multiple modalities are implemented as part of a single imaging device, the image sensors and / or illumination units may be physically offset from one another, causing the fields of view to be slightly different. Thus, even when both imaging modalities are implemented in a single imaging device, the registrationIntuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC is able to improve the accuracy of the tracking. In sum, the control system 140 may be able to assess the tissue region using additional imaging modalities while more accurately tracking of the target points across the assessment period.

[0058] After selecting and registering the one or more target points (including any control points), the control system 140 assess tissue characteristics associated with the target for an assessment period (e.g., 10 seconds, 30 seconds, 1 minute, 2 minutes, etc.). To perform the tissue assessment, the control system 140 analyzes the image data generated via an assessment imaging modality (e.g., fluorescent imaging, oxygenation imaging, ultrasound imaging, etc.) to generate a value for the tissue characteristic corresponding to the tracked point. For example, the tissue parameter values may include one or more values indicative of tissue viability, tissue health, tissue perfusion, tissue oxygenation, tissue elasticity, tissue strain, tissue impedance, a fluorescence measurement (e.g., an amount of a fluorescent molecule, such as indocyanine green (ICG)), etc.

[0059] To track the position of the selected target points throughout the assessment period, the control system 140 may execute a point tracking model. As described above, the point tracking model may track the target points across frames of image data obtained from a first imaging sensor operating in a first imaging modality and / or frames of image data obtained via a second imaging sensor operating in a second imaging modality. In some implementations, a first imaging device (e.g., a multi-modality endoscope) may be configured to operate in different imaging modalities including a first imaging modality (e.g., white light) and a third imaging modality (e.g., oxygenation). In these embodiments, the first imaging device may be configured to operate in the different modalities at different points of the assessment period (e.g., sequentially, interleaved, etc.).

[0060] The point tracking model may be implemented via known point tracking techniques. For example, the point tracking model may include a machine learning model that determines a probability that a given point on a successive frame of image data corresponds to a tracked point from a prior frame of image data. In some embodiments, the model of the tissue region is input into the point tracking model to improve the ability to track the point as the tissue moves closer and / or further away from the image sensors along the longitudinal axis.

[0061] The control system 140 may further be configured to provide an output associated with the tissue assessment to a user or clinician. The control system 140 may further provide the outputIntuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC to additional systems and networks, or store the output in a memory. The output may include indications of the target points and / or control point(s) and their corresponding tissue parameter values as an overlay of obtained image data.

[0062] In some embodiments, the control system 140 analyzes the tissue parameter values for the tracked points to provide guidance to the user. As one example, the guidance may include a recommendation for a surgical plane to transect the tissue. In this example, the tracked points and their corresponding tissue parameter values are input into a plane solver algorithm that classifies the target points as corresponding to healthy or unhealthy portions of the tissue and generates a surgical plane dividing the classes of points. It should be appreciated that the use of the word “plane” in the term surgical plane should not be understood to imply that the surgical plane is restricted to plane shapes. To this end, in view of the higher resolution of target points produced by the point selection model, the surgical plane may have any shape that segments the tissue region into healthy tissue and unhealthy tissue.

[0063] In addition to providing an indication of the surgical plane, the control system 140 may provide other types of guidance to the user, including, without limitation, reports of tissue parameter values (e.g., oxygenation levels, perfusion measurement values, etc.), indications of tissue assessment determined from tissue parameter values and obtained images (e.g., relative tissue health, tissue viability, etc.), and instructions or guidance for an action or task to be performed (e.g., locations for extractions and / or biopsies, incision paths, a path to be sutured, a location to staple, a location for ablation, a location for applying high-intensity -focused ultrasound (HiFU), a location for performing a treatment (e.g.. a therapeutic injection or electroporation), etc.). The output may further be provided via one or more means such as via a display (e.g., a monitor, laptop, medical display device, wearable device, tablet, etc.), an audio output device (e.g., speaker, headphones, wearable device, etc.), via text, or stored in a memory or provided to another system or device.

[0064] FIG. 2 is a schematic diagram of a system 200 for assessing tissue properties in clinical procedures as described herein. The system 200 may be a computer-assisted system, such as the system 100 of FIG. 1. The system includes a plurality of modules that may be executed by the control system 140 (e.g., via the machine learning module 180 of FIG. 1) to collect data, performIntuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC target point tracking, and assess tissue properties of a tissue region associated with a medical procedure.

[0065] The system 200 includes an imaging system (such as one or more instruments 122) and multiple sources of data 202. The system 200 includes sources of user input data 202a, audio data 202b, and video / image data 202c. The user input data 202a may be obtained via a user 201 via one or more user interfaces of a workstation 207 (such as the workstation 102). The audio data 202b may be obtained via one or more audio sensors located in the operating room. For example, the audio sensors may detect an instruction from the user 201 indicating a request that the system 200 “shows a surgical plane for performing a transection,” or “determines where to perform a suture.” The video / image data 202c may be obtained from by the imaging system. The video / image data 202c may be recorded using any of the imaging modalities for performing tissue assessments described herein. The medical system data source 202d may provide data associated with personnel for performing a procedure, a patient identification, medical history, etc. to adapt the analysis of the video / image data 202c in a manner that accounts for preferences of the user 201 and / or conditions of the patient.

[0066] Data from the data sources 202a-202d is provided to a point selection module 210 to determine one or more target points for performing tissue assessment. The point selection module 210 may implement an automatic speech recognition (ASR) model to convert the audio data 202b into text instructions for processing. Additionally, the point selection module 210 may include a vision-language model (VLM) to process the video / image data 202c to identify characteristics of the depicted tissue and / or provide a coarse assessment of the tissue health.

[0067] Based on the user instructions and the outputs of the VLM, the point selection module 210 may select a plurality of targets points in the tissue region to track when performing the tissue assessment. As described above, the point selection module 210 may select a first set of target points that correspond to healthy tissue and a second set of target points that corresponds to nonheal thy tissue.

[0068] After confirming the selection of the target points, the point tracking module 220 then tracks the points across frames of the video / image data 202c over the course of an assessment period. The point tracking module 220 may implement any suitable point tracking algorithm to track the target points across the frames of image data. In some embodiments, the duration of theIntuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC assessment period is derived from the user input data 202a and / or is pre-determined based upon the assessment type.

[0069] It should be appreciated that during the assessment period, the workstation 207 may present image data associated with a white light imaging modality while the system 200 analyzes data via a second imaging modality (e.g., fluorescent imaging, ultrasound imaging, oxygenation imaging) to assess the tissue health. In these embodiments, values of the tissue assessment parameter derived from the tissue assessment imaging modality may be overlaid onto the image data of the white light imaging modality as they are determined over the course of the assessment period.

[0070] The point tracking module 220 provides the tracked target points, and the corresponding tissue parameter values to an output / guidance module 240 to provide an output to a user. The output / guidance module 240 may provide the output as a visual output via a display. For example, the output may include measurements of tissue oxygenation level, and the output / guidance module 240 may provide the output as an image of the tissue region with oxygenation measurement values overlayed at various points of the image indicating oxygenation at the different tracked target points.

[0071] Additionally, the output / guidance module 240 may provide an output that is indicative of an action to be performed. In specific examples, the output / guidance module may provide a position and / or indication of a path for performing an incision, suture, transection, or another position for performing an injection or extraction. As such, the point tracking module 220 may further provide the tracked target points and their corresponding tissue assessment parameter values to a plane solver 230. The plane solver 230 may then determine a surgical plane output defining a path for performing the desired action. The plane output may include a start position, a path, and a final position for the action. Additionally, the plane output may be indicative of a nonlinear geometry, such as a non-linear path, for performing the action. The plane solver 230 may then provide the determined plane output to the output / guidance module 240 for presentation to the user 201 via the workstation 207 (e.g., visually via a display and overlay with image of a tissue region).

[0072] FIG. 3 depicts a set of target points 304 output from a point selection model. More particularly, FIG. 3 depicts image data 300 captured via an imaging modality described hereinIntuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC overlaid with indicators 304a-h corresponding to selected target points. While FTG. 3 depicts eight target points, other embodiments may have additional or fewer target points.

[0073] As illustrated, the target points corresponding to indicators 304a-c are associated with tissue that appears darker in the image data 300 than the tissue associated with the target points corresponding to the indicators 304d-h. Accordingly, when the point selection model (such as the point selection model 210) applied the coarse image analysis of the image data 300, the point selection model selection model applied an intensity threshold that coarsely separated the tissue region into a healthy region (the darker regions) and unhealthy regions (the lighter regions) and selected target points within both regions. It should be appreciated that, as the name implies, the coarse image analysis is only a rough estimate as to tissue health for the purpose of selecting target points, and the tissue assessment using the second imaging modality is used to precisely finely identify the portions of the tissue region associated with healthy and unhealthy tissue.

[0074] FIGs. 4A and 4B depict user interfaces 400a, 400b associated with presenting an output of a tissue assessment. The user interfaces 400 may be presented by a workstation (such as the workstations 102 and 207).

[0075] Starting with FIG. 4A, the user interface 400a depicts image data 402a captured via a fluorescent imaging modality during a tissue assessment period. As illustrated, the point tracking model (such as the point tracking model 220) is tracking five targe points during the tissue assessment. In the illustrated scenario, the imaging system is operating in a fluorescent imaging modality. In the fluorescent imaging modality, a probe solution including fluorescently-tagged molecules is injected into the blood vessels in the tissue region. The fluorescently-tagged molecules are earned through the blood stream to the portions of tissue. Thus, the portions of the tissue associated with strong blood flow (and thus are well oxygenated) will exhibit a higher density of fluorescing molecules.

[0076] Accordingly, the point tracking model assess the tracked points by determining an intensity of the fluorescent illumination at the tracked point. The user interface 400a includes a chart 404a depicting the fluorescent intensities for the tracked points. It should be appreciated that because the fluorescence fades over time as the probe molecules traverse the blood stream, the chart 404a depicts the fluorescent intensity over time for each target point to provide additional information about the blood flow through the target points.Intuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC

[0077] Turning to FIG. 4B, the user interface 400b depicts guidance for treating the tissue region in view of the tissue assessment. More particularly, the user interface depicts a surgical plane 406b overlaid on the image data 402b. As described herein, the surgical plane 406b may be generated by a plane solver model (such as the plane solver 230) based on an analysis of the tissue assessment of the tracked points during the assessment region. As illustrated, the surgical plane 406b has a generally-elliptical shape.

[0078] FIG. 5 is a flow diagram of a method 500 for assessing tissue properties for clinical procedures. The method may be performed by a processor system or a control system (such as the processor system 150 and control system 140 of FIG. 1). In some embodiments, the control system may implement a machine learning module (such as the machine learning module 180) to perform the tissue analysis functionality described herein. In implementations, the control system is coupled to one or more repositionable structures (such as the repositionable structures 120) that may be operably coupled to one or more instruments (such as the instruments 122).

[0079] The method 500 may begin at block 502 when the control system obtains image data from an imaging system. The imaging system may capture image data in a plurality of imaging modalities such as in a white light modality (e.g., via a visible camera and white light illumination), and infrared imaging modality, a fluorescent imaging modality, etc. The imaging system may include one or more cameras to capture the images in various imaging modalities. As described, the imaging system may include multiple imaging sensors configured to capture images at respective different imaging modalities, or the imaging system may include one or more multimodal imaging cameras or sensors that each can be configured to capture images at different imaging modalities. The obtained image data includes one or more images of a tissue region for performing a clinical operation.

[0080] At block 504, the control system further obtains input data that is indicative of a desired tissue assessment to perform. For example, the input data may be indicative of a request to perform an assessment of tissue oxygenation for various tissues in the tissue region, an assessment of tissue health, of tissue viability, etc. Additionally, the input data may be indicative of requested guidance to be determined and provided by the control system. The guidance may be determined from tissue assessments and tissue parameter values determined from the image data. For example, theIntuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC requested guidance may include guidance for performing an incision, a suture, an injection, an extraction, a transection, a biopsy, an ablation, an injection or other treatment, etc.

[0081] At block 506, the control system may execute a machine learning model, such as by the point selection module 210 of FIG. 2, to analyze the input data and select the one or more target points in the images. The point selection module may implement a VLM machine learning model to identify points in the images of the tissue region based on the input data (e.g., a determined tissue assessment, requested guidance, etc.). Additionally, the point selection module may determine a number of points for performing the tissue assessment, and may further identify one or more control points for performing tracking and further for performing relative measurements of tissue property values. The control system may further identify one or more sub-regions of the tissue region and may determine various target points based on the different sub-regions, and may further determine a density of target points based on the sub-regions, or based on the relative distance to boundaries between sub-regions.

[0082] At block 508, the control system may track, via a point tracking machine learning model (such as the point tracking model 220), the target points across multiple images of the image data. The point tracking machine learning model may track the points across images obtained by the imaging system over an assessment period of time. The assessment period of time may depend on an operation, procedure, or task being performed, or may be based on the determined tissue assessment to be performed. The point tracking machine learning model further determines and tracks one or more tissue parameter values for tissues at each of the target points across the assessment period of time. In examples, the point tracking model may track the points across images obtained at a first imaging modality (e.g., white light imaging modality), and may determine and track the tissue parameter values using images obtained at a second imaging modality (e.g., fluorescence imaging modality).

[0083] At block 510, the control system may provide an output to a user with the output based on the input data and the one or more tissue parameter values. In examples, the output may include a readout or reporting of a tissue parameter value such as tissue oxygenation level, tissue health, tissue viability, etc., or the output may include guidance such as indications of regions of healthy tissue, points and paths for performing incisions or sutures, injection sites, etc. Further, the output may be determined by a preferred type of output as indicated by the input data, such as a visualIntuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC output on a display, an audio output provided as a readout of a tissue parameter value or instructional guidance, or another type of output as may be provided to a clinician in an operating room or provided to a network or memory for providing to other systems. The control system determines the type of information included in the output, and the means or medium of providing the output from the input data, and / or the one or more tissue parameter values.

[0084] One or more components of the examples discussed in this disclosure, such as control system 140, may be implemented in software for execution on one or more processors of a computer system. The software may include code that when executed by the one or more processors, configures the one or more processors to perform various functionalities as discussed herein. The code may be stored in a non-transitory computer readable storage medium (e.g., a memory, magnetic storage, optical storage, solid-state storage, etc.). The computer readable storage medium may be part of a computer readable storage device, such as an electronic circuit, a semiconductor device, a semiconductor memory device, a read only memory (ROM), a flash memory, an erasable programmable read only memory (EPROM); a floppy diskette, a CD-ROM, an optical disk, a hard disk, or other storage device. The code may be downloaded via computer networks such as the Internet, Intranet, etc. for storage on the computer readable storage medium. The code may be executed by any of a wide variety of centralized or distributed data processing architectures. The programmed instructions of the code may be implemented as a number of separate programs or subroutines, or they may be integrated into a number of other aspects of the systems described herein. The components of the computing systems discussed herein may be connected using wired and / or wireless connections. In some examples, the wireless connections may use wireless communication protocols such as Bluetooth, near- field communication (NFC), Infrared Data Association (IrDA), home radio frequency (HomeRF), IEEE 502.11, Digital Enhanced Cordless Telecommunications (DECT), and wireless medical telemetry service (WMTS).

[0085] Various general-purpose computer systems may be used to perform one or more processes, methods, or functionalities described herein. Additionally or alternatively, various specialized computer systems may be used to perform one or more processes, methods, or functionalities described herein. In addition, a variety of programming languages may be used to implement one or more of the processes, methods, or functionalities described herein.Intuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC

[0086] While certain examples and examples have been described above and shown in the accompanying drawings, it is to be understood that such examples and examples are merely illustrative and are not limited to the specific constructions and arrangements shown and described, since various other alternatives, modifications, and equivalents will be appreciated by those with ordinary skill in the art.

Claims

Intuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PCWHAT IS CLAIMED IS:

1. A computer-assisted system for assessing tissue properties in a clinical procedure, the system comprising: an imaging system including one or more image sensors, the imaging system configured to operate in two or more imaging modalities to generate image data, wherein a first imaging modality of the one or more imaging modalities is associated with white illumination light; and a control system operably coupled to the imaging system, wherein the control system is configured to: obtain the image data generated by the imaging system; obtain input data indicative of an assessment to perform on a tissue region of a patient; analyze, using a point selection machine learning model, the input data and the obtained image data to select one or more target points in the obtained image data to track for performing the indicated assessment: track, via a point tracking model and across image data obtained via a second imaging modality of the imaging system, the one or more target points for an assessment period to determine one or more tissue parameter values for each tracked target point; and provide an output to a user based on (i) the input data, and (ii) the one or more tissue parameter values.

2. The computer-assisted system of claim 1, wherein to obtain the input data, the control system is further configured to: obtain audio data from an operator of the computer-assisted system; and process the audio data using an automatic speech recognition model to generate user input data.Intuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC3. The computer-assisted system of claim 2, wherein the user input data is further indicative of a desired output of the assessment.

4. The computer-assisted system of claim 1, wherein to obtain the input data, the control system is further configured to: receive the input data from one or more of a hospital management system, one or more sensors, a robotic system, or a memory.

5. The computer-assisted system of claim 4, wherein the received input data indicates preference data associated with how to assess the image data obtained via the first imaging modality to identify regions associated with tissue parameter values.

6. The computer-assisted system of claim 1, wherein the point selection machine learning model comprises a vision-language model.

7. The computer-assisted system of claim 6, wherein the point selection machine learning model is configured to: determine (i) a first region of the image data associated with coarse tissue parameter values exceeding a threshold value and (ii) a second region of the image data associated with the coarse tissue parameter values not exceeding the threshold value; and select target points within both the first region and the second region.

8. The computer-assisted system of claim 7 wherein the coarse tissue parameter values include at least one of a color, a histogram, an intensity, a reflectivity, and a presence of a feature or pattern in the image data.Intuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC9. The computer-assisted system of claim 8, wherein the coarse tissue parameters are indicative of tissue health and coarse tissue parameter values exceeding the threshold value indicate that a corresponding tissue is healthy.

10. The computer-assisted system of claim 8, wherein the point selection machine learning model is a density of target points proportional to a proximity of a border between the first region and the second region.

11. The computer-assisted system of claim 7, wherein to determine the first region and second region, the control system is configured to: perform a relative measurement of coarse tissue parameters values relative to a control point; and select a control point as a reference for performing the relative measurement.

12. The computer-assisted system of claim 11, wherein the control point is located within the first region of the image data.

13. The computer-assisted system of claim 1, wherein the control system is further configured to: register the selected one or more target points with a model of the tissue region of the patient.

14. The computer-assisted system of claim 1 , wherein the imaging system comprises: a first image sensor coupled to a first repositionable structure configured to capture the image data, and a second image sensor coupled to a second repositionable structure configured to capture second image data.Intuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC15. The computer-assisted system of claim 14, wherein the first image sensor is configured to operate in the first imaging modality and the second image sensor is configured to operate in a second imaging modality, and the imaging system is configured to: concurrently capture the image data from the first image sensor and the second image data from the second image sensor.

16. The computer-assisted system of claim 14, wherein the control system is further configured to: register a position of the first image sensor and the second image sensor based on kinematic data obtained from the first repositionable structure and the second repositionable structure.

17. The computer-assisted system of claim 16, wherein the control system is further configured to: correlate the selected target point in the image data from the first image sensor and the second image data from the second image sensor based on the registered position of the target point, the first image sensor, and the second image sensor.

18. The computer-assisted system of claim 14, wherein the control system is further configured to: present the image data obtained from the first image sensor on a display while the second image sensor obtains the second image data; and determine the one or more tissue parameter values based on the second image data.Intuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC19. The computer-assisted system of claim 14, wherein to provide the output, the control system is configured to: overlay the output on the image data obtained from the first image sensor.

20. The computer-assisted system of claim 14, wherein: the first image sensor is configured to operate in a third imaging modality, and to track the one or more target points for the assessment period, the control system is further configured to: track the one or more target points via (i) image data obtained by the first image sensor operating in the third imaging modality and (ii) image data obtained by the second image sensor operating in the second imaging modality.

21. The computer-assisted system of claim 1, wherein: the imaging systems comprises a multi-modality endoscope; and to track the target points, the control system is configured to: change an imaging modality of the endoscope from the first imaging modality to a second imaging modality.

22. The computer-assisted system of claim 21, wherein: the imaging system is configured to operate in a third imaging modality, and to track the one or more target points for the assessment period, the control system is further configured to: track the one or more target points for a first portion of the assessment period via the second imaging modality, and track the one or more target point for a second portion of the assessment period via the third imaging modality.Intuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC23. The computer-assisted system of claim 1, wherein: the imaging system is configured to operate in a second imaging modality, and the second imaging modality is a fluorescent imaging modality, a multispectral or hyperspectral imaging modality, or an oxygenation detection imaging modality, an ultrasound imaging modality, a speckle imaging modality, a probe-based confocal laser endomicroscopy (pCLE) imaging modality, or a Raman spectroscopy imaging modality.

24. The computer-assisted system of claim 1, wherein the one or more tissue parameter values correspond to at least one of a measurement of tissue viability, tissue health, tissue strain, tissue perfusion, tissue oxygenation, tissue elasticity, tissue impedance, and a fluorescence measurement.

25. The computer-assisted system of claim 1, wherein the output comprises a visual output provided to a user via display device.

26. The computer-assisted system of claim 25, wherein the display device comprises at least one of a monitor, laptop, medical display device, wearable device, tablet.

27. The computer-assisted system of claim 1, wherein the output comprises a visual indication of the one or more tracked target points.

28. The computer-assisted system of claim 1 , wherein the output comprises the one or more tissue parameter values.

29. The computer-assisted system of claim 1, wherein the output comprises an indication of an action to be performed by a user.Intuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC30. The computer-assisted system of claim 29, wherein the indication of the action to be performed comprises a visual indication of a path to incise, a a path to suture, a transection path for removal of an anatomical structure, a location to perform a biopsy, a location to perform an ablation, a location to apply electroporation, a location to apply a therapeutic to a region, and a location to apply high-intensity -focused ultrasound (HiFU).

31. The computer-assisted system of claim 29, further comprising inputting the tissue parameter values and the tracked target points into a plane solver to determine the action to be performed by the user.

32. The computer-assisted system of claim 31, wherein a plane output by the plane solver used to define a path includes a non-linear geometry.

33. The computer-assisted system of claim 32, wherein a resolution of the non-linear geometry is proportional to a number of tracked points.

34. The computer-assisted system of claim 29, wherein the control system is further configured to: detect additional user input data to provide a rationale for the action to be performed; and analyze, using the point selection machine learning model, the additional user input data to provide a natural language description for the rationale.

35. The computer-assisted system of claim 1, wherein the control system is further configured to: output a visual indication of the tracked target points and associated tissue parameter values.Intuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC36. A method for assessing tissue properties in clinical procedures via a computer- assisted system comprising (i) an imaging system including one or more image sensors, wherein the imaging system is configured to operate in two or more imaging modalities to generate image data, wherein a first imaging modality of the one or more imaging modalities is associated with white illumination light, and (ii) a control system operatively coupled to the imaging system, the method comprising: obtaining, via the control system, the image data generated by the imaging system; obtaining, via the control system, input data indicative of an assessment to perform on a tissue region of a patient; analyzing, using a point selection machine learning model executed by the control system, the input data and the obtained image data to select one or more target points in the obtained image data to track for performing the indicated assessment; tracking, via a point tracking model executed by the control system and across image data obtained via a second imaging modality of the imaging system, the one or more target points for an assessment period to determine one or more tissue parameter values for each tracked target point; and providing, via the control system, an output to a user based on (i) the input data, and (ii) the one or more tissue parameter values.

37. The method of claim 36, wherein obtaining the input data comprises: obtaining, via the control system, audio data from an operator of the computer-assisted system; and processing, via the control system, the audio data using an automatic speech recognition model to generate user input data.

38. The method of claim 37, wherein the user input data is further indicative of a desired output of the assessment.Intuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC39. The method of claim 37, wherein obtaining the input data comprises: receiving, via the control system, the input data from one or more of a hospital management system, one or more sensors, a robotic system, or a memory.

40. The method of claim 39, wherein the received input data indicates preference data associated with how to assess the image data obtained via the first imaging modality to identify regions associated with tissue parameter values.

41. The method of claim 36, wherein the point selection machine learning model comprises a vision-language model.

42. The method of claim 41, wherein the method further comprises: determining, via the point selection machine learning model executed by the control system, (i) a first region of the image data associated with coarse tissue parameter values exceeding a threshold value and (ii) a second region of the image data associated with the coarse tissue parameter values not exceeding the threshold value; and selecting, via the control system, target points within both the first region and the second region.

43. The method of claim 42, wherein the coarse tissue parameter values include at least one of a color, a histogram, an intensity, a reflectivity, and a presence of a feature or pattern in the image data.

44. The method of claim 43, wherein the coarse tissue parameter values are indicative of tissue health and coarse tissue parameter values exceeding the threshold value indicate that a corresponding tissue is healthy.Intuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC45. The method of claim 42, wherein the point selection machine learning model is a density of target points proportional to a proximity of a border between the first region and the second region.

46. The method of claim 42, wherein determining the first region and second region comprises: selecting, via the control system, a control point as a reference for performing a relative measurement; and performing, via the control system, a relative measurement of coarse tissue parameter values compared to the control point.

47. The method of claim 46, wherein the control point is located within the first region of the image data.

48. The method of claim 36, further comprising; registering, via the control system, the selected one or more target points with a model of the tissue region of the patient.

49. The method of claim 36, wherein obtaining the image data comprises: obtaining, from first image sensor coupled to a first repositionable structure, the image data; and obtaining, from a second image sensor coupled to a second repositionable structure, second image data.Intuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC50. The method of claim 49, wherein the first image sensor is configured to operate in the first imaging modality and the second image sensor is configured to operate in a second imaging modality, the method further comprises: concurrently capturing the image data with the first image sensor and the second image data with the second image sensor.

51. The method of claim 49, further comprising: obtaining, from the first repositionable structure and the second repositionable structure, kinematic data associated with positions or movements of the first repositionable structure and second repositionable structure; and registering, via the control system, a position of the first image sensor and the second image sensor based on the obtained kinematic data.

52. The method of claim 51, further comprising: correlating, via the control system, the selected target point in the image data from the first image sensor and the second image data from the second image sensor based on the registered position of the target point, the first image sensor, and the second image sensor.

53. The method of claim 49, further comprising: presenting, via the control system, the image data obtained from the first image sensor on a display while the second image sensor obtains the second image data, and determining, via the control system, the one or more tissue parameter values based on the second image data.

54. The method of claim 49, wherein to providing the output comprises: overlaying the output on the image data obtained from the first image sensor.Intuitive Docket No.: P06976-WO Attorney Docket No.: 33685 / 70522 / PC55. The method of claim 49, wherein: the first image sensor is configured to operate in a third imaging modality, and tracking the one or more target points for the assessment period comprises: tracking, via the control system, the one or more target points via (i) image data obtained by the first image sensor operating in the third imaging modality and (ii) image data obtained by the second image sensor operating in the second imaging modality.

56. The method of claim 36, wherein: the imaging systems comprises a multi-modality endoscope; and tracking the target points comprises: changing, via the control system, an imaging modality of the endoscope from the first imaging modality to a second imaging modality.

57. The method of claim 56, wherein: the imaging system is configured to operate in a third imaging modality, and tracking the one or more target points for the assessment period comprises: tracking, via the control system, the one or more target points for a first portion of the assessment period via the second imaging modality, and tracking, via the control system, the one or more target points for a second portion of the assessment period via the third imaging modality.

58. The method of claim 36, wherein: the imaging system is configured to operate in a second imaging modality, and the second imaging modality is a fluorescent imaging modality, a multispectral or hyperspectral imaging modality, or an oxygenation detection imaging modality, an ultrasoundIntuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC imaging modality, a speckle imaging modality, a probe-based confocal laser endomicroscopy (pCLE) imaging modality, or a Raman spectroscopy imaging modality.

59. The method of claim 36, wherein the one or more tissue parameter values correspond to at least one of a measurement of tissue viability, tissue health, tissue strain, tissue perfusion, tissue oxygenation, tissue elasticity, tissue impedance, and a fluorescence measurement.

60. The method of claim 36, wherein the output comprises a visual output provided to a user via a display device.

61. The method of claim 60, wherein providing the output comprises at least one of a monitor, laptop, medical display device, wearable device, tablet.

62. The method of claim 36, wherein the output comprises a visual indication of the one or more tracked target points.

63. The method of claim 36, wherein the output comprises the one or more tissue parameter values.

64. The method of claim 36, wherein the output comprises an indication of an action to be performed by a user.

65. The method of claim 64, wherein the indication of the action to be performed comprises a visual indication of a path to incise, a path to suture, a transection path for removal of an anatomical structure, a location to perform a biopsy, a location to perform an ablation, aIntuitive Docket No.: P06976-WOAttorney Docket No.: 33685 / 70522 / PC location to apply electroporation, a location to apply a therapeutic to a region, and a location to apply high-intensity -focused ultrasound (HiFU).

66. The method of claim 64, further comprising: inputting the tissue parameter values and the tracked target points into a plane solver to determine the action to be performed by the user.

67. The method of claim 66, wherein a plane output by the plane solver used to define a path includes a non-linear geometry.

68. The method of claim 67, wherein a resolution of the non-linear geometry is proportional to a number of tracked points.

69. The method of claim 64, further comprising: detecting, via the control system, additional user input data to provide a rationale for the action to be performed; and analyzing, using the point selection machine learning model, the additional user input data to provide a natural language description for the rationale.

70. The method of claim 36, further comprising: outputting, via the control system, a visual indication of the tracked target points and associated tissue parameter values.

71. One or more non-transitory, computer-readable media storing instructions that, when executed by a control system of a computer-assisted system, causes the control system to perform the method of any one of claims 36-70.

Images