Multi-task ai system for dynamic and intelligent robotic task planning based on a user specified persona

Abstract

A system may detect a user input indicative of characteristics of historical operation for one or more historical repositionable structures operated by one or more historical users, query historical operational data based on the characteristics, and configure or modify a task generation machine learning model, a task generation constitution, a task selection machine learning model, a task selection constitution, a robotic transformer model, or a robotic action constitution based on the historical operational data. The system may, analyze, via the task generation machine learning model, data streams to identify a plurality of tasks to be performed; select, via the task selection machine learning model, a task to be performed from the identified tasks; generate action tokens for the selected task via the robotic transformer model; and control, based on the action tokens, the one or more repositionable structures to perform the selected task.

Description

Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PCMULTI-TASK Al SYSTEM FOR DYNAMIC AND INTELLIGENT ROBOTIC TASK PLANNING BASED ON A USER SPECIFIED PERSONACROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of the filing date of provisional U.S. Patent Application No. 63 / 717,458 entitled “MULTI-TASK Al SYSTEM FOR DYNAMIC AND INTELLIGENT ROBOTIC TASK PLANNING BASED ON A USER SPECIFIED PERSONA,” filed on November 7, 2024. The entire contents of the provisional application are hereby expressly incorporated herein by reference.TECHNICAL FIELD

[0002] The present disclosure relates generally to computer-assisted systems and more particularly to adapting performance of medical procedures performed by computer-assisted systems based on a user specified persona.BACKGROUND

[0003] Medical procedures often require various tasks to be performed by individuals or devices. Computer-assisted manipulator systems (“manipulator systems”), sometimes referred to as robotically assisted systems or robotic systems, may include one or more medical devices, equipment, and sensors that can be operated with the assistance of an electronic controller (e.g., computer or control system) to move and control functions of one or more instruments or medical devices. A computer-assisted medical system generally includes a robotic system with mechanical links connected by joints. An instrument is removably (or permanently) coupled to one of the links, typically a distal link of the plural links. In some embodiments, the computer- assisted medical systems are used in conjunction with one or more auxiliary devices (e.g., a surgical bed, an insufflator, etc.).

[0004] Conventional systems for monitoring medical environments do not provide any functionality to adapt system operations to perform actions based on historical operational data associated with a desired persona (e.g., a particular surgeon, surgery team, etc.). To this end, in some conventional systems, a frozen pre-trained model is utilized in conjunction with preengineered prompt to control system operations. However, these pre-engineered prompts require significant manual effort to ensure sufficient model performance. Accordingly, this approach has inferior execution of operational tasks and lacks user customization related to different user personas.Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC

[0005] Accordingly, there is a need for improved techniques that enable dynamic adaptation of models and systems to perform medical procedure operations according to a user specified persona. Such techniques may allow improved surgical outcomes, improved task workflows, reduced system setup and calibration times, and reduced system training and setup times. These improvements further allow for wider access to surgical treatment and diagnosis across a broad range of medical and clinical domains.SUMMARY

[0006] The following presents a simplified summary of various examples described herein and is not intended to identify key or critical elements or to delineate the scope of the claims.

[0007] In some aspects, the techniques described herein relate to a computer-assisted system for performing automated tasks, the system including: one or more repositionable structures configured to support respective instruments; and a control system operably coupled to the repositionable structure, wherein the control system is configured to: detect a user input indicative of characteristics of historical operation for one or more historical repositionable structures operated by one or more historical users; query historical operational data based on the characteristics of historical operation; configure or modify at least one of a task generation machine learning model, a task generation constitution, a task selection machine learning model, a task selection constitution, a robotic transformer model, or a robotic action constitution based on the historical operational data; receive a plurality of data streams from one or more data sources; analyze, via the task generation machine learning model, the data streams to identify a plurality of tasks to be performed by the one or more repositionable structures, wherein the task generation constitution is input into the task generation machine learning model to control how the task generation machine learning model analyzes the data streams; select, via the task selection machine learning model, a task to be performed from the identified tasks, wherein the task selection constitution is input into the task selection machine learning model to control how the task selection machine learning model selects the task to be performed; generate action tokens for the selected task via the robotic transformer model, wherein: the robotic action constitution, one or more of the plurality of data streams, and an indication of the selected task are input into the robotic transformer model, and the robotic action constitution controls how the robotic transformer model generates the action tokens for the selected task based on the one or more of the plurality of data streams; and control, based on the action tokens, the one or more repositionable structures to perform the selected task.Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC

[0008] In some aspects, the techniques described herein relate to a computer-implemented method for performing automated tasks, the method including: detecting a user input indicative of characteristics of historical operation for one or more historical repositionable structures operated by one or more historical users; querying historical operational data based on the characteristics of historical operation; configuring or modifying at least one of a task generation machine learning model, a task generation constitution, a task selection machine learning model, a task selection constitution, a robotic transformer model, or a robotic action constitution based on the historical operational data; receiving a plurality of data streams from one or more data sources; analyzing, via the task generation machine learning model, the data streams to identify a plurality of tasks to be performed by one or more repositionable structures, wherein: the one or more repositionable structures are configured to support respective instruments and are operably coupled to a control system, and the task generation constitution is input into the task generation machine learning model to control how the task generation machine learning model analyzes the data streams; selecting, via the task selection machine learning model, a task to be performed from the identified tasks, wherein the task selection constitution is input into the task selection machine learning model to control how the task selection machine learning model selects the task to be performed; generating action tokens for the selected task via the robotic transformer model, wherein: the robotic action constitution, one or more of the plurality of data streams, and an indication of the selected task are input into the robotic transformer model, and the robotic action constitution controls how the robotic transformer model generates the action tokens for the selected task based on the one or more of the plurality of data streams; and controlling, based on the action tokens, the one or more repositionable structures to perform the selected task.

[0009] In some aspects, the techniques described herein relate to a computer-readable media storing instructions that, when executed by a control system of a computer-assisted system, causes the control system to perform any of the methods described herein.

[0010] It is to be understood that both the foregoing general description and the following detailed description are illustrative and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In that regard, additional aspects, features, and advantages of the present disclosure will be apparent to one skilled in the art from the following detailed description.BRIEF DESCRIPTION OF THE DRAWINGS

[0011] FIG. 1 is a diagram of a computer-assisted system in accordance with one or more embodiments.Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC

[0012] FIG. 2 is a schematic diagram of a system for performing tasks in furtherance of a medical procedure using adaptive artificial intelligence based on user input.

[0013] FIG. 3 is a schematic diagram of a robotic action module for generating actions tokens to control a repositionable structure.

[0014] FIG. 4 is a schematic diagram of an example task generation module for determining task specific data streams for controlling a repositionable structure.

[0015] FIG. 5 is a schematic diagram of an example task selection module for determining task specific data streams for controlling a repositionable structure.

[0016] FIG. 6 is a schematic diagram of an example data modality selection module for determining task specific data streams for controlling a repositionable structure.

[0017] FIG. 7 is a schematic diagram of an example task assignment module for assigning tasks to perform a procedure.

[0018] FIG. 8 is a schematic diagram of an example user input module for detecting user input, querying a data store, and modifying one or more data streams, modules, and / or constitutions according to the user input.

[0019] FIG. 9A is a schematic diagram illustrating a process for detokenizing action tokens.

[0020] FIGs. 9B and 9C depict de-tokenized commands for controlling a repositionable structure.

[0021] FIG. 10 is a flow diagram of a method for determining and performing tasks for a medical procedure utilizing adaptive artificial intelligence based on user input.

[0022] Examples of the present disclosure and their advantages are best understood by referring to the detailed description that follows. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures, wherein showings therein are for purposes of illustrating examples of the present disclosure and not for purposes of limiting the same.DETAILED DESCRIPTION

[0023] In this description, specific details are set forth describing some embodiments consistent with the present disclosure. Numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one skilled in the art that some embodiments may be practiced without some or all of these specific details. TheIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC specific embodiments disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless specifically described otherwise or if the one or more features would make an embodiment non-functional.

[0024] Further, the terminology in this description is not intended to limit the invention. For example, spatially relative terms-such as “beneath”, “below”, “lower”, “above”, “upper”, “proximal”, “distal”, and the like-may be used to describe the relation of one element or feature to another element or feature as illustrated in the figures. These spatially relative terms are intended to encompass different positions ( / .e., locations) and orientations ( / .e., rotational placements) of the elements or their operation in addition to the position and orientation shown in the figures. For example, if the content of one of the figures is turned over, elements described as “below” or “beneath” other elements or features would then be “above” or “over” the other elements or features. A device may be otherwise oriented and the spatially relative descriptors used herein interpreted accordingly. Likewise, descriptions of movement along and around various axes include various special element positions and orientations. In addition, the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context indicates otherwise. Additionally, the terms “comprises”, “comprising”, “includes”, and the like specify the presence of stated features, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups. Components described as coupled may be electrically or mechanically directly coupled, or they may be indirectly coupled via one or more intermediate components.

[0025] Elements described in detail with reference to one embodiment, implementation, system, or module may, whenever practical, be included in other embodiments, implementations, systems, or modules in which they are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment and is not described with reference to a second embodiment, the element may nevertheless be claimed as included in the second embodiment. Thus, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, implementation, or application may be incorporated into other embodiments, implementations, or aspects unless specifically described otherwise, unless the one or more elements would make an embodiment or implementation non-functional, or unless two or more of the elements provide conflicting functions.Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC

[0026] In some instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0027] This disclosure describes various devices, elements, and portions of computer-assisted systems and elements in terms of their state in three-dimensional space. As used herein, the term “position” refers to the location of an element or a portion of an element (e.g., three degrees of translational freedom in a three-dimensional space, such as along Cartesian x-, y-, and z- coordinates). As used herein, the term “orientation” refers to the rotational placement of an element or a portion of an element (e.g., three degrees of rotational freedom in three-dimensional space, such as about roll, pitch, and yaw axes, represented in angle-axis, rotation matrix, quaternion representation, and / or the like). As used herein, and for a device with a kinematic series, such as with a repositionable structure with a plurality of links coupled by one or more joints, the term “proximal” refers to a direction toward a base of the kinematic series, and “distal” refers to a direction away from the base along the kinematic series.

[0028] As used herein, the term “pose” refers to the multi-degree of freedom (DOF) spatial position and orientation of a coordinate system of interest attached to a rigid body. In general, a pose includes a pose variable for each of the DOFs in the pose. For example, a full 6-DOF pose for a rigid body in three-dimensional space would include 6 pose variables corresponding to the 3 positional DOFs (e.g., x, y, and z) and the 3 orientational DOFs (e.g., roll, pitch, and yaw). A 3- DOF position only pose would include only pose variables for the 3 positional DOFs. Similarly, a 3-DOF orientation only pose would include only pose variables for the 3 rotational DOFs. Further, a velocity of the pose captures the change in pose over time (e.g., a first derivative of the pose). For a full 6-DOF pose of a rigid body in three-dimensional space, the velocity would include 3 translational velocities and 3 rotational velocities. Poses with other numbers of DOFs would have a corresponding number of velocities translational and / or rotational velocities.

[0029] This disclosure occasionally refers to the disclosed techniques being applied to “patients” undergoing a “medical procedure.” It should be appreciated that these references are not intended to limit the application of the disclosed techniques to applied medicine contexts. For example, the described techniques can be applied to facilitate physician training, equipment testing and / or calibration, and / or other contexts. Accordingly, any reference to the term “patient” is done for ease of explanation and also envisions the application of the described techniques to a generic “subject.”

[0030] The word “task” is used herein to refer to a discrete portion of procedure that may be autonomously, semi-autonomously, or manually implemented in furtherance of a procedure. ForIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC example, a task may be to move an endoscope to a particular portion, to advance an instrument to a particular depth, to replace an instrument coupled to a manipulator, and so on. In some embodiments, a task is associated with component tasks to accomplish an overall goal. For example, a task to analyze a worksite may include component tasks related to moving an endoscope to view the worksite, advancing an instrument to predetermined depth, and enabling a functionality supported by the instrument.

[0031] Aspects of this disclosure are described in reference to computer-assisted systems, which can include devices that are teleoperated, externally manipulated, autonomous, semiautonomous, and / or the like. Further, aspects of this disclosure are described in terms of an implementation using a teleoperated surgical system, such as the da Vinci® Surgical System commercialized by Intuitive Surgical, Inc. of Sunnyvale, California. Knowledgeable persons will understand, however, that inventive aspects disclosed herein may be embodied and implemented in various ways, including teleoperated and non-teleoperated, and medical and non-medical embodiments and implementations. Implementations on da Vinci® Surgical Systems are merely exemplary and are not to be considered as limiting the scope of the inventive aspects disclosed herein. For example, techniques described with reference to surgical instruments and surgical methods may be used in other contexts. Thus, the instruments, systems, and methods described herein may be used for humans, animals, portions of human or animal anatomy, industrial systems, general robotic, or teleoperated systems. As further examples, the instruments, systems, and methods described herein may be used for non-medical purposes including industrial uses, general robotic uses, sensing or manipulating non-tissue work pieces, cosmetic improvements, imaging of human or animal anatomy, gathering data from human or animal anatomy, setting up or taking down systems, training medical or non-medical personnel, and / or the like. Additional example applications include use for procedures on tissue removed from human or animal anatomies (with or without return to a human or animal anatomy) and for procedures on human or animal cadavers. Further, these techniques can also be used for medical treatment or diagnosis procedures that include, or do not include, surgical aspects.

[0032] The disclosure generally relates to systems and methods for training and intelligently providing adaptive artificial intelligence (Al) for performing and assisting in operating room tasks. In particular, the systems and methods described herein relate to configuring or modifying Al control processes to perform and assist in the operating room tasks according to different user personas. While the disclosure primarily focuses on adapting constitutions that control task generation, task evaluation, data modality selection, task assignment, and robotic transformer machine learning models and / or modifications to parameters of the models (e.g., via fine tuning),Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC the adaptions can be made to any set of inputs to or parameters of a machine learning model implemented in a control system. In some embodiments, the control system disclosed is configured to detect a user input indicative of characteristics of historical operation for one or more historical repositionable structures operated by one or more historical users (e.g., characteristics of a user persona). The control may include a user input processing module to generate model and / or constitution modifications and identify which machine learning models should be adapted to implement the indicated user persona.

[0033] To adapt a constitution, in some embodiments the systems may store sets of predefined rules that can be incorporated into a constitution. For example, the sets of predefined rules may relate to how to prioritize different types of tasks (e.g., should speed of procedure be prioritized, should manual guidance be prioritized, etc.). Accordingly, the systems may be configured to process user input to identify an appropriate set of rules and update a constitution in accordance therewith. In some embodiments, if the systems do not detect an appropriate set of rules, the systems may utilize a generative Al model to generate one or more rules intended to implement the user input.

[0034] To fine tune a machine learning model, in some embodiments the systems may implement self-supervised or similar process known in the art. In particular, the fine-tuning process may use historical operational data that is retrieved from a data store as the training data to refine or otherwise modify the originally trained model parameter values to generate outputs that more closely align with the historical operational data. Furthermore, this fine-tuning process may be applied to discrete layers or components of a model such as an embedding layer or the like.

[0035] The user input may be provided in natural language (e.g., via a chatbot-type interface), as audio data, as non-linguistic user interactions with a GUI (e.g., tapping or gesturing on a touch- sensitive display screen), or any other suitable method of determining user intended operation of a robotic system for performing medical operations. By modifying the models and / or the constitutions based on user inputs, the systems are able to dynamically optimize the outputs of a given model for a specific task at hand, environment, and / or conditions.

[0036] FIG. 1 is a simplified diagram of an example computer-assisted system 100, according to various embodiments. The computer-assisted system 100 may be a computer-assisted medical system for assisting with performing tasks for medical procedures. Further, the computer-assisted system 100 may utilize adaptable Al models for determining and performing medical tasks as described herein. The computer-assisted system 100 may further determineIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC and utilize task-specific data streams for performing one or more medical tasks, and assign tasks as described herein. In some examples, the computer-assisted system 100 is a teleoperated system. In medical examples, the computer-assisted system 100 can be a teleoperated medical system such as a surgical system. As shown, the computer-assisted system 100 includes a follower device 104 that can be teleoperated by being controlled by one or more leader devices (also called “leader input devices” when designed to accept external input), described in greater detail below. Systems that include a leader device and a follower device are referred to as leaderfollower systems, and also sometimes referred to as master-slave systems. Also shown in FIG. 1 is an input system that includes a workstation 102 (e.g., a console), and in various embodiments the input system can be in any appropriate form and may or may not include the workstation 102.

[0037] In the example of FIG. 1 , the workstation 102 includes one or more leader input devices 106 that are designed to be contacted and manipulated by an operator 108. For example, the workstation 102 may comprise one or more leader input devices 106 for use by the hands, the head, or some other body part(s) of operator 108. The leader input devices 106 in this example are supported by the workstation 102 and can be mechanically grounded. In some embodiments, an ergonomic support 110 (e.g., forearm rest) can be provided on which the operator 108 can rest his or her forearms. In some examples, the operator 108 can perform tasks at a worksite within a workspace near the follower device 104 during a procedure, by commanding the follower device 104 using the leader input devices 106. In a medical example, the worksite may be a surgical worksite associated with a patient.

[0038] A display device 1 12 is also included in the workstation 102. The display device 112 may be configured to display images for viewing by the operator 108. The display device 112 can be moved in various DOFs to accommodate the viewing position of the operator 108 and / or to provide control functions. In embodiments where the display device 112 provides control functions, the leader input devices 106 may include the display device 112. In the example of the computer-assisted system 100, displayed images may depict a worksite at which the operator 108 is performing various tasks by manipulating the leader input devices 106 and / or the display device 112. In some examples, images displayed by display device 112 may be received by the workstation 102 from one or more imaging devices arranged at a worksite. In other examples, the images displayed by the display device 112 may be generated by the display device 112 (or by a different connected device or system), such as for virtual representations of tools, the worksite, or for user interface components. As will be explained below, in some embodiments the display device 1 12 may display one or more tasks for the operator 108 to perform with respect to any component of the computer-assisted system 100.Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC

[0039] In examples, the display device 112 may be a touch-screen device and may receive user-input via the touch screen. The touch screen may provide a user interface that a user may select from various options to provide one or more user preferences for performing a procedure or medical task. Additionally, the display device 112 may display one or more images such as an intraoperative or pre-operative image of a patient, organ, surgical site etc., and the user may provide user input via the touch screen to indicate one or more regions or elements displayed in the intraoperative or pre-operative images. The display device 1 12 may provide one or more images or user interfaces and a user may interact with and provide user indications and input via another device such as a keyboard, mouse, audio device, etc. For example, a user may provide text input such as natural language text via a keyboard to provide a user input to the system. A user may use a mouse or another similar device to click on or indicate selection of an option or user feedback based on one or more images presented by the display device. The workstation 102 may further include a microphone that may capture and record audio of a user providing indications of user preferences and user inputs to the system. One or more processors, or controllers as discussed further herein, may then derive natural language text from the recorded audio data to derive a user input to the system 100. In examples, the user input may further be determined from the audio data as a user responding to one or more prompts to confirm, reject, or otherwise indicate a user preference or input. Additionally, user input may be provided via one or more sources such as video, haptic input, data banks, instruments (e.g., by a user changing a setting on an instrument or auxiliary device), sensors, etc. It should be appreciated that while the foregoing describes obtaining the user input from the display device 112 of the workstation 102, in other embodiments, the user input may be obtained by other display devices associated with the operating room that are interacted with by personnel other than the operator 108.

[0040] As illustrated, the computer-assisted system 100 also includes a follower device 104 that can be commanded by the workstation 102. In a medical example, the follower device 104 can be located near an operating table (e.g., a table, bed, or other support) on which a patient can be positioned. In some medical examples, the workspace is provided on an operating table, e.g., on or in a patient, simulated patient, or model, training dummy, etc. (not shown). As illustrated, the follower device 104 may include a plurality of repositionable structures 120 (sometimes referred to as “manipulator arms” in robotic embodiments). In some embodiments, the repositionable structures 120 may include a plurality of links that are rigid members and joints that can be individually actuated as part of a kinematic series. Additionally, each of the repositionable structures 120 is configured to be coupled to an instrument 122. While FIG. 1 illustrates a follower device 104 that has four repositionable structures 120a-120d, in otherIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC embodiments, the follower device 104 may include one, two, three, four, five, six, or additional or fewer repositionable structures 120a-120d.

[0041] The instrument 122 can include, for example, a working portion 126 and one or more structures for supporting and / or driving the working portion 126. Example working portions 126 include end effectors that physically contact or manipulate material, energy application elements that apply electrical, RF, ultrasonic, or other types of energy, sensors that detect characteristics of the workspace environment (such as temperature sensors, imaging devices, etc.), and the like. In various embodiments, examples of instruments 122 include, without limitation, a sealing instrument, a cutting instrument, a sealing-and-cutting instrument, an energy instrument for applying energy, a gripping instrument (e.g., clamps, jaws), a stapler, an imaging instrument such as one using optical, RF, or ultrasonic imaging modalities, a sensing instrument, an irrigation instrument, a suction instrument, and / or the like. In addition, the instrument 122 may include a transmission mechanism 128 that can be coupled to a drive assembly 130 of the respective repositionable structure 120a-120d. The drive assembly 130 may include a drive and / or other mechanisms controllable from workstation 102 that transmit forces to the transmission mechanism 128 to articular or otherwise actuate the instrument 122.

[0042] As illustrated, each instrument 122 may be mounted to a portion of a respective repositionable structure 120a-120d. In FIG. 1 , this is shown with the drive assembly 130 physically coupled to the transmission mechanism 128. The distal portion of each repositionable structure 120a-120d further includes a cannula mount 124 to which a cannula (not shown) is mounted. When a cannula is mounted to the cannula mount 124, a shaft of the instrument 122 passes through the cannula and into a workspace.

[0043] In various embodiments, one or more of the working portions 126 of the instruments 122 may include an imaging device for capturing images. The imaging device may include any sensing technology capable of acquiring an image. Example imaging instruments include an optical endoscope, a hyperspectral camera, an ultrasonic sensor, etc. Imaging instruments may comprise monoscopic imagers, stereoscopic imagers, and / or the like. Imaging devices based on radiofrequency domains may capture images in any frequency spectrum, including visible light, infrared light, ultraviolet light, and / or the like. The imaging device may include an illumination source to light the region being imaged. In embodiments where the working portions 126 of one or more of the instruments 122 include an imaging device, the instrument 122 may be configured to capture images of a portion of the workspace for display via the display device 112.Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC

[0044] In some embodiments, the repositionable structures 120a-120d and / or instruments 122 can be controlled to move the working portion 126 in response to manipulation of the leader input devices 106 by the operator 108. Accordingly, the repositionable structures 120a-120d and / or instruments 122 may be said to “follow” the leader input devices 106 through teleoperation. This enables the operator 108 to perform tasks at the worksite using the repositionable structures 120a-120d and / or instruments 122. For a surgical example, the operator 108 can direct the repositionable structures 120a-120d of the follower device 104 to move the working portions 126 as part of a surgical procedure performed at an internal surgical site that is entered via one or more minimally invasive apertures or natural orifices. It should be appreciated that, in some embodiments, the follower device 104 may include non-teleoperated components that the operator 108 or other medical professional must manually manipulate to a desired pose.

[0045] In some embodiments, a repositionable structure 120a of the computer-assisted system 100 may be configured to support a working portion 126a that includes an imaging device (also referred to herein as an “imaging device 126a”). For convenience, an instrument 122 that includes an imaging device is also referred to as an “imaging instrument” herein. The control system 140 may be configured to command the repositionable structure 120a and / or the imaging instrument 122 comprising the imaging device 126a to automatically position and / or orient (“pose”) the field of view (FOV) of the imaging device 126a to provide images of the workspace and / or other instruments 122.

[0046] In the illustrated embodiment, a control system 140 is communicatively coupled to the workstation 102. In other embodiments, the control system 140 may be provided as a component of the workstation 102 and / or the follower device 104. During teleoperation, as the operator 108 moves the leader input device(s) 106, one or more sensors configured to detect the leader input device(s) 106 generate spatial and / or orientation movement data that is provided to control system 140. The control system 140 may interpret the spatial and / or orientation information to determine and / or provide control signals to the follower device 104 to control the movement of repositionable structures 120a-120d, instruments 122, and / or working portions 126. In addition to the components of the follower device 104, in some embodiments, the control system 140 is configured to interpret inputs received from the workstation 102 to control operation of one or more auxiliary devices (not depicted) utilized in a procedure. For example, the workstation 102 may be used to control a pose of a surgical bed or operation of an insufflator.

[0047] In one embodiment, the control system 140 supports one or more wired communication protocols, (e.g., Ethernet, USB, and / or the like) and / or one or more wireless communicationIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC protocols (e.g., Bluetooth, IrDA, HomeRF, IEEE 1102.11 , DECT, Wireless Telemetry, and / or the like) for communications between the control system 140 and the workstation 102 and / or the follower device 104.

[0048] In some embodiments, the control system 140 may be implemented at one or more computing systems. For example, one or more computing systems may be used to control the follower device 104. As another example, one or more computing systems may be used to control components of the workstation 102, such as movement of a display device 112.

[0049] As illustrated, the control system 140 includes a processor system 150, a memory 160, and an artificial intelligent (Al) assist module 180. The memory 160 may store a control module 170. The processor system 150 may include one or more processors having different processing architectures for processing instructions. For example, the one or more processors may be one or more cores or micro-cores of a multi-core processor, a central processing unit (CPU), a microprocessor, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a graphics processing unit (GPU), a tensor processing unit (TPU), and / or the like.

[0050] In some embodiments, the processor system 150 includes circuity to support one or more communication interfaces (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.). Additionally, a communication interface of control system 140 may include an integrated circuit for connecting the control system 140 to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, mobile network, or any other type of network) and / or to another device, such as the workstation 102 and / or the follower device 104.

[0051] Additionally, the memory 160 may include non-persistent storage (e.g., volatile memory, such as random access memory (RAM), cache memory), persistent storage (e.g., a hard disk, an optical drive such as a compact disk (CD) drive or digital versatile disk (DVD) drive, a flash memory, a floppy disk, a flexible disk, a magnetic tape, any other magnetic medium, any other optical medium, programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a FLASH-EPROM, and / or any other memory chip or cartridge. The non- persistent storage and persistent storage are examples of non-transitory, tangible machine- readable media that can store executable code that, when run by one or more processors (e.g., processor system 150), can cause the one or more processors to perform one or more of the techniques and / or methods disclosed herein.Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC

[0052] The Al assist module 180 may implement one or more machine learning models and / or training protocols therefore. For example, the Al assist module 180 may implement one or more neural networks, deep learning models, decision trees, support vector machines, linear regression, generative Al models, reinforced learning models, random forests, Naive Bayes models, large language models (LLMs), generative adversarial networks, foundation models, image recognition models, linear discriminant analysis models, creative applications, autoregressive models, supervised or unsupervised learning models, multimodal models, vision language models (VLMs), vision foundation models (VFMs), large multi-modal models (LMMs), Transformer models (including Robotic Transformer models), or another machine learning or Al model for performing the methods described herein. The structure of the one or more machine learning is described in more detail with respect to FIGs. 2-9. The Al assist module 180 may include dedicated processors and memory for storing and performing Al processes, or the Al assist module 180 may utilize resources of the processor system 150 and the memory 160 to store and / or perform any processing or tasks required to perform the methods described herein.

[0053] Additionally, the control system 140 may also include one or more input devices (such as a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, or any other type of input device) and / or output devices (such as a display device, a speaker, external storage, a printer, or any other output device). In some embodiments, the control system 140 may be implemented on a particular node of a distributed computing system (e.g., a cloud computing system). As another example, different functionalities associated with the control system 140 may be implemented on different nodes of the distributed computing system. Further, one or more elements of the aforementioned control system 140 may be located at a remote location and connected to the other elements over a network.

[0054] In an endoscopic surgery example, the imaging instrument comprising the imaging device 126a may be inserted into the patient prior to the other instruments 122, including a second instrument 122b comprising a second working portion 126b. The second instrument 122b can include any appropriate working portion 126b, and can even include a second imaging device. Accordingly, the imaging device 126a may be maneuvered to be positioned to identify a target to which other instruments may interact with as part of another task. The control system 140 may, for example, automatically command the corresponding repositionable structures 120a and 120b to position respective instruments 122a and 122b to perform one or more tasks in tandem, or sequentially based on the specific task, instruments, and positions of the repositionable structures 120a and 120b. In examples, the control system 140 may perform Al processes and algorithms via the Al assist module 180 to identify tasks for performing a medical procedure, and to furtherIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC assign the tasks to respective personnel for performing the tasks. Additionally, the control system 140 may identify certain tasks to be performed automatically by one or more robotic systems and / or devices, semi-automatically with user assistance or input to the system 100, and tasks that are to be performed entirely manually by personnel. It should be appreciated that some manual tasks may be performed independent of the computer-assisted system 100. For example, some tasks generated by the control system 140 may include sterilizing instruments or a worksite, obtaining an instrument for use later in a procedure, adjusting a pose of the subject, and so on.

[0055] The disclosed techniques enable the control system 140 to perform real-time operations (e.g., within 5ms, within 10ms, within 20ms, etc.). Accordingly, the control system 140 is able to identify and control repositionable structures to perform tasks for a medical procedure in a variety of scenarios using adaptable Al. Such techniques improve the efficiency of operating the computer-assisted system or instrument, simplify user control of the computer-assisted system 100, improve the efficiency of the medical procedure by streamlining task workflow, and allow for computer-assisted systems to be used in a much broader range of environments, and under various specific conditions without the needed for additional ML training and / or tuning. Further, although a surgical example is shown, the disclosed techniques provide an improvement to the computer-assisted system 100 in the non-surgical aspects of the procedure, and can be used to improve computer-assisted systems applied in non-medical contexts.

[0056] FIG. 2 is a schematic diagram of a system 200 for using adaptive Al for controlling repositionable structures to perform tasks in furtherance of a medical procedure. The tasks may be performed manually, semi-automatically with user input or guidance, or entirely automatically by a robotic structure or system (such as the repositionable structure 120). The system 200 includes a plurality of modules that may be executed by the control system 140 (e.g., via the Al assist module 180 of FIG. 1 ) to determine and perform the one or more tasks associated with a medical procedure. The system 200 includes sources of multi-modal data that from one or more multi-modal data streams 202, a task generation module 210, a task selection module 220, a user input module 230, a data modality select module 260, a task assignment module 270, and a robotic action module 250. It should be appreciated that in other embodiments, additional or fewer modules may be implemented. Further, in other embodiments, one or more of the described modules may be combined into a single module.

[0057] The system 200 includes one or more sources of multi-modal data 202. For example, the multimodal data 202 may include a data stream 202a indicative of force exerted upon an instrument, a data stream 202b indicative of system events generated by the repositionableIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC structure and / or a control system thereof, a data stream 202c indicative of kinematic data associated with the repositionable structure and / or instruments or auxiliary devices associated therewith, an external video data stream 202d, a procedure video data stream 202e (such as image data generated by an endoscope), a data stream 202f indicative of personnel and data associated with personnel that may be available for performing the procedure, a data stream 202g indicative of user input data such as via a user interface, and / or other sources of data that indicate a state of a procedure facilitated by the repositionable structures. In some embodiments, the data stream 202g indicative of the user input may include data indicative of characteristics of historical operation for one or more historical repositionable structures operated by one or more historical users (e.g., a person performing a task, a phase of the procedure, a subject condition that was present during the procedure, a technique or approach utilized during a procedure, an instrument used in a procedure, etc.).

[0058] Additionally, the data streams 202 may include data retrieved from one or more databases such as preoperative data (e.g., preoperative images, video, patient information, etc.). In examples, the data streams 202a-202g may include system data (e.g., data associated with system events, system capabilities, etc.), and data associated with a medical environment (e.g., via video data, image data, available instruments, etc.). In examples, the multimodal data 202 may include one or more data streams that provide layouts of medical environments or rooms and / or information associated with one or more medical systems and instruments that may be available for a procedure.

[0059] The system 200 may be configured to route the multimodal data 202 to a task generation module 210 configured to output one or more tasks that may be performed in furtherance of the procedure based on a procedure state represented by the input multimodal data 202. The task generation module 210 may identify tasks that are to be perform in the near term (e.g., tasks that respond to conditions detected in an input set of multi-modal data) and / or tasks that are to be performed in the long term (e.g., tasks that will need to be performed later in the procedure after one or more near-term tasks are performed). The task generation module 210 may further generate tasks to be performed in the future for a medical procedure scheduled to be performed at a later time or date. By also generating long-term tasks, the system 200 is able to track preparedness for performing the long-term task and generate additional tasks to prepare the control system for performing the long-term task as the procedure advances closer to the appropriate time for performing the long-term tasks. In some embodiments, the task generation module 210 may include component models to facilitate the analysis. For example, the task generation module 210 may include a scene recognition model 212 to parse image data toIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC generate natural language descriptions of the scene represented by the image data (and the corresponding locations in the image data) and a task generation model 215 configured to actually generate the one or more tasks. As one example, the scene recognition model 212 may detect and identify objects in a medical environment such as an operating room (OR) to generate an inventory of objects in the OR.

[0060] The task generation module 210 may analyze the multimodal data 202 to identify the tasks to be performed according to a set of rules. For example, a task generation constitution 211 may be input into the task generation module 210 as part of a prompt that also includes at least a portion of the multimodal data streams 202.

[0061] In some embodiments, a data stream input into the task generation module 210 may include an embedding indicative of the medical procedure, or of one or more phases of the medical procedure. The system 200 may then add or edit rules of the task generation constitution 211 according to the specific medical procedure or phases of the medical procedure that is to be performed. One such technique for detecting the type of medical procedure and / or phase thereof is described in U.S. Provisional Application No. 63 / 663,996, the entire disclosure of which is hereby incorporated by reference. Accordingly, in these embodiments, the phase information may be input into the task generation model 215 to generate tasks based on the particular phase for the medical procedure and according to the rules in the task generation constitution 21 1 .

[0062] The system 200 may then provide the output tasks (e.g., natural language descriptions of an action to be performed) to a task selection module 220 to select which tasks should actually be implemented by the control system, semi-automatically via user control / feedback and the control system, and / or by an operator thereof. As illustrated, the task selection module may include a task filter model 222 configured to filter the tasks generated by the task generation model 215 and a task selection model 225 configured to select one or more valid tasks to be performed by the control system and / or an operator thereof.

[0063] More particularly, the task selection module 220 may input a task selection constitution 221 input into a task selection model 225 along with one or more input tasks to select which tasks are to be performed. The task selection constitution 221 may include one or more sets of rules that control how the task assignment module 220 filters and selects the identified tasks. In examples, the rules of the task selection constitution 221 may cause the task selection module 220 to filter the various tasks according to one or more available data streams, available personnel, user inputs and preferences, device models, available instruments etc. The task selection module 220 may then select the tasks according to the rules of the task selectionIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC constitution 221 , and provide the selected tasks to the data modality selection module 260, the task assignment module 270, and / or the robotic action module 250.

[0064] The data modality selection module 260 may be configured to analyze the one or more selected tasks to determine one or more task specific data streams needed to implement the selected tasks. To this end, each task of a procedure or phase of a procedure may require a particular type of data to safely and effectively execute the task. Accordingly, the data modality selection module 260 may input a modality selection constitution 261 into a task analysis model 262 along with the selected tasks to determine one or more task specific data streams. Thus, the modality selection constitution 261 may be configured to control how the data modality selection module 260 determines the set of task-specific data streams.

[0065] It should be appreciated that a required data modality may include a particular data stream of the multimodal data 202 (e.g., kinematic data or procedure video) or a particular state of the multimodal data (e.g., that the procedure video is required to capture image data of a particular object, such a target anatomy). If a required data stream is unavailable, the rules of the modality selection constitution 261 may instruct the data modality selection module 260 to interact with the task selection module 220 to select a different task where the required data streams are all available and / or the task generation module 210 to generate a task to make the required data stream available (e.g., by enabling a sensor or by commanding a camera to reposition a field of view). The data modality selection module 260 may then provide a selection of the available data streams to the robotic action module 250.

[0066] Because the system 200 may be configured to process multiple tasks synchronously, the data modality selection module 260 may be configured to associate the selected available data streams with each task prior to providing the selection to the robotic action module 250. As a result, the robotic action module 250 may be able switch data modalities based on the particular tasks being converted into robotic action.

[0067] The task assignment module 270 may be configured to analyze the one or more selected tasks to assign the tasks to respective instruments, end effectors, auxiliary devices, personnel etc., according to rules of a task assignment constitution 271 . Additionally, one or more data streams of the plurality of data streams 202 may be input into the task assignment module 270 to assist in the task assignment analysis. While the task assignment module 270 is generally described with respect to instrument assignment, any such description is done for brevity, and envisions similar techniques related to selection of any actor to perform a particular task. The task assignment module 270 includes a task analysis model 272 configured to analyze the inputIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC data streams and the selected task to identify which instruments (including combinations of instruments) are capable of implementing the selected tasks and a task assignment model 275 configured to assign the task to respective in. The task analysis model 272 may identify different requirements for each task (e.g., personnel skillsets and qualifications, instrument and system capabilities, etc.). In some embodiments, a particular task may require multiple instruments and / or personnel to perform different aspects of a given task. Accordingly, the task analysis model 272 may be configured to divide the task into subtasks that are assigned to different instruments and / or personnel. The task analysis model 272 may analyze and identify instruments and / or personnel, and identify the different requirements for the tasks according to, for example, one or more rule sets included in the task assignment constitution 271 .

[0068] The task assignment module 270 further includes a task assignment model 275 (such as an LLM) that analyzes the requirements for the input tasks (and / or subtasks thereof) and assigns the various tasks to appropriate instrument according to one or more rule sets of the task assignment constitution 271 . That is, the task assignment module 270 may be configured to input both the selected tasks (and / or subtasks thereof) and the task assignment constitution 271 to the task assignment model 275 to determine the appropriate task assignments. It should be appreciated that if required instruments are not present or available, the rules of the task assignment constitution 271 may instruct the task assignment module 270 to send an indication to the task generation module 210 and / or the task selection module 220 to generate and / or select alternative tasks that can be performed based on the current instrument availability.

[0069] To perform semi-autonomous and autonomous tasks via the assigned instruments, the system 200 includes the robotic action module 250. As illustrated, the robotic action module 250 may be configured to receive the selected tasks from the task selection module 220, task assignments from the task assignment module 270, task-specific data streams from the data modality module 260, and / or the multimodal data 202. The robotic action module 250 may then process the multimodal data 202 (e.g., by embedding the determined task-specific data streams via the embedding stage 310), and the selected tasks to generate action tokens to control the manipulator system to perform the selected tasks. The robotic action module 250 may input the embeddings of the various inputs, and a robotic action constitution 251 into a RT model 315 to generate one or more action tokens. The action tokens may be natural language descriptions of a robotic action to be performed by the manipulator system in furtherance of a selected task. The robotic action module 250 may then convert the action tokens to command one or more components of a computer-assisted robotic device (such as the follower device 104 of FIG. 1 ) to perform one or more of the selected tasks.Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC

[0070] It should be appreciated that in some embodiments, one or more of the modules 210, 220, 230, 260, and 270 may be combined into a single module, for example, by implementing chain of thought (CoT) or other recurrent prompting techniques such that a single prompt performs the described functionality corresponding to the individual modules. Additionally, in other embodiments, the system 200 may include additional modules that generate inputs to the robotic action module 250 to implement autonomous tasks.

[0071] Generally, the constitutions of the system 200 (e.g., the constitutions 21 1 , 221 , 231 , 261 , 271 ) may include various sets of rules according to different types or categories of rules. For example, the constitutions may include foundational rules that define allowable robotic actions (e.g., limitations on instrument operation for patient and user safety, such as Asimov’s three rules), safety rules that define safe operation of the computer-assisted medical system (e.g., spatial limitations, sanitization standards, etc.), embodiment rules that define capabilities and limitations of the computer-assisted medical system, associated instruments and devices (e.g., degrees of freedom of robotic arm movement, imaging capabilities of endoscopes and imaging devices, volume capabilities of extraction and injection devices, etc.), and / or user preference rules (e.g., rules defined and input by a user or personnel for performing a procedure or task).

[0072] As described herein, the system 200 may be configured to adapt the constitutions of the system 200 and / or portions of the modules (e.g., modules 210, 220, 250, 260, 270) based on one or more user inputs. Accordingly, as illustrated, the system 200 also includes a user input module 230 configured to receive one or more user inputs and / or instructions from a user to adapt the constitutions in view thereof. For example, the user input module 230 may receive or otherwise detect user input that is indicative of a particular persona for the system 200 to adopt when performing all or a portion of the autonomous or semi-autonomous tasks described herein. As used herein a “persona” of the system 200 may include sets of constitution rules and / or parameter values for the various modules and machine learning models that cause the system 200 to operate the computer-assisted system 100 and, particularly, the repositionable structures 120 in a manner that conforms to identified characteristics of prior historical operation of one or more historical repositionable structures by one or more historical users.

[0073] Accordingly, the user input module 230 includes a user input detection model 232 via which the system 200 detects the user input. For example, a user may interact with a user interface (e.g., a chatbot interface, an intraoperative imaging interface, etc.) to provide the user inputs, characteristics of historical operation for the one or more historical repositionable structures operated by the one or more historical users, and / or instructions. In this example, theIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC user input detection model 232 may detect the text entered into user interface. As another example, the user input detection model 230 may receive a data stream 202g indicative of the user input, characteristics of the historical operation, instructions, etc. In this example, one or more sensors or devices in an environment, of an instrument or auxiliary device, etc. may detect the user input (e.g., via an audio sensor or a video sensor) and process the data (e.g., by transcribing and / or generating a label indicative of a gesture) using the user input detection model 232.

[0074] The user input detection model 232 may be configured to determine that the user input relates to a persona for the system 200 to adopt. For example, the user input may specify one or more characteristics of how autonomous features of the system 200 are to be implemented. In response to this determination, the user input module 230 may query via, a query module 233, a data store that stores historical operational data (see e.g., external data store 800 shown in FIG. 8) to obtain a set of historical operational data relating to the indicated persona (e.g., historical operational data 802 shown in FIG. 8). For example, where the user input includes an indication of a persona of a particular doctor or operating team, the query module 233 may query for historical operational data that indicates how the particular doctor or operating team controlled and operated historical computer-assisted systems to perform different robot assisted procedures. In general, the historical data may include details such as historical endoscopic image data, historical operating room image data, historical kinematics data from the historical computer-assisted systems, historical haptics data historical computer-assisted systems, historical force data historical computer-assisted systems, historical shape sensing data, historical environmental data, historical intraoperative imaging, historical personnel identification, etc.

[0075] It should be appreciated that in some embodiments, the indicated persona may actually indicate multiple different personas that should be implemented. For example, the user input may indicate that tasks that automate lead surgeon activities are to operate using a first persona and tasks that automate lead nurse activities are to operate using a second persona. As a result, the user input may direct system 200 to implement a team of personas that replicate persons that would otherwise be unable to physically perform the procedure with one another.

[0076] In some embodiments, the user input relating to the persona can also include an indication of one or more conditions under which the system 200 should adopt the indicated persona. In particular, the one or more conditions may include procedure phases, instrument types, actions of the repositionable structures 120, or specific tasks to be performed during theIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC procedure for which the system 200 should adopt one or more personas specified by the user input. In these embodiments, the historical operational data relates to operation of the historical repositionable structures as directed by the one or more historical users under the one or more conditions. For example, in cases where the user input includes an indication of a first persona to adopt when performing a first task and a second persona to adopt when performing a second task, the historical operational data may include first data portions relating to the first persona and the first task and second data portions relating to the second person and the second task.

[0077] The user input module 230 may then analyze the user input and / or the historical operational data from using a modification generation model 235 to generate modifications for the various constitutions and / or modules of the system 200 described herein. These modifications when applied to the various constitutions and / or modules modify operation of the system 200 to act based on the user input and / or the historical operational data. For example, applying the modifications to the constitutions and / or modules may cause the system 200 to adopt a persona specified in the user input generally for all operations of the system 200 or for operation that are applicable to specified conditions. The system 200 may continue to adopt personas specified in the user input until further alterations are triggered by future user inputs.

[0078] In general, the modifications to the conditions may include rules to include in the constitutions that direct operation of the modules of the system 200 to perform actions in a manner that conforms to the specified persona. The modifications to the modules may include different parameter values to include in machine learning models of the various modules described herein. In particular, the different parameter values may be generated by fine tuning the models using the historical operational data as training data. In some embodiments, the system 200 includes a dedicated chipset and / or partition thereof for fine-tuning models locally based on the obtained historical data.

[0079] As one example, the historical operational data may indicate a certain sequence of tasks, certain positioning of the end effectors prior to interacting with a worksite, etc. that the system 200 should replicate to adopt the persona indicated the user input. In particular, the modification generation model 235 may generate or select rules to include in the constitutions and / or fine tune the various models so that, in operation, the system 200 may perform tasks in the certain order indicated by the historical operational data, position the end effectors as indicated by the historical operational data. etc.

[0080] In some embodiments, the user may specify the persona-related preferences to the system 200 prior to or during the procedure. In some embodiments, the user may specify theIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC persona-related preferences prior to the procedure using a computer system that is distinct from the control system for the computer-assisted system 100. For example, the user may indicate the persona-related preferences using a client device (e.g., a workstation computer, a mobile phone, computer tablet, etc.) that is communicatively coupled to a server associated with the computer- assisted system 100. In these embodiments, the server may preconfigure or modify the machine learning models and / or constitutions in the manners described herein such that the system 200 is pre-configured to adopt the persona indicated by the user when the computer-assisted system 100 is initialized to perform the subject procedure.

[0081] As another example, a user may provide an instruction to begin a certain phase of a procedure. In response, the system 200 may update the task generation constitution 211 (and / or other constitutions described herein) to include procedure rules associated with the new phase of the procedure. To this end, because the task generation model 215 has a limited context window, swapping in the procedure rules for the particular phase of the procedure enables the task generation constitution 211 to be a smaller size, thereby enabling faster processing of the prompt of the ability to consider additional inputs to generate tasks more tailored to the current situation. The system 200 may also update the task generation module 210 (and / or the other modules described herein) to include new machine learning model parameters associated with the new phase of the procedure. The updated procedure rules and model parameters may include rules and parameters that cause the task generation module 210 (and / or the other modules described herein) to operate based on the historical operational data. For example, the system may be configured to select or generate the procedure rules based on the historical operational data and generate the new model parameters by fine tuning models of the task generation module 210 (and / or other modules described herein) using the historical operational data as training data.

[0082] Similarly, the user input may indicate (or confirm an automated determination based on event data) that a particular instrument and / or end effector has been coupled to the manipulator system. In response, the system 200 may modify the embodiment rules to include rules specific to the new instruments I end effectors and remove rules related to instruments I end effectors decoupled from the manipulator system. Again, this enables the constitutions described herein to include rules more tailored to the current situation, thereby reducing the size of the constitution. The system 200 may also update the parameters of the machine learning models for the various modules described herein to include parameters generated from fine tuning of the models with data regarding use of the new instruments I end effectors. The updated embodiment rules and parameters may also include rules and parameters that cause the one or more of the modules described herein to operate based on the historical operational data (e.g., rules selected orIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC generated to cause such operation and / or model parameters fine-tuned using the historical operational data as training data.

[0083] While the foregoing describes various changes to the rules included in the disclosed constitutions and / or the parameters of the disclosed machine learning models, in other embodiments, other changes can be provided for the system 200 to adopt the indicated persona. For example, the historical operational data may indicate to change a state of the procedure video data stream 202e that includes a particular view. Accordingly, the user input module 230 may be configured to output a task to modify the state of the device (e.g., an endoscope) utilized to generate procedure video data stream 202e such that the instrument arm that supports the endoscope is positioned to provide a field of view that conforms to the historical operational data. As another example, the historical operational data may indicate that the force data stream 202a is always considered. Accordingly, the user input module 230 may be configured to instruct the robotic action module 250 to always embed the force data stream 202a when generating inputs to the RT model 315. As yet another example, the historical operational data may indicate how to format the action tokens and / or detokenize the same. For example, the historical operational data may indicate or otherwise describe a relative degree by which a functionality supported by an instrument is to be activated (e.g., that a gripper should respond more strongly to operator controls, how hot a heating element should be set, etc.) to adjust the specific parameter values output by the robotic controller 325 in response to a given action token generated by the RT model 315.

[0084] In some embodiments, the modification generation model 235 may be a neural network, a LLM, or a LMM configured to accept the user inputs and / or the historical operational data as an input and output one or more modifications to the system 200 that reflect the user inputs (e.g., rules for the constitutions and / or model parameters for the modules). For example, the modification generation model 235 (or other model of the system 200) may be configured to generate or provide a constitution to one or more modules 210, 220, 250, 260, and / or 270 that include rules based on the historical operational data. In some embodiments, the modification generation model 235 may also be configured to initiate a fine-tuning process for the machine learning models of the one or more modules 210, 220, 250, 260, and / or 270 based on the historical operational data or select fine-tuned parameter values to load into the model.

[0085] In examples, the modification generation model 235 may analyze the data streams 202 (such as the even data stream 202b) to determine a configuration of the manipulator system (e.g., the types of components coupled to the manipulator system). The modification generation modelIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC235 may also analyze the historical operational data to determine characteristics of the historical operations data that are relevant to the determined configuration of the manipulator system (e.g., characteristics relevant to the types of components coupled to the manipulator system). In response, the modification generation model 235 may obtain a set of rules stored in a rules database for inclusion in the various constitutions, and / or obtain a set of parameter values from a parameter database for inclusion in the various machine learning models. In some embodiments, the rules database and the parameter database may be maintained at a data store 234.

[0086] The obtained set of rules and / or parameters may be rules or parameters that are associated with the determined configuration of the manipulator system and the characteristics of the historical operational data that are relevant to the determined configuration. For example, the system 200 may implement a neural network or other model that is configured to extract characteristics of the historical operational data and map the extracted characteristics to a set of rules that most closely implements those characteristics. It should be appreciated that in some implementations, there is a separate neural network and / or model that maps the characteristics to different sets of rules that are integrated into different constitutions input into the disclosed models.

[0087] Similarly, the modification generation model 235 may detect a user input indicative of a procedure being performed (or otherwise derive the procedure type through an analysis of the data streams 202). The modification generation model 235 may also analyze the historical operational data to determine characteristics of the historical operations data that are relevant to the procedure being performed. In response, the modification generation model 235 may obtain a set of rules stored in the rules database and / or a set of parameter values stored in the parameter database that are associated with the procedure and the characteristics of the historical operations data that are relevant to the procedure. In some embodiments, these procedure rules and / or parameters may include a set of rules and parameters that generally govern the performance of the procedure and / or a set of rules or parameters that govern a specific phase of the procedure. In these embodiments, the modification generation model 235 may obtain the sets of rules or parameters associated with the initial phase of the procedure. After generating the constitutions, the user input module 230 may then provide the generated constitutions to the respective modules 210, 22, 250, 260, and / or 270.

[0088] The rules database may store pluralities of sets of rules including different sets of embodiment rules, procedure rules, safety rules, etc. and the modification generation model 235Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC may determine one or more specific sets of rules to provide to a constitution based on a user input and / or the historical operational data. For example, the rules database may store operational mode rules associated with different modes of operation (e.g., prioritizing autonomous tasks, prioritizing operator guidance or semi-autonomous tasks, prioritizing the speed of a procedure, performing a specified or specific action, etc.). In some embodiments, the modification generation model 235 may include a neural network trained to map the user inputs to specific sets of rules that implement the user input and / or the historical operational data. Additionally or alternatively, the modification generation model 235 may include a generative Al model to automatically generate new rules and / or supplement a stored set of rules with additional rules based on the user inputs and / or the historical operational data. It should be appreciated that the prompt to the generative Al model that generates new rules may include pre-defined instructions significantly restricting the creativity of the generative Al model to improve the likelihood the new rules are effective at restricting the disclosed models in the intended manner.

[0089] Additionally, in embodiments where the user input module 230 is coupled to a chatbot interface, the user input module 230 may determine certain responses to provide to the user to solicit user input and preferences for a given environment, situation, or procedure. For example, the user input module 230 may receive video data from the data streams 202 and may determine that an anomaly has occurred during an operation, or that required personnel are not present in an operating room for performing a procedure. The user input module 230 may then provide a response to the user via a display, or via audio to solicit a preference for performing one or more new tasks responding to the anomaly. In some embodiments, the one or more new tasks may include tasks that are identified and presented to the user for approval. For example, the system 200 may identify the one or new tasks as tasks previously performed to respond to the identified anomaly. Based on user input selecting one of the identified new tasks, the modification generation model 235 may then implement the selected task. In some embodiments, implementing the selected task includes selecting a new set of rules to be added to one or more constitutions (such as the task selection constitution 221 ) to carry out the selected new task.

[0090] In other examples, the user input module 230 may determine and provide one or more responses to solicit user input based on the data streams 202, input from personnel, patient data, information pertaining to a procedure or a phase of a procedure, or other data and information. The user may then provide the solicited input in response to the provided prompts, and the user input module 230 may then identify the user input, identify any modifications, and then generate and perform the modifications to the data streams 202, constitutions 21 1 , 221 , 251 , 261 , and 271 , and / or modules 210, 220, 250, 260, and 270. It should be appreciated that while constitutionsIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC211 , 221 , 251 , 261 , and 271 are indicated by separate reference numbers, in some embodiments, a single constitution may be utilized as an input to multiple of the modules 210, 220, 250, 260, and 270. Likewise, each of the modules 210, 220, 250, 260, and 270 and the machine learning models contained therein may be combined into a single module and / or model.

[0091] Furthermore, control of the system 200 may be accomplished in relation to user inputs and preferences that are not related to a particular persona and the historical operational data. More details of such operation are shown and described in U.S. Provisional Application No. 63 / 667,234, the entire disclosure of which is hereby incorporated by reference.

[0092] While the foregoing generally describes the application of personas to autonomous implementation of tasks, similar techniques may be implemented for manually-implemented tasks. For example, for personnel that do not operate robotic equipment in furtherance of the procedure, the system 200 may still provide other outputs to indicate which tasks the personnel should perform and when. Accordingly, the system 200 may apply the disclosed persona techniques to present the outputs and / or sequence of outputs in a manner that conforms to a persona. For example, to present the outputs in a manner that conforms to a persona, the system may extract linguistic data about instructions provided by historical personnel and utilize the extracted linguistic data as inputs that modifies how the outputs are generated (e.g., by adapting a natural language generation model to provide outputs that appears as if provided by the historical personnel). As another example, indicated personnel may prefer to perform certain manual tasks in a particular order and / or manner. Accordingly, the system may provide the outputs such that the content of the output indicates the particular manner and the sequencing of outputs implements the particular order.

[0093] FIG. 3 is a schematic diagram of a robotic action module 250 (such as the robotic action module 250 of FIG. 2), for generating actions tokens to control a repositionable structure (such as the repositionable structures 120 of FIG. 1 ). The robotic action module 250 may be implemented as part of the Al assist module 180 of FIG. 1. As described with respect to FIG. 2, the robotic action module 250 may be configured to receive as inputs one or more data streams forming the multi-modal data 202 or a subset of selected data streams 252, an indication of a selected task 254, an indication of one or more specific assigned tasks 256, and a constitution such as the robotic action constitution 251 of FIG. 2.

[0094] As illustrated, the robotic action module 250 includes an embedding stage 310 that embeds the various data modalities (e.g., text, image, audio, video, sensor data, etc.) into a common data space for input into an RT model 315. The embedding stage 310 may be configuredIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC to project the input data streams into a common vector space. In examples, the embedding stage 310 may implement one or more embedding methods including, without limitation, a word2vec, GloVe, ELMo, BERT, principal component analysis, singular value decomposition, a transformer model, Doc2Vec, paragraph vectors, convolutional neural networks, pre-trained models for image embedding, node embeddings, iknowledge graph embeddings (e.g., TransE, TrensR, DistMult, etc.), word embeddings, graph embeddings, entity embeddings, or another type of embedding supported by the RT model 315. It should be appreciated that while the embedding stage 310 is depicted as a single block, each data stream 252 may be associated with a respective embedding model for generating embeddings of the respective data type included within the embedding stage 310. Further, while embedding stage 310 is depicted as a component of the robotic action module 250, in some embodiments, the embeddings generated by the embedding stage 310 are input into the various models of the task generation module 210, the task selection module 220, and the task assignment module 230. The robotic action constitution 251 may include rules for how the embedding stage 310 embeds the data streams, and which data streams 252 the embedding stage 310 embeds, which may be selected or generated based on the historical operational data.

[0095] The various embedding models implemented by the embedding stage 310 may be trained and / or fine-tuned using historical data of prior procedures. In particular the embedding models implemented by the embedding stage 310 may be fine-tuned using the historical operational data as training data. In these embodiments, characteristics of historical actions that were identified from the historical operational data by the user input module 230 may be used as training data. In some embodiments, the tuning is performed locally at the system 200 (e.g., via a dedicated chipset for tuning models). In other embodiments, the tuning may be performed remote from the system 200. In these embodiments, the model data may be obtained by the system 200 to update the appropriate models with the fine-tuned values.

[0096] As one example, an embedding model to analyze a procedure video data stream 252a may be trained to identify objects (e.g., instruments, ports, anatomical features, etc.) that are expected to be seen during the procedure when the system 200 has adopted the persona associated with the historical operational data. In this example, the embedding model for the procedure video data stream 252a may be trained and / or fine-tuned using historical image data portions of the historical operational data in which the pixels representative of the corresponding objects are labels. The embedding model may then be trained or tuned in any suitable manner that minimizes loss with respect to the set of ground truth labels. For example, the robotic action constitution 251 may provide the rules for how the RT model 315 is to analyze the embeddings to produce a corresponding action token 320.Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC

[0097] The robotic action module 250 then provides the embedded data output from the embedding stage 310 as an input to the RT model 315. The RT model 315 may implement any type of robotic transformer architecture configured to convert embeddings into action tokens for control of a robotic system. One such suitable architecture is the RT-2 architecture by DeepMind, but other RT model architectures are envisioned. The RT model 315 is configured to process the embedded data from the embedding stage 310 to generate action tokens 320 for controlling the instruments, auxiliary devices, and / or repositionable structures to perform the selected tasks. The action tokens 320 may be textual descriptions of robotic actions that the selected arm, instrument, and / or auxiliary device are to perform to implement the selected task.

[0098] It should be appreciated that the particular tokens supported by the manipulator system may vary between models. Accordingly, the system 200 may be coupled to a library of RT models 315 each fine-tuned to generate action tokens supported by a different manipulator system type. Furthermore, the library of RT models 315 may be further sub-divided based on fine-tuning with respect to different characteristics of the historical operational data associated with different personas. As a result, the particular outputs of the tuned RT model 315 are adapted to the actual manipulator system associated with the system 200 and / or an identified persona. The robotic action constitution 251 may define the rules for the RT model 315 for how to generate the action tokens 320 for performing a procedure. For example, the robotic action constitution 251 may include procedure rules, and embodiment rules, among other sets of rules that determine how the RT model 315 generates the action tokens 320 to perform certain tasks and procedures, given a specific manipulator system type.

[0099] Additionally, as shown in FIG. 3, the user input module 230 may supply modified rules for the robotic action constitution 251 . As described herein, the user input module 230 may select or generate these modified rules based on the historical operational data associated with a specified persona. In particular, the user input module 230 (or other component or module of the system 200) may identify, from the historical operational data, the characteristics of historical actions performed using the one or more historical repositionable structures. These characteristics may include details such as historical force applied (e.g., maximum force, mean force, force duration), movement patterns (e.g., when suturing), tool activation levels (e.g., heat level on a cauterizing tool), etc. From the identified characteristics, the user input module 230 (or other component or module of the system 200) may select or generate, as robot action rules for the robotic action constitution 251 , rules that direct the RT model 315 to generate action tokens that correspond to the characteristics of the historical actions. The user input module 230 mayIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC likewise select rules that control the operation of the embedding stage 310 based on the characteristics of historical actions.

[0100] The robotic action module 250 then inputs the action tokens 320 to a robotic controller 325 that de-tokenizes the action tokens 320 into actual control commands the control operation of instruments, arms, and / or auxiliary devices according to the assigned tasks 256. In examples, to perform an assigned task 256, a de-tokenized action token may include a plurality of commands including movements of one or more repositionable structures, control of an action of an instrument (e.g., power up, power down, inject, extract, incise, etc.), control of an auxiliary structure (e.g., surgical bed, display device, etc.), or another command for performing a given task. In some embodiments, the de-tokenized control commands are signals suitable for various types of robotic control architectures that may be implemented at the manipulator system. For example, the de-tokenized control commands may be a signal input into a proportional-integral- derivative (PID) controller that controls a particular joint, or a model predictive control (MPG) signal, and / or other types of control signals. As described further herein, the de-tokenized commands may be specific to a given robotic system or machine. For example, a given system may include manipulators with more degrees of freedom of motion than another system, or an instrument, such as a camera, may have a wider field of view than another camera, and the de- tokenized commands may differ depending on the various capabilities and parameters for a given robotic system.

[0101] As illustrated, in some embodiments the robotic controller 325 may also accept the robotic action constitution 251 as an input to provide rules for how actions tokens are to be converted to specific parameter values. For example, the embodiment rules may include parameter range values for controlling function degrees of freedom associated with the controlled instruments. The rules in the robotic action constitution 251 that control the robotic controller 325 may also be generated or selected based on the characteristics of the historical actions as described herein. Furthermore, in some embodiments, the user input module 230 (or other component of the system 200) may directly modify the robotic controller 325 so that the action tokens are converted into the specific parameter values in manners consistent with the historical operational data. Accordingly, by dynamically adapting the robotic controller 325, the system 200 is able to ensure that the action tokens can be implemented across a wide variety of instrument types without needing to fine-tine the RT model 315 for each new instrument type.

[0102] FIG. 4 is a schematic diagram of the task generation module 210 for determining tasks that can be performed based on a current state of the manipulator system and / or procedure asIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC reflected by the multimodal data streams 202. As shown in FIG. 2, the task generation module210 is configured to receive the multi-modal data streams 202 and the task generation constitution211 as inputs. Similar to the embedding stage 310 of the robotic action module 250, the task generation module 210 may embed the data streams 202 for input into a task generation model 215. In some embodiments, the embedding models used to embed the data streams 202 are the same as the embedding models included in the embedding stage 310 of the robotic action module 250. The task generation constitution 251 may provide rules for how the task generation module 210 embeds the data streams 202 (including those selected or generated based on the historical operational data).

[0103] For example, the embedding model for an endoscopic data stream may be routed to a vision-language model (VLM) 212 configured to output natural language descriptions of a scene depicted by the image data output by an endoscopic instrument. In surgical applications, for example, the VLM 212 may be configured to identify objects, such as surgical instruments and devices (e.g., surgical beds or tables, medical devices, display devices, etc.), individuals and personnel (e.g., clinicians, doctors, medical technicians, etc.) depicted by an image data stream.

[0104] As another example, the embedding model of the event data stream may be configured to analyze the time series event data using a transformer model 214 to generate an output token identifying a particular phase of the procedure. It should be appreciated that while the term “transformer model” is used herein, in other embodiments, other model architectures suitable for analyzing temporal dependencies may be implemented. These output tokens may be input into the task generation model 215 to generate tasks that are in furtherance of operation typically associated with the identified phase. Similarly, the transformer model 214 may be configured to detect the completion of a phase or sub-phase of the procedure and generate an output token identifying the transition. These output tokens may be input into the task generation model 215 to begin generating tasks to implement actions associated with a subsequent phase or subphase. In some embodiments, output tokens indicative of the procedure phase may also be provided to the user input module 230 to update the constitutions with the set of procedural rules associated with the new procedure phase, which may include rules selected or generated based on the historical operational data. Additionally, the transformer model 214 may be configured to detect anomalous operation associated with phase of the procedure and output token indicative of the anomaly (e.g., a token indicative that anomalous operation is occurring, a token indicating a particular type of anomaly that is occurring, and so on). These output tokens may be input into the task generation model 215 to generate tasks to correct the anomaly. In some embodiments,Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC the task generation model 215 may generate the tasks to correct the anomaly in a manner that conforms to how the anomaly was handled in the historical operational data.

[0105] In addition to generating tokens based on time-dependencies in the event data stream, the transformer model 214 may also be configured to output tokens based on an instantaneous representation in the event data stream. To this end, the event data in the event data stream may indicate the complete state of the robotic components of the manipulator system at any given time. Accordingly, the transformer model 214 may be configured to identify equipment operatively coupled to the control system (e.g., manipulator arms, instruments, auxiliary devices) to generate an inventory of devices. In some embodiments, the inventory may also be input to the task generation model 215 such that the task generation model 215 understands that capabilities of the control system and generates tasks that can be implemented thereby.

[0106] In addition to identifying equipment currently coupled to the control system, an embedding model for another data stream may be configured to detect equipment that is otherwise available for use in the procedure (e.g., sterilized equipment on-hand in an operating room, equipment available elsewhere at the site). For example, the on-hand equipment may be detected via an operating room video data stream or via an equipment scheduling data stream. Additionally, the task generation module 215 may receive data indicative of personnel that are currently available, will be available for a scheduled procedure, or are required to perform one or more tasks via the data stream indicative of personnel and data associated with the personnel data stream 202f. In some embodiments, if the task generation module 210 detects the presence of new equipment, the task generation module 210 may generate an input to the user input module 230 such that the user input module 230 can update the constitutions with the appropriate embodiment rules.

[0107] The tokens generated by the embedding models may then be input into the task generation model 215. The task generation model 215 may be a LMM configured to directly accept image data streams or an LLM that accepts the natural language description(s) of the image data stream(s) generated by the VLM 212. The task generation model 215 may then generate tasks in furtherance of the detected (or subsequent) phase of the procedure based on the available equipment, personnel, and any other conditions indicated by the input data streams 202. The task generation constitution 211 may include sets of rules that control how the task selection model 215 generates the tasks from the detected phase of procedure, available equipment, personnel, and other conditions detected or indicated by the input data streams 202.Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PCThese task generation rules may include rules that are selected or generated based on the historical operational data.

[0108] As shown in FIG. 4, the user input module 230 may provide modified rules for the task generation constitution 211. As described herein, the user input module 230 may select or generate these modified rules to conform to a specified persona. In particular, the user input module 230 (or other component or module of the system 200) may identify the historical operational data associated with the indicated persona and identify historical tasks performed in the historical operational data. More particularly, the user input module 230 (or other component or module of the system 200) may select or generate rules that direct the task generation model 215 to generate or identify tasks that conform to the historical tasks. The user input module 230 may likewise select rules that control the operation of the scene recognition model 212 and / or the transformer model 214. For example, the user input module 230 may input extracted characteristics of the historic operational data into a model that directs the user input module 230 to select a particular set of rules that causes the system 200 to behave in a manner that most closely conforms to the indicated persona (and / or historic operational data associated therewith).

[0109] Additionally, the user input module 230 may supply updated parameter values for the scene recognition model 212, the transformer model 214, and / or the task generation model 215 or initiate fine-tunning of the scene recognition model 212, the transformer model 214, and / or the task generation model 215 to generate the updated parameter values. The updated parameter values may be derived based on the historical operational data and / or the historical tasks identified therefrom. For example, the fine-tuning of the scene recognition model 212, the transformer model 214, and / or the task generation model 215 may be performed using the historical operational data that includes the historical tasks and the appropriate context for generating the historical tasks as training data.

[0110] It should be appreciated that the task generation model 215 may be configured to output multiple tasks to implement the actions associated with a particular phase of the procedure. For example, to implement a procedural phase associated with advancing the instruments to a worksite, the task generation model 215 may generate tasks associated with aligning the instruments with respective ports and advancing the instruments towards the worksite. It should be appreciated that the task generation model may be configured to generate multiple different sets of tasks for implementing the procedural phase. To this end, the different sets of tasks may be performed to implement a procedural phase. Similarly, the tasks within a set of tasks may be performed in different sequences. This enables the various different approaches to performing aIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC procedural phase to be evaluated such that an optimal set of tasks is ultimately selected for implementation.

[0111] FIG. 5 is a schematic diagram of the task selection module 220 configured to select one or more tasks and / or sets of tasks generated by the task generation module 210 for implementation by the control system. In some embodiments, the task selection module 220 may be configured to select the one or more tasks and / or sets of tasks to be implemented based on the historical operational data such that the system 200 adopts the indicated persona. As illustrated, the task selection module 220 may select the tasks in two stages. As shown in FIG. 2, the task selection module 220 is configured to receive the generated tasks from the task generation module 210 and the task selection constitution 221 as an input.

[0112] In the first stage, a task filtering model 222 is implemented to categorize the generated tasks for filtering. For example, the task filtering model 222 may categorize each task is capable of being performed autonomously, semi-autonomously, manually, or not at all. In some embodiments, the constitution 221 includes rules that define how to classify different categories of tasks as being capable of autonomous, semi-autonomous, or manual performance. Tasks that are not capable of being performed may be filtered out. It should be appreciated that while the task generation model 215 is generally configured to generate tasks in view of the configuration of the manipulator system, the generative nature of LLMs and LMMs may still result in the generation of tasks that cannot be performed. Accordingly, the task filtering model 222 may function as a sanity check mechanism on the task generation model 215 to ensure only feasible tasks are implemented.

[0113] In a second stage, the remaining tasks are input into a task selection model 225 via a prompt that also includes the task selection constitution 221. In some embodiments, the task selection model 225 is an LLM. The task selection model 225 is configured to analyze the potential tasks or sets of tasks to be performed and select a preferred task or set of tasks to implement as directed by task selection rules contained in the task selection constitution 221 . For example, the task selection model 225 may be configured to evaluate the tasks or set of tasks using one or more evaluation metrics (e.g., time to perform the task, confidence in ability to autonomously perform the task, an ease of manual implementation, a confidence in safe performance of the task, abilities or personnel, available personnel, or relevant tasks). In some embodiments, the evaluation metrics may include metrics for assessing similarity to historical tasks identified from the historical operational data.Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC

[0114] As shown in FIG. 5, the user input module 230 may supply modified rules for the task selection constitution 221 . As described herein, the user input module 230 may select or generate these modified rules based on the historical operational data associated with a specified persona. In particular, the user input module 230 (or other component or module of the system 200) may identify the historical tasks performed using the one or more historical repositionable structures. More particularly, the user input module 230 (or other component or module of the system 200) may select or generate rules that direct task selection model 225 to select task tasks that generally align with the historical tasks. In some embodiments, a metric indicative of similarity to the historically-selected tasks is used to rank and / or otherwise evaluate the tasks generated by the task generation module 210. The user input module 230 may likewise select and / or generate rules that control the operation of the task filtering model 222.

[0115] Additionally, the user input module 230 may supply updated parameter values for the task filtering model 222 and / or the task selection model 225 or initiate fine-tunning of the task filtering model 222 and / or the task selection model 225 to generate the updated parameter values. The updated parameter values may be derived from the historical operational data and the historical tasks identified therefrom in particular. For example, the fine-tuning of the task filtering model 222 and / or the task selection model 225 may be done using the selected historical tasks (and any appropriate contextual information that led to their selection) as training data.

[0116] In examples, the task selection constitution 221 may include task selection rules about the types of tasks or the sequence of tasks (e.g., which instrument to advance to a work site first) that a historical user prefers to perform. Accordingly, by inputting a prompt that includes the task selection constitution 221 into the task selection model 225, the tasks selected by the task selection model 225 will generally comply with the historical operational data and cause the system 200 to adopt the persona of the historical user.

[0117] The task selection model 220 then outputs the selected task(s) and provides the selected task(s) to the task assignment module 230, and the robotic action module 250.

[0118] FIG. 6 is a schematic diagram of an example data modality selection module 260 of FIG. 2 for determining task specific data streams for controlling a repositionable structure (such as the repositionable structures 120 of FIG. 1 ) to perform a task (such as the tasks selected by the task selection module 220). Accordingly, the data modality selection module 260 may be communicative coupled to the task selection module 220. The data modality selection module 260, as shown in FIG. 2, receives the tasks from the task selection module 220 and the modality selection constitution 261 as inputs.Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC

[0119] As illustrated, the data modality selection module 260 may include a task analysis model 262 configured to identify the data modalities required to perform an input task. The task specific data modalities may include one or more streams of data including, without limitation, one or more of image, video, audio, force feedback, user input, event, or instrument provided data streams. In some embodiments, the task analysis model is an LLM. In these embodiments, the task analysis model 262 may be configured to generate a prompt to the LLM that asks the LLM to identify the data streams required to perform the input task. In some embodiments, the input includes a description of the available multi-modal data streams 202 and / or a state thereof, as well as the modality selection constitution 261 . Additionally, the input may include a description of additional data streams that are not currently available and / or alternative states of the available data streams that may be achieved.

[0120] The modality selection constitution 261 may include sets of rules that govern how the task analysis model 262 identifies the various data modalities to perform an input task and for generating the prompt to the LLM. For example, the modality selection constitution 261 may include a set of procedural rules that define specific tasks and required data types for each given task and the task analysis model may identify the data modalities based on the required data types and tasks as defined by the procedural rules. Additionally, the modality selection constitution 261 may include phase specific rules that instruct the task analysis model on how to analyze the various data modalities to determine phase-specific data modalities required for various phases of a procedure. As shown in FIG. 6, the sets of rules for the modality selection constitution 261 may include the modified rules selected or generated by the user input module 230 to cause the system 200 to adopt a specified persona. Likewise, the user input module 230 may modify the parameter values of the task analysis model 262 to cause the system 200 to adopt the specified persona in the manner described herein.

[0121] Based on the inputs, the task analysis model 262 may be configured to output a set of data modalities required to perform implement the task and associate the input task with the corresponding set of data modalities. For example, the task analysis model 262 may identify a type of task associated with the input task (e.g., a repositioning task, an end effector activation task, a configuration task, etc.) to identify which data modalities are required to perform the input task such that non-necessary data modalities can be ignored when implementing the task thereby providing the above-described improvements in execution time by the robotic action module 250. For example, endoscopic image data may not be required to perform a task related to mounting a new instrument to a robotic arm. As another example, operating room image data may not beIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC required to ablate a target anatomy. As yet another example, a task to change the state of an auxiliary device may not require kinematic data associated with manipulator system.

[0122] If the required data modalities are available, the data modality selection module 260 may then output the task and data modalities to the robotic action module 250 for implementation at the appropriate time. If the required data modalities are not available, the modality selection constitution 261 may include rules that cause the data modality selection module 260 to generate an output to the task generation module 210 instructing the task generation module 210 of the need to make the additional data modalities available. In some embodiments, the output may be a plain text instruction (e.g., “enable endoscope video feed,” “couple endoscope to instrument X,” or “move endoscope to include view of target anatomy”). In response, the task generation module 210 may generate the instructed tasks to autonomously (or semi-autonomously) make the indicated data modalities available. It should be appreciated that the data modality selection module 260 or the robotic action module 250 may queue the originally-input task until detecting that the task to make the required data modality available has been successfully performed.

[0123] FIG. 7 is a schematic diagram of an example task assignment module 270 of FIG. 2 for assigning manual and / or semiautonomous tasks (such as the tasks selected by the task selection module 220) associated with a medical procedure. Accordingly, the task assignment module 270 may be communicative coupled to the task selection module 220. As shown in FIG. 2, the task selection module 220 is configured to receive the generated tasks from the task generation module 220 and the task assignment constitution 271 as an input.

[0124] As illustrated, the task assignment module 270 may include a task analysis model 272 configured to analyze the selected tasks and determine personnel to perform the selected tasks according to sets of rules of the task assignment constitution 271. In some embodiments, the task analysis model 272 is an LLM. In these embodiments, the task analysis model 272 may be configured to generate a prompt to the LLM that asks the LLM to identify the various personnel, equipment, robotic systems, etc. required to perform the input task, as well as the task assignment constitution 271 . For example, the task analysis model 272 may identify a type of task associated with the input task (e.g., a repositioning task, an end effector activation task, a configuration task, etc.) and identify which equipment and / or personnel are required to perform the input task. As one example, an anesthetist may be required for performing certain tasks (administering the anesthesia), while the anesthetist is not required for other tasks (manually repositioning a manipulator).Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC

[0125] After identifying the requirements to perform the task and / or subtasks thereof, the task assignment module 270 may input a prompt that includes the tasks, the data streams 202, and the task assignment constitution 271 into the task assignment model 275. It should be appreciated that the task assignment model 275 may be the same or different LLM as the task analysis model 272. In embodiments where the same LLM is used for both models 272, 275, the functionality of the models 272, 275 may be achieved via a single prompt to the LLM. In these embodiments, the task assignment constitution 271 input to the LLM with the prompt may include additional rules that govern the task assignment process.

[0126] In examples, the task assignment constitution 271 includes a set of rules defining how the LLM is to analyze the tasks to identify the required equipment and / or personnel for each task. As one example, the task assignment constitution 271 may include rules that define how to select between multiple instruments capable of performing a task. As one example, the task assignment constitution 271 may include rules the state that the instrument that has to move the least to accomplish the task should be assigned the task. As another example, the task assignment constitution 271 may include user preference rules that define how the user prefers the tasks be assigned (e.g., to a manipulator arm and / or instrument teleoperated by the user’s dominant hand). Accordingly, when the input task and the constitution 271 are input to the LLM, the LLM may output the task assignment that complies with the user preferences. Furthermore, as shown in FIG. 7, the sets of rules for the task assignment constitution 271 may include the modified rules selected or generated by the user input module 230 to cause the system 200 to adopt a specified persona. Likewise, the user input module 230 may modify the parameter values of task analysis model 272 and / or the task assignment model 275 to cause the system 200 to adopt the specified persona in the manner described herein.

[0127] In some embodiments, if the task involves semi-autonomous implementation (or an autonomous implementation with an assigned instrument), the task assignment module 270 may provide an indication of the task to the robotic action module 250 for implementation at the appropriate time.

[0128] FIG. 8 is a schematic diagram of an example user input module 230 of FIG. 2 for detecting user input, querying an external data store 234 (e.g., a data lake or similar that is remotely connected to the system 200) for historical operational data 802, and generation modifications for one or more data streams 202; constitutions 211 , 221 , 251 , 261 , and 271 ; and / or the modules 210, 220, 260, and 270 according to the user input and the historical operational data 802. As shown in FIG. 2, the user input module 230 is configured to receive one or moreIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC user inputs and preferences from a user interface or device, or may receive a data stream 202g indicative of a user input or preference or of additional patient data (e.g., operation or health history, current patient diagnosis and / or conditions, etc.).

[0129] As illustrated, the user input module 230 includes a user input detection model 232 that analyzes the user input data stream and data from one or more user input devices, and determines that a user input has been provided. The user input detection model 232 analyzes the user input to determine what types of user input are provided, and what, if any, modifications to the data streams 202 and / or constitutions 21 1 , 221 , 251 , 261 , and 271 should be performed. The user input detection model 232 also determines whether the received user input is indicative of specific persona and activated the query module 233 to query the external data store 800 for the historical operational data 802 that is relevant to any such specified persona.

[0130] The user input module 230 further includes the modification generation model 235 that receives the determined types of user input, determined modifications from the user input detection model 232, and / or historical operational data 802 that the query module 233 received from the external data store 800. The modification generation model 235 then generates the corresponding modifications and provides, or performs, the modifications to the data streams 202, the constitutions 21 1 , 221 , 251 , 261 , and 271 , and / or the modules 210, 220, 260, and 270. The modification generation model 235 may modify the data streams 202 by enabling / disabling the data sources associated with the data streams and / or by configuring the data sources to capture a particular set of data (e.g., by changing a field of view of an image sensor). Additionally, the modification generation model 235 may modify one or more of the constitutions 211 , 221 , 251 , 261 , and 271 by editing the constitution to add rules, remove rules, or further edit the rules and natural language text of a constitution. The modification generation model 235 may also modify one or more of the modules 210, 220, 260, and 270 by generating or selecting modified parameters for machine learning models of the modules.

[0131] As described above, the user input module 230 may further generate and initialize a constitution based on user input. For examples, the user input module 230 may initialize at least one of the constitutions 211 , 221 , 251 , 261 , and / or 271 , and the modification generation model 235 may generate the rules and natural language text to include in the constitutions, which may include the rules that are selected or generated to cause the system 200 to adopt the person specific in the user input. The user input module 230 may initialize any of the constitution and include any types of sets of rules in each constitution independently. For example, the user input module may initialize and / or edit any of the constitutions 211 , 221 , 251 , 261 , and 271 , to includeIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC one or more sets of foundational rules, safety rules, embodiment rules, preference rules, and / or other types of rules as described herein.

[0132] The foundational rules may include rules pertaining to the allowable robotic actions and limitations of robotic actions. For example, the foundational rules may define rules that prevent robotic actions from being performed that could harm a human or operator, rules that cause a robotic system to receive and follow orders provided by users, prevent a robotic system from harming a patient, prevent a robotic system from interfering with other devices and tasks, etc.

[0133] The safety rules may include rules that define actions and operations that are deemed safe for a given environment, operation, and scenario. For example, the safety rules may include rules that restrict or define spatial limitations to movement or operation of a robotic system, rules that pertain to maintaining sterility of a robotic system or instruments, rules that define safe use of specific instruments and devices, rules that limit a robotic systems abilities pertaining to known limits of a specific robotic system model, etc. The safety rules may be specific to a scenario or environment such as a certain institution may have different safety standards than other medical facilities and therefore the safety rules may reflect the various standards of institutions, states, regions, governments etc. Additionally, the safety rules may also be specific to certain personnel present or to performing tasks for an operation. For example, the safety rules may depend on the types of personnel and training of personnel available for performing an operation.

[0134] The embodiment rules may include rules that define capabilities and limitations pertaining to specific models of instruments, devices, and systems. For example, the embodiment rules may include rules pertaining to available degrees of freedom of motion, ranges of motion, imaging resolutions, video capabilities, audio recording parameters, etc. The embodiment rules may include rules that pertain to the dimensions and sizes of various instrument, device, or system components to control how a robotic system or instrument moves in an environment, and to prevent components from contacting other objects as required. Additionally, the embodiment rules may include rules that specify and govern actions that may be performed by specific instruments, devices, and systems such as rules that control insection, suction, imaging, injection, extraction, application of radiation, heating, etc.

[0135] The preference rules may include rules pertaining to rules that are defined by a user or based on user input and preferences and / or rules that implement the specified persona. For example, the preference rules may include rules that prioritize use one instrument or device over another based on a user provided preference or an analysis of the historical operational data 802, that prioritize use of a specific robotic arm or a manipulator system, control how one or moreIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC models (such as the task selection model 220) filters tasks into automatic, semi-automatic, and manual tasks based on user input, how the task assignment model 270 assigns tasks, what is displayed to a user, how data and prompts are displayed to a user, etc.

[0136] As another example, a set of rules may relate to safety and autonomy settings. For example, when performing a challenging or high-risk procedure, users provided inputs and / or the historical operational data 802 associated with the specified persona may indicate that the system 200 should reduce autonomy and leave more tasks for the human operator to perform. On the other hand, for easy or routine procedures, the user inputs and / or the historical operational data 802 may increase the autonomy of the robotic system to improve workflow efficiency. Accordingly, the rules in the task assignment constitution 271 may be adapted to such that the actor assigned to the generated tasks are aligned with the autonomy preferences indicated by the user or the specified persona.

[0137] While described as various specific sets of rules above, other sets of rules are envisioned to be included in the constitutions 211 , 221 , 251 , 261 , and 271. Additionally, any additional types and sets of rules may be generated by the user input module 230 and edited to be included in any of the constitutions.

[0138] FIG. 9A is a schematic diagram illustrating detokenizing action tokens to control a repositionable structure (such as the repositionable structures 120 of FIG. 1 ), an instrument, or an auxiliary device to implement a semi-autonomous task. Detokenization is a process under which a system converts an action token 320 (e.g., a text string or commands) output by the robotics transformer model 315 into actual robotic commands 330 (an “action sequence”) to control the indicated equipment, for example, by changing the pose or by activating or otherwise controlling a functionality supported by the indicated equipment.

[0139] As described above, the RT model 315 may be fine-tuned and / or selected based on the particular manipulator system model and / or the historical operational data 802. In examples, the robotic action constitution 251 may be input to the RT model 315 and include various sets embodiment rules that pertain to different manipulator system models. The robotic action constitution may instruct the RT model 315 to use specific model rules according to a corresponding manipulator system in use and the person specified by the user input. Additionally, the user input module 230 may identify a particular manipulator system model and edit the robotic action constitution 231 to include the corresponding rules for operating the particular manipulator system model generally or based on the historical operational data 802. In examples, knowledge of the capabilities of the manipulator system and the historical operational data 802 areIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC incorporated into the RT model 315 itself and / or in the robotic action constitution 251. Similarly, the RT model 315 may be configured to maintain and / or access a list of current equipment coupled to the control system. Accordingly, the action tokens 320 output by the RT model 315 may include a textual indication of the equipment that is to perform the action. For example, an output action token 320 may state “Move Instrument X coupled to Arm A to target work area” or “Use Instrument Y to cauterize incision.” Accordingly, the action tokens 320 and detokenization are specific to a robotic system or model based on the capabilities of a given robotic system and any specified personas.

[0140] In some embodiments, the action tokens 320 may be further tailored to the current state of the manipulator system and / or the historical operational data 802. For example, the RT model 315 may be configured to accept a kinematic data stream as an input such that the output action tokens indicate a pose for the instrument and / or arm in the robotic coordinate system. Accordingly, as one example, rather than outputting an action token 320 indicating that an instrument should move to the target work area, the output action token 320 may instead indicate “Move Instrument X to position (x, y, z) and orientation (a, p, y).” As a result, the action tokens 320 output by the RT model 315 are tailored to the specific state of the manipulator system improving the ability of the control system to accurately interpret and implement autonomously- generated control commands.

[0141] Similarly, because the RT model 315 is aware of the functionality supported by the instruments coupled to the control system, the RT model 315 is able to output commands that are specifically tailored to the implementing instrument. For example, rather than outputting an action token that states “Clamp blood vessel,” the RT model 315 may instead output an action token 320 that states “Place gripper around blood vessel at position (x, y, z) and engage grip to exert 50 pascals of force.” In these embodiments, the robotic action constitution 251 may include embodiment rules that define the amount of force (or other functional DOF parameter value) to implement the task. These embodiments rules may be rules that are selected or generated based on the historical operational data 802. For example, the defined amount of force may be identified from the historical operational data 802 and include an average or typical application force associated with the specified persona. As a result, adapting the robotic action constitution 251 input into the RT model 315 and / or the RT model 315 itself enables the generation of action tokens 320 that have improved alignment with the actual functionality supported by the control system, thereby enabling more precision in the generation of the action tokens 320 and more robust usage of instrument functionality.Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC

[0142] After the RT model 315 generates the action tokens 320, the action tokens 320 are input into the robotic controller 325 to convert the natural language action tokens into de-tokenized commands 330 (e.g., parameterized commands, such as PID parameter values or MPC parameter values, used by the control system actually realize the instructed actions). With simultaneous reference to FIGS. 9B and 90, depicted are example parameterized command structures utilized by the control system. In some embodiments, the robotic action constitution 251 is also input into the robotic controller 325 to provide rules for how to convert an action token into a parametric control value. These may include the modified rules selected or generated by the user input module 230 to cause the system 200 to adopt a specified persona. For example, analysis of the historical operational data 802 may indicate that the specified persona the default action tokens for a gripping command exert too little pressure and the user input module 230 may include, in the robotic action constitution 251 , a preference rule to scale up the output value for the pressure DOF by 5%.

[0143] The de-tokenized command 930 of FIG. 9B is configured to control operation of a gripper instrument that has 6 motive degrees of freedom (DOFs). Accordingly, the de-tokenized command 930 may include deltas by which the control system is to change each DOF. Additionally, the de-tokenized command 930 includes additional DOFs related to the functionality supported by the instrument. For example, in the illustrated example related to a gripper, the “gripper” DOF may indicate an amount of force the gripper should exert. Similarly, the de- tokenized command 931 of FIG. 9C is configured to control operation of a fluid extraction instrument that has 5 motive degrees of freedom (DOFs) and one functional DOF. It should be appreciated that the length of the de-tokenized control commands 330 may vary based on the number of motive and functional DOFs supported by the controlled equipment. Accordingly, the length of the de-tokenized commands 330 have fewer unused parameters, thereby enabling more efficient usage of control buses.

[0144] In some embodiments, the de-tokenized commands 330 may also control one or more user feedback devices, such as a display device. For example, a de-tokenized command 330 may be configured to cause a display unit to provide an instruction for executing a task that is to be performed manually or semi-autonomously by a user.

[0145] While FIGs. 2-9C describe a process for performing task assignment to personnel, instruments, and robotic systems to implement one or more tasks, it should be appreciated that the disclosed process may be repeated throughout the procedure until its completion. As a result, each stage of the procedure may be discretized into tasks, that are converted into de-tokenizedIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC commands such that any portion of the procedure can be implemented via closed-loop autonomous control, semi-autonomous control, or manually by personnel. It should be appreciated that in some embodiments, the modules 210, 220, and / or 230 may analyze the multimodal data stream 202 to anticipate future tasks, and assign the future tasks, that are predicted to be performed based on a current state. Thus, the tasks analyzed by the modules 210, 220, and / or 230may differ from the task being implemented by the robotic action module 250 or by personnel. As a result, the control system is able to anticipate future actions and future task assignments in order to ensure closed-loop control of the overall procedure occurs more efficiently.

[0146] In some embodiments, the control system may further include a scheduler module configured to generate a sequence of expected tasks (and their corresponding data modalities and / or assigned personnel, instruments, robotic systems, or auxiliary devices) and their corresponding triggers for execution. Accordingly, when the scheduler detects, based on the multimodal data streams 202, that a trigger event has occurred, the scheduler can route one or more subsequent tasks to robotic action module 250 or provide indications of a specific task to respectively assigned personnel. It should be appreciated that if the multimodal data streams 202 indicate that conditions have changed (e.g., an anomalous condition is detected), the scheduler can adjust the priority and / or sequencing to prioritize addressing the current conditions.

[0147] FIG. 10 is a flow diagram of a method 1000 for determining and performing tasks for a medical procedure using adaptive Al based on user input. The method 1000 may be performed by a processor system or a control system (such as the processor system 150 and control system 140 of FIG. 1 ). In some embodiments, the control system may implement an Al-assist module (such as the Al-assist module 180) to perform the functionality described with respect to the machine learning models. In implementations, the method 1000 may further be performed with a system including one or more repositionable structures that may be operably coupled to one or more instruments (such as the instruments 122).

[0148] At block 1010, the method 1000 includes detecting a user input indicative of characteristics of historical operation for one or more historical repositionable structures operated by one or more historical users.

[0149] At block 1020, the method 1000 includes querying historical operational data based on the characteristics of historical operation. In response to the query the method 1000 may include receiving the historical operational data from a data store based on the indication of the one or more historical users. The historical operational data may includes one or more of historicalIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC endoscopic image data, historical operating room image data, historical kinematics data, historical haptics data, historical force data, historical shape sensing data, historical environmental data, historical intraoperative imaging, historical personnel identification.

[0150] At block 1030, the method 1000 includes configuring or modifying at least one of a task generation machine learning model (e.g., the task generation model 215), a task generation constitution (e.g., the constitutions 211 ), a task selection machine learning model (e.g., the task selection model 225), a task selection constitution (e.g., the task selection constitution 221 ), a robotic transformer model (e.g., the RT model 315), or a robotic action constitution (e.g., the robotic action constitution 251 ) based on the historical operational data (e.g., historical operational data 802).

[0151] In some embodiments, configuring or modifying the task generation constitution based on the historical operational data includes selecting or generating, based on the historical operational data, task generation rules that direct how the task generation machine learning model identifies the tasks; and performing at least one of: adding the task generation rules to the task generation constitution, and replacing a set of rules in the task generation constitution with the task generation rules. Selecting or generating the task generation rules based on the historical operational data includes identifying historical generated tasks from the historical operational data; and selecting or generating rules that direct the task generation machine learning model to identify tasks that align with the historical generated tasks.

[0152] Additionally or alternatively, in some embodiments, the configuring or modifying the task generation machine learning model includes tuning the task generation machine learning model based on the historical operational data to cause the task generation machine learning model to identify the tasks based on the historical operational data. Tuning the task generation machine learning model based on the historical operational data includes identifying historical generated tasks from the historical operational data; and tuning the task generation machine learning model using the identified historical generated tasks as training data.

[0153] Configuring or modifying the task selection constitution may include: selecting or generating, based on the historical operational data, task selection rules that direct how the task selection machine learning model evaluates the tasks; and performing at least one of: adding the task selection rules to the task selection constitution, and replacing a set of rules in the task selection constitution with the task selection rules. Selecting or generating the task selection rules based on the historical operational data includes: identifying historical tasks performed during historical procedures associated with the historical operational data; and selecting or generating,Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC as the task generation rules, rules that direct the task generation machine learning model to select tasks based on similarity to the performed historical tasks.

[0154] Configuring or modifying the task selection machine learning model may also include tuning the task selection machine learning model based on the historical operational data to cause the task selection machine learning model to select identified tasks based on the historical operational data. Tuning the task selection machine learning model based on the historical operational data includes identifying historical tasks performed during historical procedures associated with the historical operational data; and tuning the task selection machine learning model using the identified historical tasks as training data.

[0155] Configuring or modifying the robotic action constitution may include selecting or generating, based on the historical operational data, robot action rules that direct how robotics transformer model generates the action tokens; and performing at least one of: adding the robot action rules to the robotic action constitution, and replacing a set of rules in the robotic action constitution with the robot action rules. Selecting or generating the robot action rules based on the historical operational data includes identifying, from the historical operational data, characteristics of historical actions performed using the one or more historical repositionable structures; and selecting or generating, as the robot action rules, rules that direct the robotics transformer model to generate action tokens that correspond to the characteristics of the historical actions.

[0156] Configuring or modifying the robotics transformer model may include identifying, from the historical operational data, characteristics of historical actions performed using the one or more historical repositionable structures; and tuning the robotics transformer model using the characteristics of the historical actions as training data.

[0157] The characteristics of historical operation may include an indication of a first historical user and a second historical user. The historical operational data may include first historical operational data associated with the first historical user and second historical operational data associated with the second historical user. Additionally, the characteristics of historical operation include an indication of a first task associated with the first historical user and a second task associated with the second user. In these embodiments, the method includes configuring or modifying at least one of the task generation machine learning model, the task generation constitution, the task selection machine learning model, the task selection constitution, the robotic transformer model, or the robotic action constitution to operate based on: first portions of the historical operational data associated with the first historical to cause the one or moreIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC repositionable structures to perform the first task; and second portions of the historical operational data associated with the second historical user to cause the one or more repositionable structures to perform the second task.

[0158] In some embodiments, the characteristics of historical operation include an indication of one or more conditions; and the historical operational data relates to operation of the historical repositionable structures as directed by the one or more historical users under the one or more conditions. In these embodiments, configuring or modifying the task generation constitution, the task selection constitution, or the robotic action constitution based on the historical operational data and the one or more conditions includes selecting or generating, based on the historical operational data, sets of rules that direct how the task generation machine learning model, the task selection machine learning model, or the robotics transformer model operate when the one or more conditions are applicable; and performing at least one of: adding the sets of rules to the task generation constitution, the task selection constitution, or the robotic action constitution; or replacing rules in the task generation constitution, the task selection constitution, or the robotic action constitution with the sets of rules. The task generation constitution, the task selection constitution, or the robotic action constitution may include default rules or operator preference rules that direct how the task generation machine learning model, the task selection machine learning model, or the robotics transformer model operate when the one or more conditions are not applicable.

[0159] Configuring or modifying the task generation machine learning model, the task selection machine learning model, or the robotics transformer model may include tuning the task generation machine learning model, the task selection machine learning model, or the robotics transformer model using portions of the historical operational data that are associated with the one or more conditions as training data. The one or more conditions may include associated procedure phases, instrument types, actions of the repositionable structures, or specific tasks to be performed during the procedure. The one or more historical users may include a first historical user and a second historical user. The user input may include an indication of one or more additional conditions and the historical operational data may relate to operation of the historical repositionable structures as directed by the second historical user under the one or more additional conditions.

[0160] Configuring or modifying at least one of the task generation machine learning model, the task generation constitution, the task selection machine learning model, the task selection constitution, the robotic transformer model, or the robotic action constitution may includeIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC generating, based on the historical operational data, modifications for at least one of at least one of the task generation machine learning model, the task generation constitution, the task selection machine learning model, the task selection constitution, a robotics transformer model, or the robotic action constitution based on the historical operational data. In these embodiments the method 1000 may include receiving the user input via a user input module and inputting the user input into a user input machine learning model trained to generate the modifications. The modifications include at least one of: rules to include in the task generation constitution, the task selection constitution, or the robotic action constitution; or parameter values to include in the task generation machine learning model, the task selection machine learning model, or the robotics transformer model.

[0161] In some embodiments, the user input includes natural language text. In these embodiments, the method 1000 includes receiving audio data recorded from a microphone; and deriving the natural language text from the audio data.

[0162] The method 1000 may also include generating the user input based on a user interaction with a user interface presented on a display device. The user interaction includes an indication associated with intraoperative or preoperative image data presented by the display device.

[0163] At block 1040, the method 1000 includes receiving a plurality of data streams (e.g., data streams 202) from one or more data sources. The plurality of data streams includes one or more of endoscopic image data, operating room image data, kinematics data, haptics data, force data, shape sensing data, environmental data, intraoperative imaging, personnel identification, personnel procedure history, personnel training.

[0164] At block 1050, the method 1000 includes analyzing, via the task generation machine learning model, the data streams to identify a plurality of tasks to be performed by one or more repositionable structures (e.g., the repositionable structures 120). The one or more repositionable structures are configured to support respective instruments and are operably coupled to a control system (e.g., control system 140), and the task generation constitution is input into the task generation machine learning model to control how the task generation machine learning model analyzes the data streams.

[0165] At block 1060, the method 1000 includes selecting, via the task selection machine learning model, a task to be performed from the identified tasks, wherein the task selection constitution is input into the task selection machine learning model to control how the task selection machine learning model selects the task to be performed.Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC

[0166] At block 1070, the method 1000 includes generating action tokens (e.g., action tokens 320) for the selected task via the robotic transformer model. The robotic action constitution, one or more of the plurality of data streams, and an indication of the selected task are input into the robotic transformer model, and the robotic action constitution controls how the robotic transformer model generates the action tokens for the selected task based on the one or more of the plurality of data streams.

[0167] At block 1080, the method 1000 includes controlling, based on the action tokens, the one or more repositionable structures to perform the selected task.

[0168] As described herein, the method 1000 may be performed using various systems including different robotic system models with different capabilities. The ability of the control system to modify the data streams, machine learning models (via constitutions), and control of instruments, devices, and repositionable structures of robotics systems allows for the implementation of robotic systems in performing medical procedures across a broad range of environments and specific instances and conditions. Inclusion of the user input module for analyzing user input and modifying the system 200 enables adaptive Al for use in generating, filtering, and performing tasks via robotic systems in medical settings may overcome the need to retrain systems to specific environments and scenarios which reduces onboarding time of new devices and instruments as well as increases the efficiency and availability of operations and treatments.

[0169] One or more components of the examples discussed in this disclosure, such as control system 140, may be implemented in software for execution on one or more processors of a computer system. The software may include code that when executed by the one or more processors, configures the one or more processors to perform various functionalities as discussed herein. The code may be stored in a non-transitory computer readable storage medium (e.g., a memory, magnetic storage, optical storage, solid-state storage, etc.). The computer readable storage medium may be part of a computer readable storage device, such as an electronic circuit, a semiconductor device, a semiconductor memory device, a read only memory (ROM), a flash memory, an erasable programmable read only memory (EPROM); a floppy diskette, a CD-ROM, an optical disk, a hard disk, or other storage device. The code may be downloaded via computer networks such as the Internet, Intranet, etc. for storage on the computer readable storage medium. The code may be executed by any of a wide variety of centralized or distributed data processing architectures. The programmed instructions of the code may be implemented as a number of separate programs or subroutines, or they may be integrated into a number of otherIntuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC aspects of the systems described herein. The components of the computing systems discussed herein may be connected using wired and / or wireless connections. In some examples, the wireless connections may use wireless communication protocols such as Bluetooth, near-field communication (NFC), Infrared Data Association (IrDA), home radio frequency (HomeRF), IEEE 502.11 , Digital Enhanced Cordless Telecommunications (DECT), and wireless medical telemetry service (WMTS).

[0170] Various general-purpose computer systems may be used to perform one or more processes, methods, or functionalities described herein. Additionally or alternatively, various specialized computer systems may be used to perform one or more processes, methods, or functionalities described herein. In addition, a variety of programming languages may be used to implement one or more of the processes, methods, or functionalities described herein.

[0171] While certain examples and examples have been described above and shown in the accompanying drawings, it is to be understood that such examples and examples are merely illustrative and are not limited to the specific constructions and arrangements shown and described, since various other alternatives, modifications, and equivalents will be appreciated by those with ordinary skill in the art.

Claims

Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PCWHAT IS CLAIMED IS:1 . A computer-implemented method for performing automated tasks, the method comprising: detecting a user input indicative of characteristics of historical operation for one or more historical repositionable structures operated by one or more historical users; querying historical operational data based on the characteristics of historical operation; configuring or modifying at least one of a task generation machine learning model, a task generation constitution, a task selection machine learning model, a task selection constitution, a robotic transformer model, or a robotic action constitution based on the historical operational data; receiving a plurality of data streams from one or more data sources; analyzing, via the task generation machine learning model, the data streams to identify a plurality of tasks to be performed by one or more repositionable structures, wherein: the one or more repositionable structures are configured to support respective instruments and are operably coupled to a control system, and the task generation constitution is input into the task generation machine learning model to control how the task generation machine learning model analyzes the data streams; selecting, via the task selection machine learning model, a task to be performed from the identified tasks, wherein the task selection constitution is input into the task selection machine learning model to control how the task selection machine learning model selects the task to be performed; generating action tokens for the selected task via the robotic transformer model, wherein: the robotic action constitution, one or more of the plurality of data streams, and an indication of the selected task are input into the robotic transformer model, and the robotic action constitution controls how the robotic transformer model generates the action tokens for the selected task based on the one or more of the plurality of data streams; and controlling, based on the action tokens, the one or more repositionable structures to perform the selected task.

2. The computer-implemented method of claim 1 wherein configuring or modifying the task generation constitution based on the historical operational data includes:Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC selecting or generating, based on the historical operational data, task generation rules that direct how the task generation machine learning model identifies the tasks; and performing at least one of: adding the task generation rules to the task generation constitution, and replacing a set of rules in the task generation constitution with the task generation rules.

3. The computer-implemented method of claim 2 wherein selecting or generating the task generation rules based on the historical operational data includes: identifying historical generated tasks from the historical operational data; and selecting or generating rules that direct the task generation machine learning model to identify tasks that align with the historical generated tasks.

4. The computer-implemented method of claim 1 wherein configuring or modifying the task generation machine learning model includes: tuning the task generation machine learning model based on the historical operational data to cause the task generation machine learning model to identify the tasks based on the historical operational data.

5. The computer-implemented method of claim 4 wherein tuning the task generation machine learning model based on the historical operational data includes: identifying historical generated tasks from the historical operational data; and tuning the task generation machine learning model using the identified historical generated tasks as training data.

6. The computer-implemented method of claim 1 wherein configuring or modifying the task selection constitution includes: selecting or generating, based on the historical operational data, task selection rules that direct how the task selection machine learning model evaluates the tasks; and performing at least one of: adding the task selection rules to the task selection constitution, and replacing a set of rules in the task selection constitution with the task selection rules.Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC7. The computer-implemented method of claim 6 wherein selecting or generating the task selection rules based on the historical operational data includes: identifying historical tasks performed during historical procedures associated with the historical operational data; and selecting or generating, as the task generation rules, rules that direct the task generation machine learning model to select tasks based on similarity to the performed historical tasks.

8. The computer-implemented method of claim 1 wherein configuring or modifying the task selection machine learning model includes: tuning the task selection machine learning model based on the historical operational data to cause the task selection machine learning model to select identified tasks based on the historical operational data.

9. The computer-implemented method of claim 8 wherein tuning the task selection machine learning model based on the historical operational data includes: identifying historical tasks performed during historical procedures associated with the historical operational data; and tuning the task selection machine learning model using the identified historical tasks as training data.

10. The computer-implemented method of claim 1 wherein configuring or modifying the robotic action constitution includes: selecting or generating, based on the historical operational data, robot action rules that direct how robotics transformer model generates the action tokens; and performing at least one of: adding the robot action rules to the robotic action constitution, and replacing a set of rules in the robotic action constitution with the robot action rules.1 1 . The computer-implemented method of claim 10 wherein selecting or generating the robot action rules based on the historical operational data includes: identifying, from the historical operational data, characteristics of historical actions performed using the one or more historical repositionable structures; and selecting or generating, as the robot action rules, rules that direct the robotics transformer model to generate action tokens that correspond to the characteristics of the historical actions.Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC12. The computer-implemented method of claim 1 wherein configuring or modifying the robotics transformer model includes: identifying, from the historical operational data, characteristics of historical actions performed using the one or more historical repositionable structures; and tuning the robotics transformer model using the characteristics of the historical actions as training data.

13. The computer-implemented method of claim 1 wherein: the characteristics of historical operation include an indication of a first historical user and a second historical user; and the historical operational data includes first historical operational data associated with the first historical user and second historical operational data associated with the second historical user.

14. The computer-implemented method of claim 13 wherein, the characteristics of historical operation include an indication of a first task associated with the first historical user and a second task associated with the second user; and further comprising: configuring or modifying at least one of the task generation machine learning model, the task generation constitution, the task selection machine learning model, the task selection constitution, the robotic transformer model, or the robotic action constitution to operate based on: first portions of the historical operational data associated with the first historical to cause the one or more repositionable structures to perform the first task; and second portions of the historical operational data associated with the second historical user to cause the one or more repositionable structures to perform the second task.

15. The computer-implemented method of claim 1 wherein: the characteristics of historical operation include an indication of one or more conditions; and the historical operational data relates to operation of the historical repositionable structures as directed by the one or more historical users under the one or more conditions.

16. The computer-implemented method of claim 15 wherein configuring or modifying the task generation constitution, the task selection constitution, or the robotic action constitution based on the historical operational data and the one or more conditions includes:Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC selecting or generating, based on the historical operational data, sets of rules that direct how the task generation machine learning model, the task selection machine learning model, or the robotics transformer model operate when the one or more conditions are applicable; and performing at least one of: adding the sets of rules to the task generation constitution, the task selection constitution, or the robotic action constitution; or replacing rules in the task generation constitution, the task selection constitution, or the robotic action constitution with the sets of rules.

17. The computer-implemented method of claim 16 wherein the task generation constitution, the task selection constitution, or the robotic action constitution include default rules or operator preference rules that direct how the task generation machine learning model, the task selection machine learning model, or the robotics transformer model operate when the one or more conditions are not applicable.

18. The computer-implemented method of claim 15 wherein configuring or modifying the task generation machine learning model, the task selection machine learning model, or the robotics transformer model includes: tuning the task generation machine learning model, the task selection machine learning model, or the robotics transformer model using portions of the historical operational data that are associated with the one or more conditions as training data.

19. The computer-implemented method of claim 15 wherein the one or more conditions include associated procedure phases, instrument types, actions of the repositionable structures, or specific tasks to be performed during the procedure.

20. The computer-implemented method of claim 15 wherein: the one or more historical users include a first historical user and a second historical user; the user input includes an indication of one or more additional conditions; and the historical operational data relates to operation of the historical repositionable structures as directed by the second historical user under the one or more additional conditions.21 . The computer-implemented method of claim 1 wherein configuring or modifying at least one of the task generation machine learning model, the task generation constitution, the task selection machine learning model, the task selection constitution, the robotic transformer model, or the robotic action constitution includes:Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC generating, based on the historical operational data, modifications for at least one of at least one of the task generation machine learning model, the task generation constitution, the task selection machine learning model, the task selection constitution, a robotics transformer model, or the robotic action constitution based on the historical operational data.

22. The computer-implemented method of claim 21 , further comprising: receiving the user input via a user input module; and inputting the user input into a user input machine learning model trained to generate the modifications.

23. The computer-implemented method of claim 21 wherein the modifications include at least one of: rules to include in the task generation constitution, the task selection constitution, or the robotic action constitution; or parameter values to include in the task generation machine learning model, the task selection machine learning model, or the robotics transformer model.

24. The computer-implemented method of claim 1 wherein, in response to the query the method includes: receiving the historical operational data from a data store based on the indication of the one or more historical users.

25. The computer-implemented method of any one of claims 1 -24 wherein the historical operational data includes one or more of historical endoscopic image data, historical operating room image data, historical kinematics data, historical haptics data, historical force data, historical shape sensing data, historical environmental data, historical intraoperative imaging, historical personnel identification.

26. The computer-implemented method of any one of claims 1 -24, wherein the plurality of data streams includes one or more of endoscopic image data, operating room image data, kinematics data, haptics data, force data, shape sensing data, environmental data, intraoperative imaging, personnel identification, personnel procedure history, personnel training.

27. The computer-implemented method of claim any one of claims 1 -24, wherein the user input includes natural language text.

28. The computer-implemented method of claim 27, further comprising:Intuitive Docket No.: P07025-WOAttorney Docket No.: 33685 / 70819 / PC receiving audio data recorded from a microphone; and deriving the natural language text from the audio data.

29. The computer-implemented method of any one of claims 1 -24 further comprising: generating the user input based on a user interaction with a user interface presented on a display device.

30. The computer-implemented method of claim 29 wherein the user interaction includes an indication associated with intraoperative or preoperative image data presented by the display device.31 . A computer-assisted system for performing automated tasks, the system comprising: one or more repositionable structures configured to support respective instruments; and a control system operably coupled to the repositionable structure, wherein the control system is configured to perform the computer-implemented method of any one of claims 1 -30.

32. One or more non-transitory, computer-readable media storing instructions that, when executed by a control system of a computer-assisted system, causes the control system to perform the computer-implemented method of any one of claims 1 -30.

Images