Systems and method for optimizing medical procedures

Abstract

The present disclosure relates to techniques for efficient performance of medical procedures. In some aspects, these improvements are achieved through improved intra- and inter-processor data transfer and processing, improved processor prioritization and parallelization, improved infrastructure management, and / or the implementation of an omnischeduler for efficient equipment management and / or task orchestration. These improvements also provide for improved application-layer capabilities, including improved digital twins and improved generation of synthetic data for training, for example, classification models and / or control policies.

Description

Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION SYSTEMS AND METHOD FOR OPTIMIZING MEDICAL PROCEDURESCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of the filing date of provisional U.S. Patent Application No.63 / 721,379 entitled “SYSTEMS AND METHODS FOR OPTIMIZING MEDICAL PROCEDURES,” filed on November 15, 2024. The entire contents of the provisional application are hereby expressly incorporated herein by reference.FIELD

[0002] Disclosed examples relate to a system and devices for more efficient operation of devices used in furtherance of a medical procedure.BACKGROUND

[0003] Use of specialized processing units such as Graphics Processing Units (GPUs) and Field-Programmable Gate Arrays (FPGAs) is increasingly common for handling complex processing tasks. However, the allocation of tasks between / among such specialized units in systems employing multiple or varied types of processing units presents considerable challenges, including suboptimal processing efficiency and complexities in data transfer that negatively impact overall system performance.

[0004] In specific applications such as robotic-guided medical procedures, the need for efficiently processing high-resolution image data with optimal efficiency is particularly challenging for typical systems. For example, transferring and processing image data between FPGAs and GPUs, while maintaining synchronization across various processing stages, introduces significant challenges. These challenges are compounded by the variable nature of workloads in such applications, which can fluctuate based on the procedure, the resolution and frame rate of captured images, and the specific processing tasks required. This variability adds to the complexity of system design and operation, making the dynamic allocation of workloads between FPGAs and GPUs a daunting task.

[0005] Moreover, typical systems often struggle with managing workload distribution among various processing units, such as CPUs, FPGAs, and GPUs, and / or distributed processing resources (e.g., edge computing, cloud computing), leading to inefficiencies in data transfer and processing times. Such typical techniques for managing workload distribution frequently involve static or semi-static resource allocations, which may not adequately accommodate fluctuations in computational demands or the specific needs of different applications. This often results in inefficiencies, including underutilization of resources during periods of low demand and potential bottlenecks during peak times. Additionally, integrating artificial intelligence (Al) agents and features introduces further complexity, which typical techniques are unable to manage and thereby ensure that compute resources are optimally matched with the processing needs of such advanced functionalities.

[0006] Similar to the above problems related to processing unit level efficiency, various nodes and / or other such computing devices that integrate these processing units also have increased need to interact to perform an overall procedure and / or functionality. For example, in a hospital environment, a user may need a number of different specialized devices and / or various components to carry out an overall procedure on a patient.

[0007] However, the introduction of multiple devices and / or components into a system may introduce inefficiencies into the environment. For example, in a hospital environment, a device may need to either be activated and / or pre-configured before a portion of the procedure can be performed (e.g., slowing the overall procedure), or is kept on at all times in preparation for a procedure and / or component task thereof (e.g., wasting power and / or computing resources). Similarly, a device may needAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONparticular data before being able to perform a portion of the procedure, and may therefore wait until data has been input and analyzed. As such, it may be desirable to introduce systems and methods to improve the overall efficiency and operation scheduling of devices in such an environment.

[0008] In additional aspects, it may be beneficial to simulate a procedure within a virtual environment. For example, simulated environments provide safe environments where procedures can fail without causing real-world harm. However, there are challenges associated with being able to leverage these virtual environments. For example, virtual environments that act as a digital twin often model the physical environment. However, many medical procedures require detailed knowledge of patient internal anatomy that cannot be readily modeled using conventional digital twin techniques. Additionally, while virtual environments can simulate predetermined scenarios to generate synthetic data, without detailed knowledge of the procedures being simulated, the resulting synthetic training data is unfocused and results in lower predictive performance of models trained therewith.SUMMARY

[0009] The following presents a simplified summary of various examples described herein and is not intended to identify key or critical elements or to delineate the scope of the claims.

[0010] In some aspects, the techniques described herein relate to a compute node configured to optimize intra-processor data transfer and processing, the compute node including: one or more processors, including at least one Graphics Processing Unit (GPU) and at least one other programmable logic device; and one or more memories storing one or more applications and instructions thereon that, when executed by the one or more processors, cause the compute node to: receive a sequence of subframe chunks of a frame captured by a sensor device, each sub-frame chunk of the sequence of sub-frame chunks representing a portion of the frame, determine one or more tasks associated with at least one application of the one or more applications to which the sequence of sub-frame chunks correspond, cause the at least one GPU or the at least one other programmable logic device to analyze one or more sub-frame chunks of the sequence of sub-frame chunks based on the one or more tasks, iteratively compile the frame based on sequential analysis of the at least one GPU or the at least one other programmable logic device of the one or more sub-frame chunks, and determine an outcome of the one or more tasks during the iterative compiling of the frame, wherein the outcome is associated with operation of a computer-assisted system.

[0011] In some aspects, the techniques described herein relate to a system configured to optimize intra-processor data transfer and processing, the system including: one or more processors, including at least one Graphics Processing Unit (GPU) and at least one other programmable logic device; and one or more memories storing one or more applications and instructions thereon that, when executed by the one or more processors, cause the system to: receive a sequence of sub-frame chunks of a frame captured by a sensor device, each sub-frame chunk of the sequence of sub-frame chunks representing a portion of the frame, determine one or more tasks associated with at least one application of the one or more applications to which the sequence of subframe chunks correspond, cause the at least one GPU or the at least one other programmable logic device to analyze one or more sub-frame chunks of the sequence of sub-frame chunks based on the one or more tasks, iteratively compile the frame based on sequential analysis of the at least one GPU or the at least one other programmable logic device of the one or more sub-frame chunks, and determine an outcome of the one or more tasks during the iterative compiling of the frame, wherein the outcome is associated with operation of a computer-assisted system.

[0012] In some aspects, the techniques described herein relate to a method for optimizing intra-processor data transfer and processing, including: receiving, at one or more processors, a sequence of sub-frame chunks of a frame captured by a sensor device, each sub-frame chunk of the sequence of sub-frame chunks representing a portion of the frame; determining, by the oneAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONor more processors, one or more tasks associated with at least one application of one or more applications to which the sequence of sub-frame chunks correspond; causing, by the one or more processors, at least one GPU or at least one other programmable logic device to analyze one or more sub-frame chunks of the sequence of sub-frame chunks based on the one or more tasks; iteratively compiling, by the one or more processors, the frame based on sequential analysis of the at least one GPU or the at least one other programmable logic device of the one or more sub-frame chunks; and determining, by the one or more processors, an outcome of the one or more tasks during the iterative compiling of the frame, wherein the outcome is associated with operation of a computer-assisted system.

[0013] In some aspects, the techniques described herein relate to a system for optimizing compute resource management, the system including: a compute node including a central processing unit (CPU) and a graphics processing unit (GPU), the CPU and the GPU having a plurality of processing cores configured to execute computer-executable instructions; one or more applications accessible by the compute node from an application server, each application having associated tasks with different compute resource requirements; and a compute partitioning coordinator configured to: determine (i) a set of tasks for the one or more applications to be executed by the compute node, (ii) a task prioritization level for each task of the set of tasks, and (ill) compute resource requirements for each task of the set of tasks, partition, based on (i)-(iii), the set of tasks into a task schedule that specifies a respective processing core of the CPU or the GPU to perform execution of one or more of tasks of the set of tasks, and cause the GPU and the CPU to execute the partitioned tasks in accordance with the task schedule.

[0014] In some aspects, the techniques described herein relate to a method for optimizing compute resource management, including: determining, by one or more processors executing a compute partitioning coordinator, (i) a set of tasks for one or more applications to be executed by a compute node, (ii) a task prioritization level for each task of the set of tasks, and (ill) compute resource requirements for each task of the set of tasks; partition, by the one or more processors and based on (i)-(iii), the set of tasks into a task schedule that specifies a respective processing core of a CPU or a GPU of the compute node to perform execution of one or more of tasks of the set of tasks; and cause the GPU and the CPU to execute the partitioned tasks in accordance with the task schedule.

[0015] In some aspects, the techniques described herein relate to a cloud-based application server for hardware-aware application management, the cloud-based application server including: one or more processors; and one or more memories storing one or more applications and a processing environment analysis module that, when executed by the one or more processors, cause the cloud-based application server to: connect to a plurality of compute nodes requesting access to the one or more applications, wherein a first compute node and a second compute node request access to a first application of the one or more applications, determine a respective processing environment configuration for the first compute node and the second compute node indicating (i) a first compute processing capacity of the first compute node, (ii) a second compute processing capacity of the second compute node, (iii) a first compute processing hardware configuration of the first compute node, and (iv) a second compute processing hardware configuration of the second compute node, and compare the respective processing environment configurations of the first compute node and the second compute node to application processing configuration requirements for the first application, and perform at least one of: provision access to the first application for the first compute node based on the first compute processing capacity and the first compute processing hardware configuration satisfying the processing configuration requirements for the first application, and deny access to the first application for the second compute node based on the second compute processing capacity or the second compute processing hardware configuration failing to satisfy the processing configuration requirements for the first application.Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION

[0016] In some aspects, the techniques described herein relate to a method for hardware-aware application management, including: connecting, by one or more processors, to a plurality of compute nodes requesting access to the one or more applications, wherein a first compute node and a second compute node request access to a first application of the one or more applications; determining, by the one or more processors, a respective processing environment configuration for the first compute node and the second compute node indicating (i) a first compute processing capacity of the first compute node, (ii) a second compute processing capacity of the second compute node, (iii) a first compute processing hardware configuration of the first compute node, and (iv) a second compute processing hardware configuration of the second compute node; and comparing, by the one or more processors, the respective processing environment configurations of the first compute node and the second compute node to application processing configuration requirements for the first application, and perform at least one of: provisioning access to the first application for the first compute node based on the first compute processing capacity and the first compute processing hardware configuration satisfying the processing configuration requirements for the first application, and denying access to the first application for the second compute node based on the second compute processing capacity or the second compute processing hardware configuration failing to satisfy the processing configuration requirements for the first application.

[0017] In some aspects, the techniques described herein relate to a method for orchestrating efficient usage of a plurality of local computing nodes, the method including: receiving, by one or more processors of a computing device, procedure data from one or more local computing nodes of a plurality of computing nodes, the one or more local computing nodes used to support a procedure; analyzing, by the one or more processors, the procedure data via a trained machine learning model to generate procedure analysis data; predicting, by the one or more processors and based on the procedure analysis data, a likelihood of at least one node being a future node to be used as part of the procedure; and configuring, by the one or more processors, the at least one node based on the predicted likelihood.

[0018] In some aspects, the techniques described herein relate to a method for orchestrating efficient usage of a plurality of local surgical nodes, the method including: receiving, by one or more processors of a computing device, surgical procedure data from one or more local surgical nodes of a plurality of computing nodes, the one or more local surgical nodes used as part of a surgical procedure; analyzing, by the one or more processors, the surgical procedure data via a trained machine learning model to generate procedure analysis data; predicting, by the one or more processors and based on the procedure analysis data, one or more destination surgical nodes of the plurality of computing nodes; and transmitting, by the one or more processors, the surgical procedure data to the one or more predicted destination surgical nodes for processing.

[0019] In some aspects, the techniques described herein relate to a method for presenting a digital twin of a subject, the method including: obtaining, by one or more processors of a computing device, preoperative image data of a subject; based on the preoperative image data, generating, by the one or more processors, a three-dimensional (3D) model of the subject; importing, by the one or more processors, the 3D model into a virtual environment; and presenting, by the one or more processors, the virtual environment to a viewer device.

[0020] In some aspects, the techniques described herein relate to a method for training a classifier to predict surgical procedure outcomes, the method including: obtaining, by one or more processors of a computing device, a set of historical procedure data indicative of a plurality of historical procedures; based on the set of historical procedure data, determining, by the one or more processors, a phase of the procedure associated with a threshold correlation to an outcome of the procedure; generating, by the one or more processors, (i) a plurality of scripts for performing the phase of the procedure, and (ii) a plurality of three-dimensional (3D) models of a subject on which the procedure is to be performed in a simulated environment; executing, by the one or more processors, a plurality of simulations within the simulated environment, wherein the plurality of simulationsAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONimplement respective combinations of one or more scripts of the plurality of scripts and one or more 3D models of the plurality of 3D models; extracting, by the one or more processors, simulated data for each executed simulation, wherein the simulated data includes procedure outcome data and simulated image data; and based on the extracted simulated data, training, by the one or more processors, a classifier model to predict procedure outcome based on intraoperative image data.

[0021] In some aspects, the techniques described herein relate to a method for predicting surgical procedure outcomes for a procedure, the method comprising: obtaining, by one or more processors of a computing device, a set of intraoperative multimodal data; inputting, by the one or more processors, the set of intraoperative multimodal data into a classifier model trained via the method of the above aspect; detecting, by the one or more processors, an output from the classifier model indicative that a potential negative procedure outcome is to occur; and generating, by the one or more processors, an alert indicative of the potential negative procedure outcome.

[0022] In some aspects, the techniques described herein relate to a method for training a control policy for autonomous control of robotic equipment, the method comprising: obtaining, by one or more processors of a computing device, model data for robotic equipment controlled by a robotic controller; importing, by the one or more processors, the model data into a virtual environment via which a virtual model of the robotic equipment is configured to operate autonomously in accordance with a control policy; executing, by the one or more processors, an unsupervised learning process within the virtual environment to train the control policy, wherein the unsupervised learning process implements a reward policy that penalizes harm to a subject and rewards successful procedure outcomes; and exporting, by the one or more processors, the trained control policy for implementation in the robotic equipment.

[0023] In some aspects, the techniques described herein relate to a method for autonomous control of robotic equipment, the method comprising: obtaining, by one or more processors of a computing device, an indication that the robotic equipment is to be utilized to perform a procedure; obtaining, by the one or more processors, a control policy for autonomous control of the robotic equipment trained via the method of the previous aspects; and executing, by the one or more processors, the control policy to autonomously control the robotic equipment during performance of the procedure.

[0024] In additional aspects, systems comprising a non-transitory memory storing computer-executable instructions and a processor that executes the instructions may implement any of the described methods. Similarly, in additional aspects, computer-readable storage media may store storing computer-executable instructions that, when executed by one or more processors, implement any of the described methods. Moreover, while the above sets forth various aspects, it should be appreciated that any combination of the above aspects may be implemented in a single embodiment.

[0025] It is to be understood that both the foregoing general description and the following detailed description are illustrative and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In that regard, additional aspects, features, and advantages of the present disclosure will be apparent to one skilled in the art from the following detailed description.BRIEF DESCRIPTION OF THE DRAWINGS

[0026] FIG. 1A depicts an example computing environment in which local computing system is coupled with a cloud system to improve medical procedures.

[0027] FIG. 1B1 depicts an example computing environment in which several local computing are is coupled with the cloud system to improve medical procedures.Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION

[0028] FIG. 1B2 depicts an example computing environment in which a local cloud is implemented to comply with local privacy regimes.

[0029] FIGS. 10 and 1D are example block diagrams of a data connector at which the various techniques disclosed herein may be implemented.

[0030] FIG. 2A depicts an example robotic controller architecture, in accordance with various embodiments described herein.

[0031] FIG. 2B depicts an example robotic controller architecture utilizing sub-frame chunking of data to improve processing efficiency, in accordance with various embodiments described herein.

[0032] FIG. 2C depicts an example compute resource partitioning scheme that is managed by a compute partitioning coordinator, in accordance with various embodiments described herein.

[0033] FIG. 2D depicts an example distributed computing environment in which applications are made conditionally available from the cloud computing environment in accordance with the compute resources available at the local computing environments, in accordance with various embodiments described herein.

[0034] FIG. 2E depicts a flow diagram representing an example computer-implemented method for optimizing intraprocessor data transfer and processing, in accordance with various embodiments described herein.

[0035] FIG. 2F depicts a flow diagram representing an example computer-implemented method for optimizing compute resource management, in accordance with various embodiments described herein.

[0036] FIG. 2G depicts a flow diagram representing an example computer-implemented method for hardware-aware application management, in accordance with various embodiments described herein.

[0037] FIG. 3A depicts an example configuration of a processing system for retrieving information from various nodes, determining which nodes are to be used in a procedure, and configuring the node(s) accordingly.

[0038] FIG. 3B depicts an example configuration of a processing system similar to that of FIG. 3A, but in which the determination is made at a local orchestration node rather than a cloud node.

[0039] FIG. 3C depicts an example configuration of a processing system similar to that of FIG. 3A, but in which the determination is made at a local orchestration node and a cloud node functioning in conjunction with one another.

[0040] FIG. 3D depicts an example configuration of a node with an orchestration component configured to determine with which nodes to communicate, to be implemented in a system similar to that of FIGS. 3A-3C.

[0041] FIG. 3E depicts a flow diagram illustrating a method for determining a future node to be used in a procedure and, based on the determination, configuring the future node, to be implemented in the system of FIGS. 1A-1D.

[0042] FIG. 3F depicts a flow diagram illustrating a method for determining a future node to be used in a procedure and transmitting data for further use in the procedure based on the determination, to be implemented in the system of FIGS. 1 A-1 D.

[0043] FIG. 4A depicts an example computing environment in which methods and systems for presenting a digital twin and / or generating synthetic data may be implemented, according to embodiments.

[0044] FIG. 4B depicts a flowchart of an example method for presenting a digital twin, according to embodiments.

[0045] FIG. 4G depicts a block diagram for training an outcome prediction classifier, according to embodiments.Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION

[0046] FIG. 4D depicts a block diagram of an example process for training a robotic control policy via a simulated environment, according to embodiments.

[0047] FIG. 4E illustrates a flow chart of an example method for training a classifier to predict surgical procedure outcomes, according to embodiments.

[0048] FIG. 4F illustrates a flow chart of an example method for predicting surgical procedure outcomes, according to embodiments.

[0049] FIG. 4G illustrates a flow chart of an example method for training a control policy for autonomous control of robotic equipment, according to embodiments.

[0050] FIG. 4H illustrates a flow chart of an example method for autonomous control of robotic equipment, according to embodiments.DETAILED DESCRIPTION

[0051] In the following description, specific details are set forth describing some examples consistent with the present disclosure. Numerous specific details are set forth in order to provide a thorough understanding of the examples. It will be apparent, however, to one skilled in the art that some examples may be practiced without some or all of these specific details. The specific examples disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in association with one example may be incorporated into other examples unless specifically described otherwise or if the one or more features would make an example non-functional. In some instances, well known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the examples.

[0052] It should be appreciated that while the following sets out section headers, these are provided to improve readability, not to delineate between distinct embodiments. To this end, in some embodiments, the techniques described in different sections may be implemented in the same embodiments. Instead, each section may be viewed as describing a separate “tech stack” that can be implemented in any combination with the other tech stacks described herein.

[0053] Healthcare providers (HCPs) generate a significant amount of data. For example, HCPs interact with patient medical data, electronic medical records (EMR), inventory data, scheduling data (for patients, personnel, and equipment), and various other types of data generated by the suite of applications that support healthcare operations. Additionally, there are many different types of computing systems located at a HCP. For example, there may be planning workstations that doctors use to access pre-operative planning utilities (e.g., patient notes, reviewing medical records, determining courses of treatment, etc.), intraoperative workstations (e.g., workstations a surgeon utilizes intraoperatively in furtherance of a procedure, such as workstations coupled to a user input device for robotic equipment, workstations that provide guidance during a procedure), robotic equipment controllers, on-premises data servers, etc. Moreover, the data may be stored at a variety of different locations at different times (e.g., at a local client device such as a workstation accessing the data, at an on-premises data server, one or more cloud servers, etc.).

[0054] As different types of data have different priorities as it relates to performance of medical procedures, efficiencies may be gained by ensuring the right computing system processes the right data at the right time. Accordingly, to ensure efficient data processing, techniques described herein relate to routing data to a “data connector” that processes and routes the data to the appropriate computing system based on the current processing requirements. In some scenarios, this may include couplingAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONcomputing systems utilized intraoperatively to auxiliary processing devices to ensure sufficient compute resources for timely processing of the various data streams. In some aspects, the use of data connector supports further improvements, such as improved ways of processing data at an individual computing system, providing configuration-specific applications, and / or enabling local processing for tasks typically performed at a cloud (e.g., training or fine-tuning machine learning models).

[0055] FIGS. 1A-1D depict an example environment 100 in which a data connector 20 is implemented to support the various efficiencies described herein. As illustrated, a hospital 10 (or other type of HCP) includes a data connector 20 located thereat. As will be described in more detail below, depending on the particular implementation, the data connector 20 may be implemented a several different computing devices located at the hospital 10. For example, the data connector 20 may be implemented at an onpremises server, at a local workstation, at an operating room workstation, etc. While various examples provided herein relate to “surgical” medical procedures, it should be appreciated that the disclosed techniques may be applied to other types of medical procedures (e.g., diagnostics procedures, rehabilitation procedures, etc.). Additionally, while the following sets forth examples that relate to a “hospital,” it should be understood that the term “hospital” envisions individual HCPs that operate in a hospital setting.

[0056] In some embodiments, the data connector 20 may be a network of computing devices distributed about the hospital 10. For example, the data connector 20 may include a centrally-located computing system that is communicatively coupled to a plurality of computing systems located in respective operating rooms (such as local orchestration nodes or local distribution nodes). In this arrangement, the local computing systems implement perform efficient routing and / or processing of high-priority intraoperative data during a procedure, while synchronizing lower-priority data back to the central data connector 20 for other operations (e.g., obtaining additional data needed to assist the procedure, uploading data to the cloud, etc.). This arrangement may enable lower-latency in the efficient processing of intraoperative data than a purely centralized data connector implementation.

[0057] As illustrated, the data connector 20 is communicatively coupled to a plurality of computing systems at the hospital 10. For example, the data connector 20 may be coupled to a medical data database 22 (e.g., a database that stores medical imaging and / or pre-operative data associated with a patient), an electronic medical record database 24 (e.g., a database that stores EMRs), an inventory database 26 (e.g., a database indicating the availability of medical tools, equipment, devices, etc.) and / or a scheduler 28 (e.g., a scheduling system that schedule personnel and / or equipment for procedures). Additionally, the data connector 20 may be communicatively to robotic equipment 12 (and / or a control system thereof) located at the hospital 10. For example, the robotic equipment may include a single-port system (e.g., a cart mounted robotic arm), a multi-port system (e.g., the da Vinci® Surgical System commercialized by Intuitive Surgical, Inc. of Sunnyvale, California), an endoluminal surgical system (e.g., the Ion® Surgical System commercialized by Intuitive Surgical, Inc. of Sunnyvale, California), etc. The data connector 20 may obtain data from these data sources to, for example, predictively configure operating room equipment in preparation for a procedure (such as the robotic equipment 12), prioritize and / or route different data streams, efficiently perform I / O operations with a data storage system (such as the data lake 60), etc.

[0058] As illustrated, the data connector 20 is coupled to two types of cloud systems - the HIPAA compliant cloud 50 and the “other” cloud 80 that is not HIPAA compliant. It should be appreciated that while “HIPAA” refers to data protection requirements specific to the United States, any reference to the term HIPAA envisions alternative data protection regimes associated with the jurisdiction at which the hospital 10 is located. Generally, the data connector 20 may interface with the cloud 50 for storing protected data and / or executing applications that interact with the protected data, and the cloud 80 for storing and / or interfacing with data not subject to the privacy restrictions. Accordingly, the data connector 20 may include a set of rules that direct the data connector 20 to enforce the privacy regime by routing data to the appropriate cloud 50, 80.Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION

[0059] In addition to jurisdictional regulatory requirements, the cloud 50 may also support customer-specific data protection requirements and / or preferences. For example, different HCPs at the hospital 10 may have different preferences for how their data is utilized (e.g., use for training Al models, which parties have access, etc.). Accordingly, the cloud 50 may be configured to establish different instances, workspaces, or other types of partitions to enforce the privacy policies for each HCP associated with the hospital 10.

[0060] As illustrated, the cloud 50 may include a data lake 60 (and / or other suitable cloud storage architecture) at which a plurality of protected data is maintained. For example, the data lake 60 may maintain sets of historical procedure data 62 associated with procedures performed using the robotic equipment 12. The historical procedure data may include any of the various types of multimodal data described herein, including kinematic data, event data, force data, intraoperative image data, procedure state data, etc. Based on the historical procedure data 62, the cloud 50 (and / or other suitable computing device) may train and store a plurality of Al models 64 for usage during a procedure. For example, the Al models may include scene recognition models, classifier models, language models, robotic control policies, and / or other types of Al models, including those described elsewhere herein. During operation, when the data connector 20 is pre-configuring robotic equipment 12 to perform a procedure, the data connector 20 may obtain the model data for the appropriate Al models 64 to be utilized in the procedure. For example, the data connector 20 may obtain control policies associated with the schedule type of robotic equipment, an outcome classifier associated with the scheduled type of procedure, and / or other types of Al models that facilitate autonomous operation of the robotic equipment and / or processing of intraoperative multimodal data.

[0061] Additionally, the customer (e.g., an operator of the hospital 10) may include their own customer applications 66 that interact with protected medical data. For example, the customer may include software services provided by other parties, such as systems related to processing EMRs, proprietary diagnostic tools, etc. Accordingly, the data connector 20 may include an extensible interface (e.g., an application programming interface) via which the customer applications 66 can interface with the data connector 20 such that the improve data processing techniques disclosed herein can be adapted for the customer applications 66.

[0062] Similarly, the provider of the data connector 20 may also host provider applications 68 at the cloud 50. For example, the provider applications 68 may include a 3D model generator that converts sets of 3D pre-operative image data (e.g., tomosynthesis data, cone beam computed tomography (CBCT) data, etc.) into 3D models thereof. These 3D models may be stored at the data lake 60 and / or provided to the data connector 20 for integration into a 3D modeling system. For example, the 3D modeling system may be utilized for coordinate registration during a procedure and / or to generate a digital twin of the subject anatomy within a virtual environment.

[0063] In some embodiments, the provider applications 68 also include an extensible interface for integration with the customer applications 66. For example, a customer application 66 may relate to perform customer specific analyses of medical data (e.g., calculating objective performance indicators (OPIs), customer-specific machine learning models, customer-specific analysis rule sets, etc.). In these embodiments, the customer applications 66 may invoke the provider applications 68 to facilitate, for example, privacy-protected access to their data, automatic model fine tuning, correction of out-of-fold (OOF) performance for models, and / or other model maintenance and / or update applications. In this way, the provider applications 68 are able to ensure that customer defined Al models associated with the customer applications 66 are maintained in a manner that ensures the models accurately analyze over time.

[0064] Although not depicted, in some embodiments, the hospital 10 may be associated with one or more remote workstations communicatively coupled to the data connector 20 and / or the robotic equipment 12. For example, the remote workstations may be a telesurgical workstation for generating control commands to operate the robotic equipment 12. As anotherAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONexample, the remote workstation may be a mentoring and / or guidance workstation for providing guidance to users locally interacting with the robotic equipment 12 at the hospital 10. In these embodiments, the remote workstation may also be communicatively coupled to the clouds 50, 80 to directly obtain the appropriate medical data 22, EMR 24, and / or Al models 64 from the clouds 50, 80 without using the data connector 20 as an interface. In these embodiments, the remote workstation may also implement a version of the data connector 20 local to the remote work site for implementing the various efficient task execution techniques described herein.

[0065] As illustrated in FIG. 1B1, in some embodiments, the cloud 50 (and / or the cloud 80) maybe communicatively coupled with a plurality of hospitals 10 (e.g., the hospitals 10 are part of a network of hospitals). In this arrangement, each hospital may have a local data connector 20 configured to implement the various techniques disclosed herein. This arrangement where the cloud 50 is coupled to a network of data connectors 20 expands the amount of historical procedure data 62 that can be obtained, resulting in the ability to train Al models 64 that exhibit greater predictive performance than otherwise possible.

[0066] That being said, in some networks, the hospital 10A (and / or any of the HCPs associated therewith) may refrain from sharing their protected data with other hospitals 10B, 10C (and / or the other HCPs associated with the hospital 10A) and maintain their own instance of the cloud 50 dedicated to maintaining, servicing, and / or processing their own data. For example, some hospitals 10 may want to prevent their data from being used for training the Al models 64 that may be implemented at other hospitals. Alternatively, the cloud 50 may still be coupled to all of the hospitals 10A-10C, but the cloud 50 may partition off the portion of the data lake 60 that maintains the historical procedure data 62 associated with the hospital 10A from the other historical procedure data such that customer application 66 and / or provider applications 68 executed at the hospitals 10B, 10C do not have access to the historical procedure data 62. As a result, the individual hospitals 10 are in control of how their data is used and who has access to Al models 64 trained on their data.

[0067] In another arrangement illustrated in FIG. 1B2, customer (e.g., a network of hospitals) has locations in different jurisdictions. For example, the hospital 10A may be located in the European Union, the hospital 10B may be located in the United States, and he hospital 10C may be located in China. Due to local privacy regimes, data generated at each hospital 10 may not be able to be exported to the other hospitals. In this scenario, the cloud 50 may be a local cloud 50A configured to host, process, and / or serve data within the local jurisdiction. As a result, when the customer applications 66 and / or the provider applications 68 execute on the hospital data, the data is not exported to a jurisdiction with differing privacy regulations. It should be appreciated that, as a result, machine learning models trained at the local cloud 50A are trained based on data that complies with local privacy regulations.

[0068] As illustrated, the customer 50 may still include a global server 55 to synchronize processing algorithms between different jurisdictions. For example, the global server 55 may ensure that the same machine learning architectures, models, and / or training algorithms are implemented at each local cloud 50 to ensure a unified approach to processing the locally maintained data. As will be described below, in scenarios where the localized approach results in insufficient data to train an accurate model, the local data sets may be supplemented using synthetically-generated data sets. In these embodiments, the synthetic data may be generated in a manner the reflects to characteristics of the data locally maintained at the corresponding local server 50.

[0069] FIG. 1C depicts a block diagram of the data connector 20 being operatively coupled to a plurality of different computing entities that may be located in an operating room 11 at the hospital 10. It should be appreciated that the data connector 20 may also be coupled to computing devices associated with additional operating rooms 1 T in a similar manner as described with respect to the operating room 11. As discussed above, the data connector 20 may be coupled to robotic control systems 13 for controlling the robotic equipment 12. As described herein, the robotic control systems 13 may be configured to implementAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONautonomous control of the robotic equipment based on multimodal intraoperative data, for example, by implementing the techniques described in U.S. Patent Application serial no. 63 / 671,983 entitled “SYSTEM AND METHODS FOR ENFORCING SAFETY IN INTELLIGENT SURGICAL ROBOTS,” and filed on July 16, 2024, the contents of which are incorporated herein in its entirety by reference. Accordingly, after the completion of a procedure, the robotic controller 13 may process and compile the intraoperative multimodal data for transmission to the data connector 20 (and storage at the cloud 50).

[0070] In some embodiments, the robotic controller 13 may apply indexable fields to the set of multimodal data such that the cloud 50 and / or the applications 66, 68 hosted thereat are able to filter the pool of historical procedure data. For example, one filed may include an indication of robotic equipment type and / or version. Accordingly, the applications 66, 68 may be able to determine whether the procedure was performed using the latest robotic equipment 12 or legacy robotic equipment 12. Because data obtained from legacy robotic equipment 12 may be less predictive for newer versions of robotic equipment, techniques described herein may apply a lower weight, scale, or other appropriate technique for lessening how the training algorithm evaluates the historical procedure data collected via legacy systems. Additionally or alternatively, because the robotic equipment type may be an indexable field, the techniques disclosed herein may be applied to train models that are specific to any particular robotic equipment version. In these embodiments, the synthetic training data techniques described elsewhere herein may be applied to increase the volume of training data available for training the version-specific Al models 64.

[0071] In addition to obtaining procedural data from the robotic controllers 13, the data connector 20 may transmit data to the robotic controllers 13 to pre-configure the robotic controllers 13 for performing a particular procedure. For example, the data connector 20 may transmit pre-operative data associated with the procedure, Al models (including control policies) for performing the procedure, and / or other data relevant to performance of the procedure.

[0072] As another example, the data connector 20 may be in communication with a remote viewing system 25 configured to host a simulated virtual environment that virtually replicates the operating room 11. For example, the data connector 20 may configure the remote viewing system 25 with 3D models representative of the robotic equipment 12, the Al models used to control the robotic equipment 12, 3D models of the patient, and so on. In some embodiments, the data connector 20 may also function as a router between the remote viewing system 25 and remote devices viewing the simulated environment (e.g., for performing teleoperation of the robotic equipment, providing tutoring or guidance, or just general observation of the procedure).

[0073] Additionally, the data connector 20 may be communicatively coupled to other nodes located in the operating room 11, such as a local orchestration node 30, a local distribution node 15, and / or a surgical workstation 16. For example, these nodes may analyze data generated locally in the operating room 11 (e.g., intraoperative multimodal data) and coordinate with the data connector 20 if additional actions need to be performed by other systems associated with the hospital 10. For example, the data connector 20 may receive requests for additional personnel and / or equipment to be brought to the operating room 11, to process particular sets of data using the additional processing power of a computing system external to the operating room 11 (e.g., at a local or cloud-based server, such as the clouds 50, 80).

[0074] As another example, the data connector 20 may determine a configuration of the robotic control systems 13 for determining a set of applications (e.g., packages of analysis tools) that may be efficiently executed via the computing devices included in the operating room 11. For example, certain applications 66, 68 may have minimum processing requirements for efficient execution. Accordingly, by communicating obtaining the configuration information, the data connector 20 is able to obtain the appropriate version of the application for setup implemented at the operating room 11. In some embodiments, this may include obtaining a different version of an application that has certain advanced features disabled.Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION

[0075] As described herein, in some embodiments, the computing devices located in the operating room 11 may include a port via which the computing device can be operationally coupled to an auxiliary processing system 17 that has a processing unit architecture specifically adapted to perform a set of analyses on intraoperative data. In these embodiments, the auxiliary processing system 17 may be configured to provide the outputs to the computing device to which it is coupled and / or directly to the data connector 20.

[0076] FIG. 1 D is a block diagram of a data connector 20. As described above, the data connector 20 may be implemented in several different form factors and / or implemented at other computing structures described herein (e.g., at a surgical workstation, a local orchestration or distribution node, at a cloud, etc.). Regardless of the particular integration of the data connector 20 into the overall system, the data connector may include structures described with respect ot FIG. 1 D.

[0077] As illustrated, the data connector 20 includes a processing unit 32. The processing unit 32 may include one or more processors having different processing architectures for processing instructions. For example, the one or more processors may be one or more cores or micro-cores of a multi-core processor, a central processing unit (CPU), a microprocessor, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a graphics processing unit (GPU), a tensor processing unit (TPU), and / or the like.

[0078] The data connector 20 may also include network interfaces 34 to support one or more communication interfaces (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.). For example, a network interface of the data connector 20 may include an integrated circuit for connecting the data connector 20 to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, mobile network, or any other type of network) and / or to another device, such as the computing devices located in the operating 11 and / or the clouds 50, 80.

[0079] The data connector 20 also may include a memory 116. The memory 116 may include non-persistent storage (e.g., volatile memory, such as random access memory (RAM), cache memory), persistent storage (e.g., a hard disk, an optical drive such as a compact disk (CD) drive or digital versatile disk (DVD) drive, a flash memory, a floppy disk, a flexible disk, a magnetic tape, any other magnetic medium, any other optical medium, programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a FLASH-EPROM, and / or any other memory chip or cartridge. The non-persistent storage and persistent storage are examples of non-transitory, tangible machine-readable media that can store executable code that, when run by one or more processors (e.g., the processing unit 32), can cause the one or more processors to perform one or more of the techniques and / or methods disclosed herein.

[0080] As illustrated, the data connector 20 may also store and / or execute a set of applications 40. The set of applications 40 may include applications dedicated to performing the various analyses described herein. One type of application 40 is an Al agent 41. An Al agent 41 may be an application hosted in the runtime environment to process new data as it obtained at the data connector 20. For example, another application 40 may include a data collection application configured to obtain intraoperative multimodal data from the robotic controller 13. In this example, as the data collection application obtains new data, the data collection application may transmit an event over an event bus describing the received data. The Al agent 41 may detect this event and determine if one or more Al models (such as the Al models 64) are adapted to execute on the received data, and, if so, request the data for executing the corresponding Al model. In some embodiments, the Al agent 41 may only execute in response to detecting a stimulus (e.g., a user input indicating that a particular analysis is to be performed). It should be appreciated that other examples of how runtime agents may process streams of data to execute Al algorithms on the data are envisioned.Processing InfrastructureAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION

[0081] Turning to how more particularly the various computing systems (e.g., “nodes”) described above may be configured, many computing systems leverage specialized processing units, such as Graphics Processing Units (GPUs) and Field-Programmable Gate Arrays (FPGAs), to handle complex processing tasks efficiently. GPUs, known for their high throughput, have been instrumental in performing complex computations, especially in areas requiring intense parallel processing capabilities such as camera processing and depth mapping. FPGAs offer a flexible architecture, making them suitable for simpler, static processing tasks due to their reconfigurability and efficiency in executing well-defined operations.

[0082] However, dynamically allocating and optimizing tasks between and within such specialized processing units is a challenge for typical systems employing multiple units and / or multiple types of processing units. For example, many existing systems face difficulties in managing workloads between GPUs and FPGAs, different GPUs, and / or different FPGAs, leading to suboptimal processing efficiency. Additionally, the process of data transfer between these units without incurring significant latency poses another layer of complexity that negatively impacts the overall system performance.

[0083] As one example, in the context of robotic-guided medical procedures, an endoscope may acquire image data of a surgical region of interest, and that image data may need to be processed with optimal efficiency to effectively perform guidance and / or other operations as part of the medical procedure. However, efficiently transferring image data between an FPGA and GPU while maintaining synchronization across processing stages is complex. High-resolution image data generates significant bandwidth demands, and any latency or synchronization issues can impact real-time processing performance and the quality of the output. Further, the workload in endoscopic imaging can vary significantly depending on the procedure, the resolution and frame rate of the captured images, and the specific processing tasks required. Dynamically allocating workloads between the FPGA and GPU to optimize processing efficiency and response times, while adapting to these variations, can add significant complexity to system design and operation.

[0084] Moreover, the increasing complexity of algorithms, along with the demand for real-time processing in fields such as medical applications, robotics, and feedback-driven environments, has necessitated more efficient parallel processing and job scheduling on processing units. Many typical systems often struggle with efficiently managing the workload distribution among, e.g., CPUs, FPGAs, and GPUs, leading to potential inefficiencies in data transfer and processing times, especially when the systems do not include / utilize kernel prioritization which can result in suboptimal utilization of available computing resources.

[0085] Virtualization techniques have introduced possibilities for creating isolated environments or “sandboxes” to minimize negative interactions among concurrently running tasks. However, typical approaches may not fully leverage these techniques to ensure the safety and reliability required in certain applications, such as those in the medical field. Existing systems also face challenges in dynamically scaling computing resources based on the complexity of tasks and the computational demands of algorithms, potentially leading to underutilization of larger, more powerful GPUs or overburdening less capable systems.

[0086] Existing techniques for enhancing processing capabilities typically involve static or semi-static resource allocation, which may not adequately accommodate fluctuations in computational demands or the specific needs of different applications. This generally results in inefficiencies, including underutilization of resources during periods of low demand and potential bottlenecks during peak times. Integrating artificial intelligence (Al) agents and features introduces further complexity, requiring sophisticated mechanisms to ensure that compute resources are optimally matched with the needs of these advanced functionalities. The challenge is further exacerbated when considering scalability challenges and the integration of external hardware while maintaining strict separation of functionalities to avoid negative interactions and ensure compliance with relevant regulations.Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION

[0087] By contrast, the present techniques overcome these, and other, challenges by (1) determining tasks associated with sub-frame chunks of sensor data to cause a processing device to analyze the chunks based on these tasks and iteratively compiling the frame based on sequential analysis of the processing units to determine an outcome of the tasks during the iterative compiling, (2) partitioning tasks to be executed by one or more processing units into a task schedule that assigns the tasks to specific cores of the processing units, and / or (3) selectively provisioning access to available applications for compute nodes that satisfy processing requirements of the applications and / or dynamically allocating additional compute resources to facilitate the compute nodes receiving access to such applications.

[0088] Moreover, the present techniques utilize specific contextual details when determining how to allocate available processing resources to perform tasks / sub-tasks, and thereby improve upon conventional techniques. Typical systems often utilize a static / quasi-static processing resource allocation that does not / cannot account for the changing priorities of a dynamic environment (e.g., a medical environment). Accordingly, these typical systems consistently allocate similar / identical processing resources to each processing task, regardless of environmental changes, and thereby do not provide prioritized processing for tasks that may frequently require / benefit from increased processing resources. Similarly, these typical techniques commonly overcommit processing resources to processing tasks that may not require prioritized processing, thereby wasting energy and processing cycles.

[0089] The present techniques overcome at least these challenges by leveraging moment-to-moment phase predictions and / or other phase indications to determine / recognize full-length phases or tasks / sub-tasks (referenced herein as “medical tasks”) associated with one or more medical procedures (e.g., surgical, diagnostic). These phases may refer to high-level, universal activities that can occur in different types of procedures, and can include, for example: exposure (e.g., visualizing and accessing a surgical site), dissection (e.g., cutting, separating and removing tissues or anatomical structures to gain access to specific areas), transection (e.g., severing or cutting a structure using a surgical instrument), extraction (e.g., removal of a tissue, organ, foreign object, or other anatomical structure from the body), and / or reconstruction (e.g., restoring or rebuilding a damaged or missing tissue, organ, or body part). The medical tasks may refer to specific groups of actions performed during medical procedures, such that each phase may be or include one or more medical tasks.

[0090] For example, a single surgery may include the performance of several medical tasks. Locating a tumor may constitute a first medical task, excising the tumor a second medical task, and closing the surgery site a third medical task. Each medical task may include multiple actions, e.g., a tumor excision medical task may require several cutting actions and several cauterization actions. While some surgeries require that medical tasks assume a specific order (e.g., excision occurs before closure), the order and presence of some medical tasks in some surgeries may be allowed to vary (e.g., the elimination of a precautionary medical task or a reordering of excision medical tasks where the order has no effect).

[0091] The techniques of the present disclosure may generate and / or otherwise receive data indicating one or more phase changes and / or medical task changes associated with a medical procedure and may utilize this data to adjust task / sub-task allocation to one or more processing units (e.g., GPUs, FPGAs, CPUs). In this manner, the present techniques can ensure that the highest priority tasks / sub-tasks for particular phases / medical tasks of a medical procedure are performed quickly to provide relevant personnel the information they may require most urgently. For example, during a transection phase of a surgical procedure, the present techniques may allocate substantial GPU processing resources to perform high-demand image analysis algorithms to ensure surgical personnel are able to clearly view the structure(s) to be cut. This improves upon conventional techniques that ignore and / or otherwise do not utilize such medical context (or other similar context) to inform the allocation ofAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONprocessing resources by dynamically allocating processing resources in response to changes in a dynamic environment (e.g., medical environment).

[0092] Further, the present disclosure includes specific features other than what is well-understood, routine, conventional activity in the field, or adding unconventional steps that demonstrate, in various embodiments, particular useful applications, e.g., (1) determining one or more tasks associated with at least one application of the one or more applications to which the sequence of sub-frame chunks correspond, causing the at least one GPU or the at least one other programmable logic device to analyze one or more sub-frame chunks of the sequence of sub-frames chunks based on the one or more tasks, iteratively compiling the frame based on sequential analysis of the at least one GPU or the at least one other programmable logic device of the one or more subframe chunks, and / or determining an outcome of the one or more tasks during the iterative compiling of the frame; (2) determining (i) a set of tasks for the one or more applications to be executed by the compute node, (ii) a task prioritization level for each task of the set of tasks, and (ill) compute resource requirements for each task of the set of tasks, and / or partitioning, based on (i)-(iii), the set of tasks into a task schedule that specifies a respective processing core of the CPU or the GPU to perform execution of one or more of tasks of the set of tasks; and / or (3) determining a respective processing environment configuration for the first compute node and the second compute node indicating (i) a first compute processing capacity of the first compute node, (ii) a second compute processing capacity of the second compute node, (iii) a first compute processing hardware configuration of the first compute node, and (iv) a second compute processing hardware configuration of the second compute node, comparing the respective processing environment configurations of the first compute node and the second compute node to application processing configuration requirements for the first application, and perform at least one of: provisioning access to the at least one application for the first compute node based on the first compute processing capacity and the first compute processing hardware configuration satisfying the processing configuration requirements for the first application, and / or denying access to the at least one application for the second compute node based on the second compute processing capacity or the second compute processing hardware configuration failing to satisfy the processing configuration requirements for the first application, among others.

[0093] Of course, it should be appreciated that the advantages and technical improvements described above and elsewhere herein are not the only advantages and / or technical improvements that may be realized as a result of the techniques described herein.

[0094] FIG. 2A depicts an example robotic controller architecture A200, in accordance with various embodiments described herein. The example robotic controller architecture A200 generally includes multiple components / devices that are configured to capture image data (e.g., as part of a surgical / medical procedure) and analyze the image data to perform one or more actions. For example, the actions performed by the example robotic controller architecture A200 may include generating and transmitting control instructions that cause one or more actuating components to maneuver or otherwise adjust the positioning of the endoscope A202 camera, adjust the intensity / focus of the lighting provided by the endoscope A202, controlling one or more surgical instruments, and / or any other suitable actions or combinations thereof.

[0095] The example robotic controller architecture A200 includes an endoscope A202 that captures image data and transfers raw image frames A203 including the image data to an endoscope system controller (ESC) A204. The ESC A204 may process (e.g. pre-process) the image data to ensure that each frame and / or sub-frame chunks thereof have appropriate image quality for any further processing tasks associated with the captured image data. The ESC A204 may be communicatively coupled with one or more downstream nodes A205 which may receive data from the ESC A204 via a fiber A206 and / or any other suitable communication medium, such as the pre-processed frames A207. The downstream nodes A205 may each process the framesAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WG PATENT APPLICATIONA207 and / or sub-frames thereof in a manner consistent with tasks associated with the image data captured by the endoscope, which may include various ML tasks and / or other tasks that require high-speed data processing, as described herein.

[0096] As mentioned, the endoscope A202 generally captures image data through an optical system (not shown), which typically consists of a lens or a series of lenses and a light source at the tip of the endoscope. This optical system focuses light from the area being examined onto an image sensor (not shown), such as a Charge-Coupled Device (CCD) or Complementary Metal-Oxide-Semiconductor (CMOS) sensor, located within the endoscope. The image sensor generally captures image data, such as live video, in a sequence of frames (e.g., frames A203) that represent the endoscope’s A202 field of view (FOV) over time.

[0097] Once the image data is captured by the endoscope A202, the endoscope A202 may transfer the raw image frames A203 to the ESC A204, which includes both a Field-Programmable Gate Array (FPGA) AA208 and a Graphics Processing Unit (GPU) A213. The FPGA A208 may be subdivided into a camera pipeline front end A210 and a camera pipeline back end A211, which may generally perform one or more pre-processing functions on the received raw image frames A203. Broadly speaking, the FPGA AA208 and GPU A213 may each process the received raw image frames A203, and each processing unit may perform certain processing tasks based on advantageous characteristics of the respective processing units.

[0098] For example, the raw image frames A203from the endoscope A202 image sensor maybe first sent to and processed by the FPGA AA208 within the ESC A204, because FPGAs are generally efficient at performing parallel processing tasks and can be programmed to handle specific image processing functions such as noise reduction, image stabilization, and / or initial data compression, which may be performed before other pre-processing steps. Thus, the FPGA AA208 may preprocess the raw image frames A203 to enhance the overall image quality and prepare the raw image frames A203 for further processing in real-time and may transfer the partially pre-processed image frames A212 to the GPU A213. This transfer may occur over a high-speed data connection within the ESC A204, designed to handle the bandwidth requirements of high-resolution video data.

[0099] The GPU A213 may then perform one or more other processing tasks, such as advanced image enhancement techniques, real-time video analytics, and / or generation of 3D reconstructions from the captured image data. GPUs generally provide powerful parallel processing capabilities and are typically capable of performing more complex image processing tasks more quickly than the FPGA AA208. In certain embodiments, the GPU A213 may process multiple frames A212 simultaneously, ensuring smooth and high-quality video output, e.g., at the one or more downstream nodes A205. Additionally, or alternatively, the GPU A213 may compress the image data, format the data for compatibility with display devices, integrate the image data with other relevant data (e.g., patient data) for comprehensive visualization, and / or otherwise process the image data to prepare the data for display or storage by the GPU A213.

[0100] However, in some embodiments, the FPGA AA208 may not transfer image data to the GPU A213 and may instead transfer the image data directly to the downstream nodes A205 after pre-processing or without pre-processing. For example, the image data acquired by the endoscope A202 may not require complex image processing to be suitable for the task associated with the image data, such that the FPGA AA208 may be capable of performing the pre-processing entirely. In these circumstances, the FPGA AA208 may pre-process the image data (e.g., via the camera pipeline front end A210 and backend A211) and may transmit the pre-processed image frames A207 to the downstream nodes A205 for task processing and / or display.

[0101] Within the FPGA AA208, the camera pipeline front end A210 and the camera pipeline back end A211 may each perform different pre-processing tasks. The camera pipeline front end A210 may generally be responsible for acquiring raw image data (e.g., frames A203) from the endoscope A202 and performing one or more corrections / adjustments to the frames A203 before sending the frames A203 to the camera pipeline back end A211. This typically involves interfacing with the endoscope A202 image sensor to receive raw pixel data, e.g., in a Bayer pattern or other raw formats. In one example, the camera pipeline front end A210Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WG PATENT APPLICATIONmay perform black level correction, demosaicing (e.g., converting the Bayer pattern to full-color images), and / or white balance adjustment to convert the raw image frames A203 into a format that more closely represents the visual scene. The camera pipeline front end A210 may also apply noise reduction algorithms to the image frames A203 to improve image quality, for example, in low-light conditions where the endoscope A202 image sensors may introduce significant noise. Further, the camera pipeline front end A210 may correct the image frames A203 for optical distortions introduced by the endoscope's A202 lens system, ensuring that the images accurately represent the observed scene without warping or other artifacts. Additionally, or alternatively, the camera pipeline back end A211 may correct and / or otherwise perform portions of the image frames A203 correction for optical distortions, such as de-warping the image frames A203 after an initial frame of the image frames A203 has buffered and / or otherwise been transmitted to the camera pipeline back end A211.

[0102] More generally, the camera pipeline back end A211 may perform more complex image processing tasks, such as edge enhancement, contrast adjustment, and / or color correction, to further improve image data quality and ensure that certain details are visible (e.g., to medical professionals performing a surgical operation). As one example, the camera pipeline back end A211 may apply compression algorithms to the pre-processed image frames to reduce their file size for storage or transmission, which may help manage the large volumes of data generated during endoscopic procedures. The back end A211 may also implement image stabilization techniques to compensate for motion or vibrations, ensuring that the image frames remain clear and stable despite movements of the endoscope A202 and / or the patient. To assist with real-time display or recordings, the back end A211 may also encode the processed images into video streams by converting the sequence of image frames (e.g., partially pre-processed image frames A212) into a standard video format that can be easily viewed and / or stored.

[0103] When the ESC A204 has finished pre-processing the raw image frames A203, the ESC A204 may transmit the pre-processed image frames A207 to the one or more downstream nodes A205. The downstream nodes A205 may generallyrepresent computing devices configured to further process pre-processed image data (e.g. pre-processed image frames A207) received from medical imaging devices, such as the endoscope A202. The downstream nodes A205 may be responsible for performing multiple tasks, in accordance with a set of applications A215 stored on and / or otherwise accessed by the downstream nodes A205. For example, the downstream nodes A205 may further refine and enhance the pre-processed image frames A207 to improve visibility and clarity for a surgical team by adjusting brightness, contrast, and / or sharpness, as well as applying advanced image processing algorithms to highlight features or structures within the image frames A207. The downstream nodes A205 may also render the enhanced image data for real-time display on monitors and / or heads-up displays used by the surgical team to enable the surgeons to view high-quality images and / or video feeds during procedures, thereby aiding in navigation and decision-making. As part of these displays, the downstream nodes A205 may also integrate the processed image data with other relevant patient data and / or surgical information, such as overlaying annotations, measurements, and / or other visual aids onto the images, as well as synchronizing the image data with surgical instrument positions or other procedural data. The downstream nodes A205 may also manage the recording and storage of image / video data for documentation, review, and / or analysis purposes by encoding the data into suitable formats for storage, organizing the data for easy retrieval, and / or ensuring data integrity and security. Further, the downstream nodes A205 may also facilitate the transmission of processed image data to other components of the surgical system (e.g., cloud-based servers) and / or to external systems for further analysis, telestration, and / or remote consultation.

[0104] More specifically, the downstream nodes A205 may include an FPGA A214 that processes received data in accordance with instructions included in the set of applications A215. The FPGA A214 may also transmit data to a GPU A216 that may be configured to process the data in a manner that the FPGA A214 cannot and / or is otherwise inconsistent with the requirements of an application ofthe set of applications A215. For example, the FPGA A214 may receive the pre-processed frames A207 and may transmit the frames A217, in accordance with instructions from an application of the set of applications A215, to theAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION GPU A216 for task processing. In certain scenarios, the GPU A216 may process the frames A217 in tandem and / or otherwise in combination with the FPGA A214 to perform one or more tasks indicated by the application, or the GPU A216 may independently execute the instructions of the application from the set of applications A215 to process the frames A218 accordingly. Of course, while illustrated in FIG. 2A as a single node A205, it should be appreciated that the architecture A200 may include a plurality of downstream nodes A205. Moreover, each downstream node A205 may include any suitable set (e.g., type or quantity) of processing devices / units, such as FPGAs A214, GPUs A216, CPUs, and / or any programmable logic device (e.g., microcontrollers).

[0105] The set of applications A215 may generally include instructions configured to enhance, analyze, and / or otherwise manage image data (e.g., pre-processed image frames A207) received from the ESC A204 for use in, e.g., robotic-assisted surgical procedures. For example, the set of applications A215 may include image enhancement applications that cause the FPGA A214 and the GPU A216 to apply real-time image enhancement techniques to the pre-processed image frames A207, such as sharpening, denoising, color correction, and / or dynamic range adjustment to improve the visual quality of endoscopic images and thereby aid surgeons in identifying anatomical structures more clearly. As another example, the set of applications A215 may include 3D reconstruction applications that leverage parallel processing power of the GPU A216 to reconstruct 3D models of a surgical site from 2D endoscopic images. In this example, the FPGA A214 and the GPU A216 may work together to perform computationally intensive tasks such as stereo matching, depth estimation, and / or surface rendering to provide surgeons with a 3D perspective that can enhance spatial understanding and navigation during surgery.

[0106] In certain embodiments, the set of applications A215 includes augmented reality (AR) applications that overlay virtual information, such as surgical planning data or anatomical labels, directly onto a live endoscopic video feed. In these embodiments, the FPGA A214 and / or the GPU A216 may process the pre-processed image frames A207 to accurately align and integrate the virtual overlays with the real-world images, thereby enhancing situational awareness without obscuring visual information. In some embodiments, the set of applications A215 includes video encoding and streaming applications that are configured to encode the enhanced and (possibly) augmented image data into efficient video formats for real-time display, recording, and / or remote streaming. For these applications, the GPU A216 may handle the encoding tasks due to its efficiency in parallel processing of video data, while the FPGA A214 may manage data transmission and network communication tasks via afiber A206.

[0107] Moreover, in some embodiments, the set of applications A215 includes artificial intelligence (Al) and machine learning (ML) applications. These AI / ML applications may be configured to analyze the pre-processed image frames A207 to support tasks such as automatic detection of surgical instruments, identification of anatomical landmarks, and / or real-time monitoring of surgical progress. The GPU A216 may execute complex ML models and algorithms by leveraging its computational power for rapid analysis, while the FPGA A214 may be used for preprocessing or accelerating specific Al tasks.

[0108] More generally, it should be appreciated that the downstream nodes A205 can include one or multiple computing devices that are co-located or distributed. Moreover, in some embodiments, the downstream nodes A205 may be located / stored in a remote location from the ESC A204 (e.g., a cloud-based server). In these embodiments, the ESC A204 accesses the downstream nodes A205 by transmitting data (e.g., pre-processed image frames A207) to the cloud-based server. The downstream nodes A205 analyze the inputs (e.g., based on the set of applications A215), generates outputs (e.g., enhanced / processed image frames / data), and the cloud-based server returns these outputs to the ESC A204.

[0109] It should also be understood that the ESC A204 and downstream nodes A205 include memories that store executable instructions (e.g., set of applications A215) that are configured to, when executed by one or more of the processing units illustrated in FIG. 2A (e.g., FPGAs AA208, A214, GPUs A213, A216), cause the processing units to analyze data (e.g., raw image framesAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONA203, pre-processed image frames A207) received from the endoscope A202 and output various values (e.g., control instructions, processed image data). These control instructions may also include executable instructions, as well as other data.

[0110] Further, each of the processing units illustrated in FIG. 2A (e.g., FPGAs AA208, A214, GPUs A213, A216) may include any suitable number of processors. For example, the GPU A216 may include one or more GPUs and the FPGA A214 may include one or more FPGAs. Generally, each of the processing units AA208, A213, A214, and A216 maybe configured to execute software instructions (e.g., set of applications A215) stored in each of the corresponding memories (not shown), which may each include one or more persistent memories (e.g., a hard drive and / or solid-state memory).

[0111] It will be understood that the above disclosure is one example and does not necessarily describe every possible embodiment. As such, it will be further understood that alternate embodiments may include fewer, alternate, and / or additional steps or elements.

[0112] FIG. 2B depicts an example robotic controller architecture A220 utilizing sub-frame chunking of data to improve processing efficiency, in accordance with various embodiments described herein. The example robotic controller architecture A220 may utilize many similar elements of the architecture A200 illustrated in FIG. 2A but may execute instructions configured to optimize intra-processor data transfer and processing. Namely, the example robotic controller architecture A220 may iteratively process sub-frame chunks of raw image data utilizing optimal processing units to quickly and efficiently determine outcomes associated with the processing tasks and / or the corresponding system (e.g., a robotic surgery system).

[0113] The example robotic controller architecture A220 includes an endoscope A222 that transmits raw image frames A223 to an ESC A224 that is communicatively connected with one or more downstream nodes A225 via a fiber (e.g., a fiber-optic cable) A226 configured to transmit pre-processed image frames A227. More specifically, the ESC A224 may transmit one or more subframe chunks A227a of one or more of the pre-processed image frames A227 to the one or more downstream nodes A225 to begin processing on the sub-frame chunks A227a, for example, in advance of one or more other sub-frame chunks of the same image frame. The downstream nodes A205 may then iteratively compile fully processed frames based on analysis of each of the subframe chunks A227a and determine outcomes of the individual tasks and / or implications for the system(s) (e.g., control instructions) during the iterative compiling (e.g., prior to completing full-frame processing). In this manner, the example robotic controller architecture A220 leverages efficient data transfer without full-frame latency by employing sub-frame (e.g., striped) processing to hide / minimize latency and manage boundary effects.

[0114] The ESC A224 includes an FPGA A228 and a GPU A233 that may be configured to pre-process the raw image frames A223, as described herein. For example, the FPGA A228 may initially receive the raw image frames A223 and may pre-process the raw image frames A223 using the camera pipeline front end A230 and / or the camera pipeline back end A231 and / or may transmit one or more frames A232 and / or portions thereof to the GPU A233 for pre-processing.

[0115] In particular, the FPGA A228 and / or the GPU A233 may also execute instructions (e.g., in accordance with one or more of the applications A235) that cause the FPGA A228 and / or the GPU A233 to chunk one or more of the individual raw image frames A223 into sub-frame chunks (e.g., sub-frame chunk A227a). To chunk the raw image frames A223, the FPGA A228 and / or the GPU A233 may divide each raw image frame A223 into smaller chunks along one dimension, typically horizontally or vertically, resulting in one or more sub-frame chunks A227a, each containing a part of the image data of the pre-processed image frames A227. Thus, each sub-frame chunk A227a may correspond to a portion, a “line”, or a “stripe” of the image frame, covering a specific section from one edge to the otherwhere, e.g., in a horizontally divided image frame, each stripe might represent a horizontal slice of the image frame. Of course, the sub-frame chunks described herein may be or represent groupings of the full-frame pixel data in any suitable configuration (e.g., sub-rectangles).Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION

[0116] By breaking the image frames A223 into sub-frame chunks A227a, the systems described herein can transmit and / or process each sub-frame chunk A227a independently, allowing for parallel processing of different sub-frame chunks A227a of an individual image frame, reducing the overall computational load and generally expediting the processing and / or transmission time. After transmission and / or processing, the downstream nodes A225 may reassemble the sub-frame chunks A227a into a completed, processed image frame by aligning and combining the sub-frame chunks A227a in the correct order to accurately reproduce the complete image frame A227.

[0117] The downstream nodes A225 may receive the sub-frame chunks A227a and determine how to process each individual chunk by determining tasks associated with applications to which the sub-frame chunks A227a correspond. For example, the sub-frame chunks A227a may represent portions of a medical image (e.g., during a surgical procedure), which are to be processed in accordance with a medical imaging application to assist a surgical team during the procedure. The downstream node A225 may receive an indication that the sub-frame chunks A227a are associated with the medical procedure from the ESC A204 (e.g., as a result of the pre-processing performed thereon) and / or the node A225 may make the determination independently through analysis performed by the FPGA A234 and / or the GPU A236. This determination may further involve the FPGA A234 and / or the GPU A236 accessing / interacting with the applications A235 to determine one or more tasks include as part of the applications with which the sub-frame chunks A227a may be associated.

[0118] As an example, the sub-frame chunks A227a may represent medical images of one or more organs within a human body that are captured using the endoscope A222, and the application A235 may include an image analysis application that is configured to analyze image data for the presence of particular organs within the images and / or any abnormalities on / near the particular organs. In this example, the FPGA A234 may be optimally configured to handle one or more processing tasks in association with this medical imaging application A235, such as extracting one or more features (e.g., edges, textures, shapes) of a sub-frame chunk to perform subsequent object identification or diagnosis, real-time filtering to enhance specific aspects of the sub-frame chunk (e.g., edge sharpening, speckle noise reduction), compressing the data for later storage, and / or executing a pattern recognition algorithm to identify predefined patterns (e.g., organ shapes / colors, etc.) or anomalies (e.g., foreign bodies) within the image data, among others. The GPU A236 may be optimally configured one or more other processing tasks in association with the medical imaging application A235, such as executing learning models (e.g., convolutional neural networks (CNNs), vision transformers (ViTs)) for tissue classification, lesion detection, automated diagnosis, 3D model reconstruction of observed tissues / organs from the 2D sub-frame chunks, image segmentation to delineate specific structures / organs or other regions of interest within the image (e.g., separating healthy tissue from diseased tissue), rendering high-quality, and / or real-time visualizations of medical images / 3D reconstructions for display during procedures, among others.

[0119] For example, ViTs generally divide input images into smaller patches or chunks, such as sizes of 8x8, 16x16, or 32x32 pixels, and then process these patches as individual data points. Each image patch may be treated similarly to a token in natural language processing, allowing the transformer architecture to analyze the image in a sequence of patches. The ViT may apply self-attention mechanisms across these patches to capture the relationships and contextual information between different parts of the image, enabling the ViT to learn and extract features from the image for tasks like classification, detection, or segmentation, as described above.

[0120] Moreover, in certain embodiments, the raw image frames A223 may correspond to images captured via any suitable imaging modality, such as magnetic resonance imaging (MRI), computed tomography (CT), ultrasound, fluorescence, hyperspectral, Raman imaging, and / or any other modalities. The downstream nodes A225 may ensure that image frames A223 from different imaging modalities are captured in parallel by, e.g., coordinating with the imaging devices to receive dataAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONsimultaneously, allowing for a comprehensive view of the subject (or other object) being imaged from multiple perspectives or modalities.

[0121] Upon receiving the image frames, the downstream nodes A255 may identify the specific imaging modality of each frame, as different modalities may require unique processing techniques to optimize image quality. For example, MR images might undergo noise reduction and contrast enhancement, while ultrasound images may require edge sharpening to improve clarity. To address the storage and transmission challenges associated with high-resolution and multi-modal imaging data, the downstream nodes A225 may apply efficient compression techniques to the image frames A223, such as by encoding 3D video as a delta from a 2D image to leverage the similarities between consecutive frames or across modalities and reduce redundancy. For instance, the downstream nodes A225 may compress fluorescence or hyperspectral imaging data by exploiting correlations with structural information provided by MR or CT images.

[0122] The downstream nodes A25 may also explore ways to leverage information across different imaging modalities to achieve more efficient compression and storage. By analyzing the relationships and complementary information provided by each modality, the nodes A225 can identify opportunities to encode and store data in a manner that reduces overall file size without compromising, e.g., diagnostic value. For example, the nodes A225 may create a composite data structure that integrates data from all modalities, where the information from one modality enhances or refines the data from another.

[0123] After processing and compression, the downstream nodes A225 may store the image frames A223 in a structured manner that facilitates easy retrieval. For example, the nodes A225 may organize data by patient, imaging session, and / or modality, and implement indexing mechanisms to quickly access specific images or modalities as needed. The downstream nodes A225 may also generate a 4-panel video that simultaneously displays image frames from different modalities, providing a comprehensive view of the imaging data. Additionally, or alternatively, the downstream nodes A225 may store data across files in a way that maintains the association between modalities while optimizing for storage efficiency and access speed.

[0124] The downstream nodes A225 may then cause the FPGA A234, the GPU A236, and / or any other programmable logic device (not shown) to analyze the relevant sub-frame chunks A227a based on the tasks associated with the relevant application(s) A235. The processing devices (A234, A236) may then also iteratively compile the frame based on sequential analysis of the subframe chunks A227a to determine an outcome of the tasks associated with the corresponding application A235 during the iterative compiling of the frame. This outcome may generally be related to and / or otherwise influencing the operation or outcomes of a computer-assisted system, such as the example robotic controller architecture A220. It should also be appreciated that the subframe chunking may be performed by either the ESC A224 or the downstream nodes A225, depending on the particular implementation and / or the transfer latency of the particular types / amount of image data being transmitted between the ESC A224 and the downstream nodes A225.

[0125] As an example, the endoscope A222 may capture image data during a surgical procedure of a particular organ of interest (e.g., pancreatic and bile ducts) that has obstructions / blockages, and / or damaged / diseased tissue. The endoscope 22 transfers the raw image frames A223 of the ducts to the ESC A224 for pre-processing and / or sub-frame chunking and transfer to the downstream nodes A225 as sub-frame chunks A227a. As these sub-frame chunks A227a arrive, the FPGA A234 and / or the GPU A236 may process each chunk A227a, A237a, A238a sequentially in accordance with algorithms / models / instructions associated with one or more of the applications A236 that are configured to detect abnormalities or diseased tissue within the ducts. When the FPGA A234 and / or the GPU A236 identifies a potential issue, such as an area of constriction or the presence of stones, the downstream node A225 may immediately generate an output (e.g., visual markers or alerts that are superimposed on a live video feed displayed to the surgeons). The FPGA A234 and / or the GPU A236 may continue evaluating one of more of the sub-Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONframe chunks A227a after the output is displayed, to reduce / minimize the processing latency typically associated with image processing in these medical imaging tasks (and others).

[0126] In certain embodiments, the tasks described herein performed by the downstream nodes A225 and / or the ESC A224 may be performed automatically or upon receipt of a user input to perform such tasks. The downstream nodes A225 may continuously monitor for user input, which could be in the form of voice commands or natural language queries from, e.g., a surgeon during a procedure. The downstream nodes A225 may implement this listening function using a speech recognition system or natural language processing (NLP) module integrated into the downstream nodes A225. Upon receiving a verbal request or question from the surgeon, such as “Please help me with this anatomy to find the tissue plane,” the downstream nodes A225 may process this input using NLP techniques to extract key information and intent from the natural language query and identify / trigger one or more actions or models relevant to the surgeon's query. The downstream nodes A225 may activate specific computational models, image processing algorithms, and / or robotic controls tailored to assist with the identified task. For example, if the surgeon requests assistance with identifying a tissue plane, the downstream nodes A225 may activate models designed for anatomical segmentation or enhancement of relevant imaging modalities, as described herein.

[0127] To determine which sub-frame chunks A227a to process first, the systems described herein may evaluate a full frame of image data received from the endoscope A222 to identify an expected position of a feature or region of interest. Once identified, the applications A235 may include instructions that cause the FPGA A234, the GPU A236, and / or the processing components of the ESC A224 to transfer / analyze (e.g., prioritize) sub-frame chunks (e.g., A227a, A237a, A238a) from the expected position within any subsequently received image frames first to reduce / minimize the latency in high-demand image processing tasks for the region of interest. For example, the expected position of the feature or region of interest may be represented in coordinate values (e.g., Cartesian coordinates) corresponding to pixel positions within the endoscope imaging system FOV.

[0128] Continuing the prior example, the ESC A224 and / or the downstream nodes A225 may analyze a full frame of image data captured by the endoscope A222 that represents the pancreatic and / or bile ducts. The downstream nodes A225 may process / analyze the full frame of image data in accordance with one or more applications A235 to identify the region of interest within the full frame, which may extend from a top left corner of (400, 300) to a bottom right corner of (800, 600) as a rectangular area for an imaging system with a resolution of 1920 by 1080 pixels. The downstream nodes A225 may store this value and prioritize analyzing sub-frame chunks A227a, A237a, A238a within this coordinate range to expedite the image processing tasks associated specifically with the region of interest. The downstream nodes A225 may also transmit this value to the ESC A224 to cause the ESC A224 to prioritize transmission of sub-frame chunks A227a that are within this coordinate range. Thus, when the endoscope A222 captures subsequent image frames of the bile ducts, the image processing tasks associated with the region of interest may be performed as quickly as possible by prioritizing the sub-frame chunks A227a, A237a, A238a that are known to include image data corresponding to the region of interest.

[0129] In certain embodiments, in addition to focusing on sub-frame based compute usage, certain activities, such as the image processing tasks associated with these regions of interest, other processing tasks, determinations and / or decision-making steps may make appropriate use of compute based on different timescales on which the activities operate. Specifically, certain of these activities may operate on different timescales to deliver desired context relevant information using appropriate compute. For example, some may operate in much slower timescales from seconds to minutes, as the use-cases and / or other complementary processes allow for such processing timescales. Al agents implementing complex ML models and algorithms may be able to take more times to process these slower timescale tasks, determinations, decision-making steps because those can be operated at these slower timescales without interrupting the relevant use cases. In these embodiments, rather than relying on sub-frameAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONchunking and / or prioritized transmission of particular sub-frame chunks A227a indicating a region of interest, the Al agents may be utilized to allocate or schedule appropriate compute power necessary to execute the activities at a slower timescales. For example, these slower timescales activities may be operating in the background by utilities compute power over seconds to minutes, continuously or discretely by dividing the activities into sub-activities while scheduling faster timescales activities to utilize compute power either in parallel with the slower timescales activities or weaved in between computer power usage of the slower timescales activities. For example, a robotic assistant device performing image analysis tasks as part of a surgical procedure may receive a query from a surgeon that states “What anatomy am I looking at? Can you track it and provide me a notification / update if it goes offscreen?” The robotic assistant device may be able to perform these tracking / notification tasks on the order of seconds. Al agents can allocate or schedule the needed compute power usage knowing the timescales over which the activities will be completed in light of other compute usage needed for other timescales. Alternatively, this may be performed using the sub-frame chunking described herein to improve the efficiency of such processes by identifying, prioritizing, and transmitting the specific subframe chunks A227a associated with the “anatomy” the surgeon is currently viewing. As another example, the surgeon may ask the robotic assistant device to “Please let the patient’s family know when I am closing up the surgical site”, which may generally require on the order of minutes for the robotic assistant device to complete. The timescales context aware Al agents can appropriately schedule efficient use of compute usage in addition to using chunking techniques described herein to improve the processing time required for such an operation by focusing on regions of interest that indicate when / whether the surgeon is closing the surgical site, and prioritizing notifications of the patient’s family when sub-frame chunks A227a corresponding to the surgical site indicate the surgeon completing the surgical procedure by closing the surgical site.

[0130] In certain embodiments, the downstream nodes A225 may receive (e.g., from the ESC A224) a model of lumens for the endoscope A222 and / or kinematic data of the endoscope A222. The downstream nodes A225 may then also determine the expected position of the feature of interest (e.g., within the raw image frames A223 and / or sub-frame chunks A227a, A237a, A238a) based on the model and the kinematic data. Generally, a model of lumens for the endoscope A222 may refer to the design or configuration of the light-conducting channels within the endoscope A222 that illuminate the area being examined. Lumens may provide lighting to capture clear images of internal body structures during endoscopic procedures, and the model of lumens may detail the number of lumens, their arrangement, the intensity and type of light they emit, and / or how they are controlled to optimize visibility. The kinematic data of the endoscope A222 pertains to the positional and movement information of the endoscope A222, such as its orientation, bending angles, insertion depth, and / or any rotational movements, which may enable the downstream nodes A225 to understand the endoscope's A222 physical location and orientation within the body during a procedure.

[0131] Thus, the downstream nodes A225 may utilize this information (e.g., model of lumens, kinematic data) to help determine the region of interest within the captured image data. For example, the nodes A225 may utilize the model of lumens to determine how well an area is illuminated, which directly affects the visibility of features within the captured images. By understanding the lighting configuration, the downstream nodes A225 can generate control instructions / recommendations that may be implemented by the ESC A224 to enhance the illumination of specific areas, making features of interest more discernible. As another example, the downstream nodes A225 may utilize the endoscope’s A222 precise position and orientation at the time each image is captured, as indicated by the kinematic data, to map features within the images to their actual locations within the body. Knowing the endoscope's A222 bending angle and insertion depth at a particular time may enable the nodes A225 to, e.g., pinpoint the exact location of a lesion or abnormality seen in the image at the particular time.

[0132] As another example, the downstream nodes A225 may combine the model of lumens with the kinematic data to correlate specific features seen in the images with their physical locations. For instance, if a feature of interest is best illuminated under certain lumen configurations and the kinematic data indicates the endoscope's A222 position, the downstream nodes A225Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONmay calculate the feature's location relative to the endoscope's A222 tip or to known anatomical landmarks. Further, in another example, the downstream nodes A225 may utilize the model of lumens and the kinematic data to inform image processing algorithms (e.g., included as part of the applications A235) to enhance the visibility of features of interest, correct for distortions, provide augmented reality overlays that indicate the feature's position within the body, and / or any other suitable actions or combinations thereof.

[0133] In certain embodiments, the downstream nodes A225 may also cause the FPGA A234, the GPU A236, and / or any other programmable logic device(s) to determine processing requirements of the tasks (e.g., dynamic / static) for a particular application and processing requirements of the particular application (e.g., ordering of tasks overall, high / low priority tasks) to allocate the tasks between the FPGA A234, the GPU A236, and / or other programmable logic device(s). This task allocation performed by the downstream nodes A225 may generally be based on the respective processing capabilities of the FPGA, A234, the GPU A236, and / or any other programmable logic device(s), as compared to the processing requirements of the corresponding tasks and the overall application. In some embodiments, the processing requirements relate to a task / phase of a medical procedure, and the image frames (e.g., raw image frames A223) and the sub-frame chunks (e.g., chunks A227a) are medical images of, e.g., a surgical procedure.

[0134] As mentioned, different processing units may have different advantages regarding particular processing task types. For example, GPUs are generally designed for highly parallel operations, making them configured for tasks that can be divided into smaller, concurrent processes, such as image and video processing, deep learning, and scientific simulations. FPGAs are inherently flexible and can be reprogrammed to perform specific tasks with optimized hardware configurations. To leverage these specific advantages from the various computing resources, the downstream nodes A225 may allocate the tasks associated with any particular application / program among the available computing resources (e.g., the FPGA A234, the GPU A236, and / or any other programmable logic device(s)).

[0135] The downstream nodes A225 and / or any other computing devices described herein may perform this task allocation and data transfer between the relevant computing resources in various manners. For example, in the context of the endoscope A222, the ESC A224 and / or the downstream nodes A225 may perform benchmarking and / or run-time monitoring of the computing resources (e.g., FPGAs, GPUs, CPUs, etc.) when performing the tasks. The ESC A224 and / or the downstream nodes A225 may monitor the end-to-end vision process (e.g., utilizing the endoscope A222) and processing pipelines using servo-sync timestamps and application-level instrumentation. By tagging data packets or processing steps with servo-sync timestamps, the ESC A224 and / or nodes A225 may accurately track the timing and sequence of tasks / operations, thus enabling the analysis of latency and synchronization issues. The ESC A224 and the downstream nodes A225 may utilize application-level instrumentation by, e.g., embedding monitoring and diagnostic tools directly into the software or firmware of the system, allowing for detailed tracking of how data moves and is processed within the system and for insights into performance at the application layer.

[0136] The ESC A224 and / or the downstream nodes A225 may also determine empirical measurements of timing information around the key testing points along these pipelines (e.g., how long it takes for data to be transferred between the FPGA and GPU domains, processed, and outputted). The ESC A224 and / or the nodes A225 may thereby determine the overall latency and perform iterations towards optimal design and implementation by facilitating changes to the allocation the tasks to one or more different processing resources and / or task sequencing changes that optimize the overall processing efficiency of the set of tasks associated with a particular application / program.

[0137] In particular, transferring the data from the FPGA A234 to the GPU A236 and / or other programmable logic devices may also be optimized by the nodes A225 to facilitate efficient data processing. For example, many known methods (e.g., PCIeAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONdirect transfers, NVIDIA GPUDirectfor RDMA, NVLink / NVSwitch (Experimental) and Direct memory mapping with shared memory buffers) provide FPGA-to-GPU data exchange, but in medical applications, each method may have risks and challenges. Notably, many of these data transfer methods frequently experience issues around compatibility and data integrity, and addressing these risks typically requires compatibility verification, stringent synchronization, and thorough testing under target workloads. Others have their different challenges, such as data synchronization issues, cache coherency problems, debugging and error handling. Thus, to mitigate these risks, the ESC A224 and / or the nodes A225 may utilize / implement explicit memory barriers, buffer management, and / or performance profiling under target workloads to provide stringent control over access patterns and systemlevel tuning for reliable and efficient operation.

[0138] Moreover, the algorithms / models represented by the applications A235 may be configured to operate on partial frames (e.g., sub-frame chunks A227a, A237a, A238a), such that the processing units (e.g., FPGA A234, GPU A236) are not required to waitfor the full frame to complete transfer (e.g., from the ESC A224) before processing can begin. Thus, the applications A235 can execute via the processing devices / units in a manner that significantly reduces latency, as full-frame transfers are no longer required to perform tasks on regions of interest within image data from the endoscope A222. In certain embodiments, the algorithm / models of the applications A235 may process the sub-frame chunks A227a, A237a, A238a as if the algorithms / models were processing full frames, and the sub-frame transfer, processing, boundary-artifacts removal, etc. are performed by the ESC A224 and / or nodes A225.

[0139] Each of the sub-frame chunks A227a, A237a, A238a may need to overlap, so the nodes A225 may later remove any boundary artifacts between chunks A227a, A237a, A238a. The ESC A224 and / or the nodes A225 may thus modify the sub-frame chunks A227a, A237a, A238a to ensure that the tasks allocated to the processing units (e.g., FPGAs, GPUs, etc.) may adequately perform each of the tasks. For example, the ESC A224 and / or the nodes A225 may extend the margins of each sub-frame chunk A227a, A237a, A238a to include a margin of pixels from adjacent stripes / lines included in the sequence of sub-frame chunks. In this manner, the ESC A224 and / or the nodes A225 may provide the necessary context around the boundaries of each sub-frame chunk A227a, A237a, A238a to enable the processing units to perform the operations specified in the application tasks. The ESC A224 and / or the nodes A225 may also merge the overlapping margins of processed sub-frame chunks (e.g., A237a, A238a) to avoid duplications or inconsistencies within the compiled image data.

[0140] Additionally, or alternatively, the ESC A224 and / or the nodes A225 may encode the image frames A227 using a coarse-to-fine approach (e.g. discrete-wavelet-transform (DWT), discrete-cosine-transform (DCT), etc.) to reduce latency for algorithms / models (e.g., as part of applications A235) that may need to operate on full frame (e.g. for context). For example, the ESC A224 may transfer a lower-resolution version of the image frames A227 to the nodes A225 first, allowing the downstream algorithms / models (e.g., associated with applications A235) to initiate processing, while the ESC A224 transfers the high-resolution details subsequently. This progressive transfer of higher resolution image frame data allows for progressive refinement (e.g. segmentation boundaries are refined using the high-resolution data in a second pass) of the image data analyzed, and generally allows for faster processing of the tasks to generate outputs regarding the region(s) / feature(s) of interest.

[0141] In certain embodiments, the downstream nodes A225 and / or ESC A224 may modify one or more processing parameters for an analysis algorithm (e.g., as part of an application A235) executed by the GPUs, the FPGAs, or other programmable logic devices. These modifications may adjust the analysis performed as part of the analysis algorithms and may also result in different allocations of the available processing resources (e.g., GPU A236, FPGA A234) to accomplish the modified analysis. The downstream nodes A225 and / or the ESC 224 may also cause the GPUs, the FPGAs, or other programmable logicAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONdevices to analyze the sub-frame chunks A227a and / or other data in accordance with the one or more modified processing parameters for the analysis algorithm.

[0142] For example, the boundary regions of the sub-frame chunks A227a may include fewer surrounding pixels, such that a smaller kernel or window may be used to perform certain operations than in higher pixel-density regions of the sub-frame chunks A227a. The downstream nodes A225 may thereby modify one or more processing parameters of image analysis algorithms included as part of the applications A235 by reducing the size of the kernel or window used in operations like convolution or smoothing when applied near the edges of the sub-frame chunks A227a and / or other regions of the chunks A227a with relatively few surrounding pixels and / or otherwise having limited context.

[0143] As another example, the downstream nodes A225 and / or ESC A224 may adjusting one or more analysis algorithms to apply different weights or rules near the boundaries of one or more sub-frame chunks A227a to compensate for any missing context. After the sub-frame chunks A227a are analyzed with the modified parameters, the processed chunk(s) may be integrated back into the larger frame or combined with other processed chunks. The processing parameters configured for areas with full contextual information can lead to edge artifacts when applied indiscriminately near boundaries, such as blurring, ringing, or abrupt changes in intensity. Thus, the nodes A225 and / or ESC A224 modifying the weights or rules applied near boundary regions as part of an analysis algorithm may create seamless transitions between chunks that do not exhibit discontinuities or artifacts resulting from the analysis.

[0144] In certain embodiments, the outcomes of the tasks / sub-tasks performed by the ESC A224 and / or the downstream nodes A225 are displayed on a collaboration interface (not shown). The display may incorporate one or more features of the outcomes produced by the sub-frame chunking and / or other processing as part of the tasks / sub-tasks and may enable collaboration with external entities to optimize task scheduling and data transfer. The downstream nodes A225 may provide processing data / metrics that indicate the efficiency of assigned / allocated tasks / sub-tasks to the available resources (e.g., GPU A236, FPGA A234), and may include outcomes as part of the collaboration interface that indicate whether the outcome was satisfactory. For example, if the tasks / sub-tasks are associated with analyzing certain prioritized sub-frame chunks A227a and providing an updated / annotated view of a region of interest in a certain period of time (e.g., milliseconds, seconds), the collaboration interface may include timing / processing details (e.g., processing load, data transfer rates, latency) related to the sub-frame chunk processing and an indication of whether the downstream nodes A225 hardware accomplished the processing tasks / sub-tasks in the allotted times. Further, this display may feature any of the sub-frame chunks A227a, A237a, A238a, assembled into partial / complete frames, as described herein.

[0145] Further, in certain embodiments, the downstream nodes A225 may be communicatively coupled with a cloud-based application server (not shown). In these embodiments, the downstream nodes A225 may interact with the cloud-based application server to leverage cloud or hybrid cloud solutions to supplement processing power, access cloud-based applications, and / or otherwise communicate with the cloud-based application server. The hybrid cloud solutions may include both edge computing nodes and cloud computing resources, such as the cloud-based application server. For example, the cloud-based application server may include one or more processing resources (e.g., GPUs, CPUs), and the downstream nodes A225 may transfer one or more sub-frame chunks (e.g., chunks A227a, A237a, A238a) to the cloud-based application server for processing in accordance with instructions included as part of one of the applications A235 and / or one or more applications stored on the cloud-based application server (e.g., for an application that may not be high priority in the current medical context). The cloud-based application server may process the data and may transfer the processed data back to the downstream nodes A225 and / or to the ESC A224. Continuing the prior example, the cloud-based application server may transfer processed sub-frame chunk data to the downstreamAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONnode A225, where the node A225 may combine the processed sub-frame chunk with one or more other processed sub-frame chunks that were processed at the downstream nodes A225 and / or by the cloud-based application server.

[0146] FIG. 20 depicts an example compute resource partitioning scheme A240 that is managed by a compute partitioning coordinator A262, in accordance with various embodiments described herein. Generally, the example compute resource partitioning scheme A240 depicts a change in compute resource allocation between the first node A242 that lacks a partitioning coordinator and a second node A243 that includes a compute partitioning coordinator A262. In certain instances, the second node A243 may be the first node A242 after install ation / instantiation of the compute partitioning coordinator A262.

[0147] As illustrated in FIG. 20, the example computer resource partitioning scheme A240 includes a first node A242 and a second node A243. The first node A242 have multiple applications A244-A248, a CPU A249, and a GPU A250. Each of the CPU A249 and the GPU A250 may have multiple cores, such as CPU cores A251-A253, and the GPU cores A254-A256. In particular, N and M may be any integer values, such that the A / th Core A253 of the CPU A249 and the Mth Core (e.g., instruction unit) A256 of the GPU A250 may represent any suitable core (e.g., 2, 4, 5, 10, etc.).

[0148] The example computer resource partitioning scheme A240 further includes the second node A243 that has multiple applications A257-A261, the compute partitioning coordinator A262, a CPU A263, and a GPU A264. Each of the CPU A263 and the GPU A264 may have multiple cores / instances, such as CPU cores A2651-A267, and the GPU instances A268-A270. In particular, N and M may be any integer values, such that the A / th Core A267 of the CPU A263 and the Mth Core (e.g., instruction unit) A270 of the GPU A264 may represent any suitable instance (e.g., 2, 4, 5, 10, etc.).

[0149] The compute partitioning coordinator A262 may handle processing resource allocation by determining how computational tasks are distributed between CPU A263 and the GPU A264, as well as among their respective cores. The coordinator A262 may assesses the processing requirements of each task, such as whether it is better suited for sequential processing by the CPU A263 or parallel processing by the GPU A264 and may allocate the task accordingly to optimize performance.

[0150] Moreover, the compute partitioning coordinator A262 may perform task scheduling by scheduling tasks in a manner that maximizes resource utilization while minimizing idle / blocking time. This may involve queuing tasks, prioritizing them based on urgency or importance, and / or dynamically adjusting the schedule as tasks are completed or as new tasks arrive. For tasks that can be parallelized, the coordinator A262 may manage the division of the task into sub-tasks that can be executed simultaneously across multiple cores / instances of the CPU A263 or the GPU A264, to thereby ensure that dependencies between sub-tasks are respected and that data is appropriately synchronized between cores.

[0151] More specifically, the compute partitioning coordinator A262 may determine (i) a set of tasks for the one or more applications (e.g., applications A257-A261) to be executed by the compute node, (ii) a task prioritization level (e.g., high priority, low priority, medical task, non-medical task, etc.) for each task of the set of tasks, and / or (ill) compute resource requirements (e.g., task completion time, memory, processing time, maximum latency level, etc.) for each task of the set of tasks. Based these determinations, the compute partitioning coordinator A262 may further partition the set of tasks into a task schedule that specifies a respective processing core (e.g., A265-A267, A268-A270) of the CPU A263 and / or the GPU A264 to perform execution of one or more of tasks of the set of tasks. The compute partitioning coordinator A262 may further cause the GPU A264 and / or the CPU A263 (and any other suitable processing resources) to execute the partitioned / designated tasks in accordance with the task schedule. In certain embodiments, the compute partitioning coordinator A262 may determine the task schedule for each application A257-A261 based on the task schedule for every other application A257-A261 to be executed by the node A243.Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION

[0152] In order to perform these task scheduling functions, the compute partitioning coordinator A262 may utilize various scheduling schemes to manage the computational workload. For example, the compute partitioning coordinator A262 may implement and / or otherwise utilize a hypervisor, a container, and / or timeslicing scheduling. In virtualized environments, the coordinator A262 may utilize a hypervisor to partition the compute resources into virtual machines (VMs), each with allocated the CPU A263 and / or the GPU A264 resources. The coordinator A262 may then manage how resources are allocated to the different VMs.

[0153] The compute partitioning coordinator A262 may implement CPU A263 virtualization in numerous manners. For example, the coordinator A262 may utilize full virtualization (e.g., VMware ESXi, KVM), paravirtualization (e.g., Xen), hardware-assisted virtualization, OS level virtualization (e.g., LXC or other container services), and / or binary translation. In particular, hardware-assisted virtualization may enable the coordinator A262 to perform both hardware virtualization and partitioning by leveraging CPU A263 extensions to enable more efficient virtualization, allowing a hypervisor to create isolated partitions where multiple virtual machines can run with direct access to the hardware.

[0154] In containerized environments, the coordinator A262 may utilize and / or otherwise leverage a container management tool to isolate tasks within containers. The coordinator A262 may then schedule and manage containers across the CPU A263 and / or the GPU A264 resources. Additionally, or alternatively, for time-sharing systems, the coordinator A262 may implement and / or otherwise utilize timeslice or time-quantum scheduling, where each task is given a certain amount of processing time on the CPU A263 and / or the GPU A264 before switching to the next task. In this manner, the compute partitioning coordinator A262 ensures fair access to compute resources (e.g., CPU A263, GPU A264) across tasks.

[0155] More generally, the compute partitioning coordinator A262 may employ a variety of techniques to manage the workload of the available GPU A264 and / or CPU A263 resources / cores. It should be understood that the coordinator A262 may utilize any suitable combination(s) of the techniques described herein to manage the workloads of the available processing resources. For example, the compute partitioning coordinator A262 may manage the workload of the GPU A264 through single process concurrency at an application level by managing the concurrent execution of multiple tasks or threads within a single application running on the GPU A264. The compute partitioning coordinator A262 may organize an application's workload into parallelizable tasks that can be executed simultaneously by different cores (e.g., A268-A270) of the GPU A264, thereby ensuring that dependencies between the tasks / sub-tasks can be managed and that resources are allocated efficiently to maximize the GPU's A264 utilization without causing contention or bottlenecks.

[0156] As another example, the coordinator A262 may perform CUDA multi-process service (MPS) at a process level for the GPU A264. CUDA MPS may generally enable multiple CUDA applications to share a single GPU (e.g., GPU A264). The compute partitioning coordinator A262 may implement MPS by configuring the GPU A264 to allow concurrent execution of processes from different applications. The coordinator A262 may then manage the allocation of GPU A264 resources to these processes to provide fair access and optimize throughput by balancing the workload across available cores A268-A270.

[0157] As yet another example, the coordinator A262 may perform multi-instance GPU (MIG) at the hardware level. MIG is a technique that may partition a single GPU (e.g., GPU A264) into multiple smaller instances, each with its own set of resources (e.g., cores A268-A270, memory, etc.). In certain examples, and as previously mentioned, each of the cores A268-A270 illustrated in FIG. 2C may be individual instances of the GPU A264 that have their own resources. In any event, the compute partitioning coordinator A262 may implement MIG by dividing the GPU A264 into instances (e.g., A268-A270) tailored to the specific needs of different tasks / sub-tasks and / or applications (e.g., applications A257-A261). This technique may enable isolated executionAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WC PATENT APPLICATIONenvironments within the same GPU A264, which generally improves resource utilization and provides greater flexibility in managing workloads with varying requirements while minimizing chances of adverse impact across the isolated environments.

[0158] As yet another example, the coordinator A262 may perform GPU virtualization (vGPU) at a system level. GPU virtualization generally involves abstracting the GPU hardware (e.g., GPU A264) to create virtual GPUs that can be allocated to virtual machines (VMs) or containers. The compute partitioning coordinator A262 may implement vGPU by interfacing with virtualization software (not shown) to allocate virtual GPU resources to different VMs or containers based on their processing needs, such as assigning virtual functions (VFs) to different VMs. This technique may enable the coordinator A262 to provide efficient sharing of GPU A264 resources across multiple virtualized environments, allowing for scalable and flexible deployment of GPU-accelerated applications.

[0159] In some embodiments, the compute partitioning coordinator A262 may be further configured to create one or more virtual sandboxes within the GPU A264, where each sandbox may isolate a respective task to minimize negative interactions between concurrent tasks. Virtual sandboxes may generally represent an isolated environment within the GPU A264 and / or the CPU A263 where specific tasks, applications (e.g., A257-A261), and / or processes can run securely and independently from the rest of the node A243. In some examples, the coordinator A262 may generate virtual sandboxes within the GPU A264 that correspond to a process relevant to a medical procedure (e.g., processing kinematic data, closed loop control of tools using a force sensor, image processing of target features). These processes may require high processing stability (e.g., guaranteed timing consistency / completion in a specified time), such that the virtual sandboxes may provide the necessary processing isolation to ensure that no other external processes interfere with execution of the tasks / sub-tasks executed within the sandbox.

[0160] As an example, for applications involving real-time surgical imaging, the coordinator A262 may create a virtual sandbox within the GPU A264 to allocate dedicated GPU A264 resources to ensure the continuous and smooth processing of imaging data. In other words, the virtual sandbox within the GPU A264 may isolate the imaging application from other lower-priority processes, minimizing the risk of interference and ensuring reliable image delivery during surgeries. Accordingly, the compute partitioning coordinator A262 yields an improved reliability of task completion and enhanced image quality, to further ensure surgical precision and patient safety.

[0161] Additionally, or alternatively, the coordinator A262 may utilize sandboxing techniques to enhance fault-handling within the node A243. By creating isolated environments (sandboxes) for one or more applications (e.g., A257-A261), the coordinator A262 may prevent a fault in one sandbox from affecting others, supporting graceful degradation. Thus, the sandboxing performed by the coordinator A262 may improve fault handling by containing errors within isolated environments, preventing system-wide crashes and enabling quicker recovery times. Sandboxing also simplifies software development on the node A243 by allowing applications A257-A261 to operate as if they have sole ownership of resources, eliminating the need for the applications A257-A261 and / or any other component (e.g., coordinator A262) to account for resource sharing.

[0162] Further, the sandboxing performed by the coordinator A262 may improve the security of sensitive data involved in the processes isolated to the sandbox. For example, the coordinator A262 may create one or more sandboxes in the GPU A264 that are dedicated to processing data in an electronic health record (EHR). The coordinator A262 may segregate different modules (e.g., patient records, billing, scheduling) into separate sandboxes. This sandbox configuration may improve the data and overall EHR reliability by ensuring that a fault in one module (e.g., patient records) does not affect others (e.g., billing), may facilitate modular upgrades, and may enhance data security by isolating sensitive patient information to specific sandboxes.

[0163] In certain embodiments, the compute partitioning coordinator A262 also performs load balancing with respect to the CPU A263 and the GPU A264. The coordinator A262 monitors the load on the CPU A263 and / or the GPU A264 and mayAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONredistribute tasks as needed to prevent overloading one resource (e.g., while another is underutilized). This load balancing helps maintain optimal system performance by ensuring maximal utilization of available resources while minimizing the wear and deleterious impacts from overloading resources (e.g., due to excessive heat accumulation). For example, the coordinator A262 may determine that a second core A266 of the CPU A263 is overloaded by the current task and may reassign the task and / or a portion of the task (e.g., a sub-task) to the first core A265 of the CPU A263. The compute partitioning coordinator A262 may also handle and / or otherwise respond to failures and bottlenecks within the compute resources. The coordinator A262 may detect and respond to failures or bottlenecks by reallocating tasks to available resources or adjusting the task schedule to mitigate the impact on overall system performance.

[0164] Further, in certain embodiments, the compute partitioning coordinator A262 may be responsible for ensuring that medical software and non-medical software are effectively segregated / partitioned to minimize potential risks and impacts on medical applications (e.g., one or more of applications A257-A261). The coordinator A262 may generally create a clear separation between medical and non-medical software by establishing barriers between suitable hardware and / or software components. In particular, the coordinator A2626 may determine a level of isolation required, such as hardware isolation (e.g., separate physical components are dedicated to medical and non-medical software) or software isolation with hardware sharing (e.g., both types of software run on the same hardware but are isolated at the software or operating system level).

[0165] To make these determinations, the coordinator A262 may implement dedicated watchdogs and / or monitoring logic within the hardware and collaborate with the node’s A243 compiler to define a specific memory location of the barrier between medical and non-medical software. Specifying a memory location for the barrier ensures that each side of the barrier has optimized access to the compute and memory resources (e.g., CPU A263, GPU A264) required, without compromising security or performance. The watchdog and monitoring logic mechanisms may then monitor software tasks and processes to detect any attempts by non-medical software to access medical software resources (e.g., across the barrier). If such an attempt is detected, the watchdogs / monitoring logic may raise error flags, alerting the coordinator A262 to the potential breach.

[0166] The coordinator A262 may also evaluate any residual interactions across the partition boundaries (e.g., the barrier) that did / could occur despite the partitioning. For example, such residual interactions may be the result of limited bandwidth to shared memory or bus contention, which may indirectly impact performance. The coordinator A262 may evaluate whether shared resources, like GPU cores A268-A270 or memory, might lead to issues such as thermal throttling or reduced device clock speed due to stress tests run by one partition affecting others. As a result of these implemented mechanisms and corresponding evaluations, the coordinator A262 may determine a level of concern regarding residual interactions and their potential impact on medical applications. The coordinator A262 may then evaluate a degree of need for dedicated hardware for medical software, software isolation with hardware sharing, and / or partitioning larger devices to accommodate more significant barriers between the respective applications / processes. The coordinator A262 may also balance the benefits of dedicated resources, such as improved security and performance, against the cost and complexity of implementing separate hardware for medical applications before generating a partitioning output that indicates the degree of need for dedicated hardware and / or software isolation to sufficiently minimize the impact of residual interactions on medical applications from non-medical applications / processes.

[0167] Further, in certain embodiments, the compute partitioning coordinator A262 may receive and incorporate phase / medical task data when determining the task schedule. The node A243 may generate and / or otherwise receive data indicating one or more phase changes and / or medical task changes associated with a medical procedure and the coordinator A262 may utilize this data to adjust task / sub-task allocation to the available processing units (e.g., GPU A264, CPU A263). For example, during a transection phase of a surgical procedure, the compute partitioning coordinator A262 may allocate one or more GPU coresAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONA268-A270 to perform high-demand image analysis algorithms to ensure surgical personnel are able to clearly view the structure(s) to be cut. When the transection phase of the surgical procedure is completed and the surgical procedure proceeds to an extraction phase, the coordinator A262 may reallocate one or more of the GPU cores A268-A270 to perform a different task / sub-task, as the high-demand image analysis algorithms may not be as high priority in the extraction phase as during the transection phase. Accordingly, the compute partitioning coordinator A262 may dynamically allocate processing resources in response to changes in a dynamic environment (e.g., medical environment).

[0168] In some embodiments the compute partitioning coordinator A262 may also determine the task schedule based on an applicable timeframe indicating when the compute tasks / sub-tasks may need to be completed and / or which tasks / sub-tasks must be completed by particular timelines. For example, an image processing task for one or more sub-frame chunks of a region of interest in a medical procedure may have a higher task prioritization level than other processing tasks associated with the medical procedure. Thus, the coordinator A262 may generate a task schedule that allocates more compute resources (e.g., GPU cores A268-A270) and / or skips the sub-frame chunks ahead of other tasks (upon receipt of the chunks) in the schedule to ensure that the processing tasks associated with the chunks are completed in accordance with the applicable timeframe indicated in the task schedule.

[0169] Generally, the compute partitioning coordinator A262 may implement efficient, deterministic, and reliable scheduling of computational tasks on, e.g., the GPU A264 by generating the task schedules to prioritize tasks based on their urgency and deadlines (e.g., by incorporating medical context). This ensures that high-priority processing tasks, such as analyzing diagnostic images or monitoring patient vitals in real-time, are given precedence and completed within their required time frames. To optimize GPU A264 utilization and reduce overhead, the coordinator A262 may utilize adaptive batching techniques by dynamically grouping smaller tasks / sub-tasks into batches for simultaneous execution on the GPU A264. The coordinator A262 may adjust the batch size based on current workload and task prioritization levels to maintain real-time performance without sacrificing efficiency.

[0170] For instances that include kernels within the GPU A264, the compute partitioning coordinator A262 may utilize CUDA Graphs to define and execute a collection of CUDA operations as a single entity. By predefining the execution graph of related tasks, the coordinator A262 may reduce the overhead of launching individual tasks and ensure more predictable and streamlined execution on the GPU A264. The compute partitioning coordinator A262 may also evaluate kernels within the GPU A264 to determine the priority of the kernels (e.g., CUDA kernels) or processing tasks based on their importance and urgency in a medical data processing pipeline. For example, tasks associated with real-time diagnosis or surgical assistance may be assigned higher task prioritization and may accordingly receive more prioritized kernel allocation. The coordinator A262 may allocate GPU A264 resources, such as streaming multiprocessors, one or more threads of one or more cores A268-A270, and / or memory, to different CUDA kernels and / or tasks based on their task prioritization and task resource (e.g., processing resource) requirements.

[0171] To achieve compliance with the processing completion timeframes indicated in the task schedule, the compute partitioning coordinator A262 may implement one or more scheduling algorithms that may guarantee completion of processing tasks within specific time frames. This may involve the coordinator A262 establishing deadlines for task completion and dynamically adjusting resource allocation and / or task execution order to meet these deadlines, ensuring that the processing tasks are completed in real-time or near-real-time, as required for the medical procedure associated with the processing task(s). For example, the coordinator A262 may utilize a scheduling algorithm that includes partitioning tasks into segments with finite runtimes, saving state between executions, and / or systematically launching tasks according to their defined task prioritizations and deadlines. Moreover, the coordinator A262 may perform real-time monitoring of task execution and resource utilization to make adjustments on-the-flyAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WC PATENT APPLICATIONto address any bottlenecks or delays, thereby maintaining compliance with the task completion deadlines indicated in the task schedule.

[0172] The coordinator A262 may also implement various strategies / techniques to maintain overall system efficiency while satisfying the task schedule timel i nes / deadl ines. For example, the coordinator A262 may minimize idle GPU resources (e.g., cores A268-A270), reducing context switching overhead, and / or optimizing data transfer between the CPU A263 and the GPU A264 to prevent bottlenecks. The compute partitioning coordinator A262 may also continuously monitor the execution of CUDA kernels (or other kernels) within the GPU A264 and the performance of the processing pipelines to adjust task prioritizations, reallocate resources, and / or modify the task schedule to address any delays or inefficiencies that arise (e.g., based on real-time performance data). Further, if the coordinator A262 determines that one or more tasks may not complete within the designated time frame (e.g., in the task schedule), the coordinator A262 may implement one or more fallback strategies, such as temporarily reducing the quality and / or resolution of processing to expedite completion or offloading tasks to additional GPUs (not shown), the CPU A263, and / or other GPU cores A268-A270, if available.

[0173] In certain embodiments, the second node A243 may be an edge compute node that is connected to a cloud-based application server (not shown). The edge compute node A243 may receive compute tasks from the cloud-based application server to execute as part of one or more of the applications (e.g., A257-A261) accessible by the compute node. The compute partitioning coordinator A262 may schedule at least one task from one of the applications A257-A261 for execution by the cloud-based application server.

[0174] In certain embodiments, the second node A243 may further include one or more FPGAs (not shown), and the compute partitioning coordinator A262 may further partition one or more high-priority tasks for execution on one or more cores A268-A270 of the GPU A264 or the one or more FPGAs.

[0175] Moreover, in certain embodiments, the compute partitioning coordinator A262 may consolidate computation performed on the second node A243 onto fewer, larger GPUs (e.g., GPU A264) to optimize resource utilization and performance.

[0176] FIG. 2D depicts an example distributed computing environment A271 in which applications are made conditionally available from the cloud computing environment in accordance with the compute resources available at the local computing environments, in accordance with various embodiments described herein. The example distributed computing environment A271 generally includes a first local computing environment A273a, a second local computing environment A273b, a third local computing environment A273c, and a cloud computing environment A279 (also referenced herein as a “cloud-based application server”) that are communicatively coupled to each other. Each of the local computing environments A273a, A273b, and A273c may access the cloud computing environment A279 to utilize one of the applications A279a stored as part of the environment A279. The processing environment analysis module A279b may evaluate (e.g., upon receiving a request from a local environment) the processing capacity and / or hardware configurations of these local environments A273a, A273b, and A273c to determine whether each environment A273a, A273b, and A273c may receive access to respective applications A279a on the cloud computing environment A279. Each of the local computing environments A273a, A273b, A273c may also be referenced herein as a “compute node” or a “node”.

[0177] The local computing environments A273a, A273b, and A273c may include a CPU A274a, A274b, and A274c, a GPU A275a, A275b, and A275c, an FPGA A276a, A276b, and A276c, and / or a memory A277a, A277b, and A277c. Each of the CPUs A274a, A274b, and A274c may include one or more cores, such as Core 1 through Core N or Core X, and each of the GPUs A275a, A275b, and A275c may include one or more cores, such as Core 1 through Core M or Core Y, where N, M, X, and Y may be any suitable integer values (e.g., 1, 2, 4, 6, 10, 64, 100, 1000, etc.). Further, each of these cores may have any suitable numberAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONof executable threads to increase the number of tasks performed by each core at any given time. Each of the processing units (e.g., CPUs, GPUs, FPGAs) illustrated in FIG. 2D may be configured to execute software instructions stored in each of the corresponding memories A277a, A277b, and A277c. The memories A277a, A277b, and A277c may each include one or more persistent memories (e.g., a hard drive and / or solid-state memory) and may store one or more applications, modules, and / or models, such as a compute partitioning coordinator A278a, A278b, and A278c, as described herein.

[0178] The cloud computing environment A279 generally include / host the applications A279a and the processing environment analysis module A279b. The applications A279a may generally include computer-executable instructions that cause the cloud computing environment A279 and / or one or more of the local computing environments A273a, A273b, and A273c to perform one or more tasks / sub-tasks in association with the applications A279a. Each application A279a may include different sets of tasks and sub-tasks, such as a machine learning application that may involve data preprocessing tasks, model training tasks, and / or inference tasks. Each of these tasks may be further divided into sub-tasks, such as data cleaning, feature extraction, model evaluation, and the like. Further, the tasks / sub-tasks may have different processing demands based on, e.g., its complexity and the volume of data it handles. Data preprocessing might require significant memory and I / O bandwidth for handling large datasets, while model training might demand extensive computational power for processing complex algorithms over extended periods. For example, when analyzing image data captured by an endoscope (e.g., endoscope 202, 222), a machine learning model included as part of an application A279a may include image preprocessing, feature extraction, segmentation, and / or 3D reconstruction. These tasks may require significant computational power for processing large image datasets, real-time analysis capabilities for diagnostic purposes, and / or advanced algorithms for accurate image interpretation.

[0179] The cloud computing environment A279 may receive requests from one or more of the local computing environments A273a, A273b, and A273c to access one or more of the applications A279a, and the processing environment analysis module A279b may connect to the one or more local computing environments A273a, A273b, and A273c (e.g., also referenced as “compute nodes”) to evaluate their respective capabilities to execute the one or more requested applications A279a. Each of the local computing environment A273a, A273b, and A273c may have different computing resources (e.g., the third local computing environment A283c may not have an FPGA A276c), and thus different capacities to execute various applications A279a that have greater / lesser processing demands. Moreover, each local computing environment A273a, A273b, and A273c may perform different functions (e.g., surgical procedure assistance, diagnostics), such that the local computing environments A273a, A273b, and A273c may not all require access to the same applications A279a.

[0180] In any event, the processing environment analysis module A279b may further determine a respective processing environment configuration for one or more of the local computing environments A273a, A273b, and A273c indicating a compute processing capacity of the one or more local computing environments A273a, A273b, and A273c, and a compute processing hardware configuration of the local computing environments A273a, A273b, and A273c. The compute processing capacity may generally indicate the total amount of computational power available within each of these environments A273a, A273b, and A273c for executing tasks and processing data. The module A279b may determine this capacity based on several factors, such as the number and type of processors (e.g., CPUs, GPUs, FPGAs), the speed and architecture of these processors, the amount of random-access memory (RAM), and / or the availability of specialized hardware accelerators or co-processors designed for specific tasks (e.g., machine learning or image processing) of the one or more local computing environments A273a, A273b, and A273c. For example, the first local computing environment A273a may include a high-performance CPU A274a featuring multiple cores for parallel processing, a powerful GPU A275a for handling computationally intensive tasks such as medical image analysis, and ample RAM to support large datasets and complex computations. By contrast, the second local computing environment A273bAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONmay not include such high-performance processors, and may thus not have as much total processing / computational power as the first local computing environment A273a.

[0181] The compute processing hardware configuration may generally indicate the specific arrangement and specifications of the hardware components within the one or more local computing environments A273a, A273b, and A273c. For example, the module A279b may analyze the type, number of cores, and / or clock speed of the CPUs A274a, A274b, and A274c, which may impact the overall processing speed and multitasking capabilities of the respective local computing environments A273a, A273b, and A273c. The module A279b may also analyze the model, number of cores, and / or memory capacity of the GPUs A275a, A275b, and A275c, which may generally indicate their capacity to handle tasks / sub-tasks requiring parallel processing. The module A279b may also analyze the total amount of RAM available in each environment A273a, A273b, and A273c, along with its speed, which may impact their ability to handle large datasets and run multiple applications simultaneously. Further, the module A279b may analyze the type (e.g., SSD, HDD) and capacity of storage solutions (e.g., in memory A277a, A277b, and A277c), to determine how much data may be stored locally and how quickly it may be accessed.

[0182] Moreover, the processing environment analysis module A279b may compare the respective processing environment configurations of the local computing environments A273a, A273b, and A273c to application processing configuration requirements (e.g., minimum required processor capabilities, dedicated RAM, available storage space in memory) for the applications A279a. Based on this comparison, the processing environment analysis module A279b may provision / deny access to an application A279a for one or more of the local computing environments A273a, A273b, and A273c based on their respective compute processing capacity and compute processing hardware configuration satisfying or failing to satisfy the processing configuration requirements for the application A279a.

[0183] In this manner, the processing environment analysis module A279b may be configured to selectively offer different services (e.g., via the applications A279a) to each of the local computing environments A273a, A273b, and A273c based on the determined configurations of the local computing environments A273a, A273b, and A273c. For example, the second local computing environment A273b may have processing capacity and configuration sufficient to support the complete set of tasks / sub-tasks associated with a first application of the applications A279a. By contrast, the third local computing environment A273c may not have sufficient processing capacity and / or configuration to support the complete set of tasks / sub-tasks associated with the first application, and instead, may only have the capacity / configuration sufficient to support a portion the tasks / sub-tasks, maybe unable to handle all tasks / sub-tasks by the requisite deadline, and / or may otherwise be unable to handle the complete set of tasks / sub-tasks. Accordingly, the processing environment analysis module A279b may provision access to the first application from the application A279a for the second local computing environment A273b while the module A279b may deny access to the first application for the third local computing environment A273c.

[0184] In certain embodiments, the module A279b may provide distributed access to a particular application from the applications A279a to, for example, supplement a lack of necessary processing capacity and / or configuration within a single local computing environment and / or to expedite certain tasks / sub-tasks of an application A279a. For example, the module A279b may assign a first set of tasks (e.g., high-priority medical image processing tasks) to the first local computing environment A273a that has one or more high throughput GPUs A275a and may assign a second set of tasks (e.g., low-priority processing tasks) to the second local computing environment A273b that does not have such high throughput GPUs A275a, and instead has one or more lower quality GPUs A275b and / or CPUs A274b.

[0185] Additionally, or alternatively, the module A279b may interact with the compute partitioning coordinators A278a, A278b, and A278c to optimally designate processing resources for execution of the application A279a tasks / sub-tasks. AsAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONdiscussed herein, each compute partitioning coordinator A278a, A278b, and A278c may be configured to prioritize tasks between compute processing hardware (e.g., GPUs, CPUs, FPGAs) of the local computing environments A273a, A273b, and A273c based on application (e.g., local applications or applications A279a) processing configuration requirements and / or real-time processing needs. The module A279b may, for example, receive a task schedule from the coordinator A278a, A278b, and A278c to analyze how the processing resources are currently utilized, and to make subsequent determinations regarding how such resources could / should be utilized to manage the tasks / sub-tasks of an application A279a to which the local computing environment A273a, A273b, A273c may be provisioned access. The module A279b may provision access to a local computing environment A273a, A273b, A273c and may transmit instructions to the corresponding coordinator A277a, A277b, A277c to partition the processing resources accordingly and / or update / generate a task schedule that includes the tasks / sub-tasks associated with the application A279a to which the local computing environment A273a, A273b, A273c has been provisioned access.

[0186] The module A279b may perform this processing / memory / hardware resource analysis for the local computing environments A273a, A273b, and A273c utilizing various architectures or implementations. For example, the module A279b may leverage tools provided by the local operating systems (e.g., Ispci on Linux for listing connected devices) and / or vendor-supplied tools (e.g., nvidia-smi, nvml-api) to obtain detailed information about the local computing environments A273a, A273b, A273c.

[0187] Additionally, or alternatively, the module A279b may employ policy-driven resource allocation, where predefined policies based on service level agreements (SLAs), priority levels, and / or time-based constraints guide the allocation / provisioning decisions. The module A279b may utilize these policies to determine where and when to allocate / provision resources, ensuring that hardware resources (e.g., at the local computing environments A273a, A273b, and A273c and / or across multiple computing environments) are optimally distributed among tasks / sub-tasks and applications A279a to meet performance requirements and optimize efficiency.

[0188] The module A279b may additionally or alternatively utilize an automated placement engine (not shown) to allocate / provision workloads and / or access to the most suitable resources in the local computing environments A273a, A273b, A273c based on defined policies and real-time load conditions. Generally, the automated placement engine may allocate processing workloads to any of the individual local computing environments A273a, A273b, A273c, to multiple of the local environments (e.g., A273a and A273b), to a local computing environment and a cloud-based computing environment / resource (e.g., A273c and A279), and / or any other suitable combination of local / cloud-based resources. More specifically, the module A279b may provision access to the applications A279a, for example, in medical imaging, based on the type of processing resources available at a local computing environment (e.g., GPUs A275a, A275b, A275c) and its functional requirements. The engine may thus determine how many resources to allocate at a particular time, taking into account the current demand and availability of resources.

[0189] The module A279b may dynamically allocate resources within a local environment A273a, A273b, A273c and / or across multiple environments to accomplish one or more tasks / sub-tasks associated with one or more applications A279a to which the module A279b may have provisioned access. For example, the module A279b may determine that the first local computing environment A273a should be provisioned access to a first application from the applications A279a, but that the first environment A273a may ideally receive additional processing capacity from a cloud-based resource (e.g., the cloud computing environment A279) to perform a low-priority task of the first application. Thus, the module A279b may utilize an automated placement engine to allocate the additional workload of the low-priority task to the cloud computing environment A279.

[0190] Additionally, or alternatively, the processing environment analysis module A279b may include monitoring tools to continuously observe the utilization and performance of hardware resources at the local computing environments A273a, A273b,Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONA273c. The module A279b may further implement predictive algorithms, such as time series forecasting, to anticipate future demand and pre-allocate resources accordingly (e.g., at one or more local computing environments A273a, A273b, and A273c and / or across multiple computing environments). This proactive approach allows the module A279b to adjust resource allocation dynamically, ensuring that sufficient resources are available to handle expected workloads.

[0191] Thus, in certain embodiments, the module A279b may cause the cloud computing environment A279 to dynamically adjust provisioning of the one or more applications A279a to respective local computing environments A273a, A273b, A273c and processing task allocations based on (i) updated application processing configuration requirements of the provisioned applications, (ii) updated compute processing capacity of one or more of the local computing environments A273a, A273b, A273c, and / or (iii) updated compute hardware configurations of one or more of the local computing environments A273a, A273b, A273c.

[0192] As mentioned, the processing environment analysis module A279b may determine that one or more processing tasks should be offloaded to other available processing resources that may not be included as part of a local computing environment (e.g., A273a, A273b, A273c) to which application (e.g., A279a) access was provisioned. Thus, the module A279b may cause the cloud computing environment A279 to offload processing tasks of the one or more applications A279a to a cloud or hybrid cloud environment (e.g., within the environment A279 and / or another cloud-based environment (not shown)) based on processing demands and existing hardware capabilities of the local computing environment(s).

[0193] As part of provisioning tasks / sub-tasks to multiple computing environments (e.g., local and cloud-based), the module A279b may be configured to ensure that the data transmitted between the different environments is secure and private. In certain embodiments, one or more of the set of applications A279a provisioned by the cloud computing environment A279 (e.g., via the processing environment analysis module A279b) may be configured to assist as part of a medical / surgical procedure, which may require significant data security / privacy controls. Thus, the module A279b may be configured to maintain the data security / privacy of medical data (e.g., medical imaging data) transmitted to / from the applications A279a to ensure that sensitive patient information is not inadvertently exposed or otherwise leaked.

[0194] In particular, the module A279b may implement a zero-trust architecture (ZTA) which may operate on the principle that no entity, whether inside or outside the network (e.g., network A272), is trusted by default. The module A279b may rigorously authenticate, authorize, and / or encrypt every access request to the relevant environments (e.g., A273a, A273b, A273c, A279). This ZTA approach may thereby minimize the risk of unauthorized access and data breaches. The module A279b may encrypt data both in transit (e.g., across the network A272) and / or at rest (e.g., stored in an environment A273a, A273b, A273c, A279) to protect the data against interception and / or unauthorized access. The module A279b may utilize, for example, techniques like transport layer security (TLS) for data in transit and may utilize encryption standards such as AES-256 and the like for data at rest.

[0195] As one example, the module A279b may implement data de-identification for sensitive medical imaging data to be transmitted across the network A272 and / or stored within a computing environment A273a, A273b, A273c, A279. Before sending the medical imaging data to the cloud, the module A279b may utilize, e.g., pseudonymization or tokenization to remove or replace personally identifiable information (PH) and protected health information (PHI) with non-identifiable placeholders, preserving patient privacy while allowing data to be processed.

[0196] In another example, the module A279b may utilizes blockchain or other ledger technologies to create an immutable record of data access, modification, and processing activities. This ensures transparency and accountability, providing a verifiable trail of data handling that can be audited for compliance and security purposes.Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION

[0197] In yet another example, the module A279b may utilize one or more machine learning algorithms to automatically blur or redact PHI / PII in captured images, recorded videos, and / or speech-to-text conversions. The module A279b may leverage such models / algorithms to obscure faces in images, mute or “beep” out sensitive information in recorded speech, and / or delete or redact text containing PHI / PII in case notes. Moreover, in certain embodiments, the applications A279a may include one or more machine learning algorithms and the cloud computing environment A279 (e.g., via the processing environment analysis module A279b) may be configured to determine corresponding processing resource requirements and / or predicted real-time processing needs of the tasks / sub-tasks associated with the machine learning algorithms.

[0198] In still another example, the module A279b may implement hybrid processing approaches, where sensitive data is initially processed locally (e.g., at local computing environments A273a, A273b, A273c) to extract features or perform preliminary analysis. The module A279b may then instruct the environments to transfer only the resulting feature vectors, which contain no direct PHI / PII, to the cloud computing environment A279 (and / or other cloud-based processing environment) for further processing. This approach minimizes the amount of sensitive data leaving the local environments, reducing privacy risks.

[0199] Additionally, or alternatively, the module A279b may ensure compliance with other relevant rules and / or regulations that may apply to the applications A279a and / or any data associated with the applications A279a. For example, in medical applications, several sets of applicable rules / regulations may apply to data transmission in cloud and / or hybrid processing environments. Thus, the module A279b may ensure regulatory compliance for the applications A279a running on the cloud and / or hybrid environments (e.g., A273a, A273b, A273c, A279), especially for intraoperative use cases, and may involve a comprehensive approach focusing on data security, system performance, change management, interoperability, risk management, and / or emergency recovery.

[0200] As an example, the module A279b may ensures compliance with relevant policies / regulations (e.g., HIPAA, GDPR), and / or other data privacy regulations by implementing end-to-end encryption for data in transit and at rest (as discussed), controlled access mechanisms, and / or regular audits. The module A279b may also implement / establish strong identity management, logging, and / or tracking systems to monitor data handling in real-time environments securely. To comply with standards like IEC 62304, which covers medical software lifecycle processes, the module A279b may validate system reliability and low latency to help ensure intraoperative performance. This may involve the module A279b leveraging edge computing components to minimize latency and ensure that performance SLAs are met.

[0201] In some embodiments, the module A279b may also implement privacy / regulatory controls that ensure compliance with policies and / or regulations associated with data protection (e.g., privacy, security) that are associated with any suitable geographic region / locality and / or entity. For example, the module A279b may identify the specific data privacy regimes, policies, and / or regulations applicable to the locality where the data is being processed, whether such policies / regulations are based on geographic regions and / or entity-specific guidelines (e.g., hospital-specific requirements). The module A279b may then implement mechanisms to restrict the storage, processing, and / or transfer of data to data centers or computing resources located within the defined locality, ensuring that all operations adhere to the local data privacy standards. This could involve, e.g., dynamically routing data to appropriate local servers, encrypting data in transit, and / or applying access controls based on the locality's requirements, thereby maintaining compliance with the relevant data privacy / security regulations / policies.

[0202] As another example, the module A279b may implement a robust continuous integration / continuous deployment (CI / CD) pipeline to manage infrastructure and / or software updates. The module A279b may perform and / or otherwise leverage such a CI / CD pipeline to include testing, validation, and / or rollback protocols to ensure compliance with other relevant regulations (e.g., ISO 13485) and / or FDA guidance, with continuous monitoring in place to maintain system functionality and safety. Further,Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONthe module A279b may ensure that the applications A279a adhere to HL7, DICOM, and / or other interoperability standards for seamless data transfer during intraoperative use and to thereby support effective communication between local environments (e.g., A273a, A273b, A273c) and cloud environments (e.g., A279), facilitating the integration of medical imaging data with other clinical systems.

[0203] In yet another example, the module A279b may implement proactive risk assessment and mitigation strategies to comply with ISO 14971, focusing on risk management in medical devices. The module A279b may include instructions and / or other algorithms configured to anticipate and manage potential failure points that could impact patient outcomes, ensuring that risks are identified and addressed promptly. The module A279b may also implement emergency recovery plans, such as regular backups, disaster recovery, and / or failover mechanisms to guarantee system reliability and availability in case of infrastructure failure. These measures implemented by the module A279b may generally satisfy regulatory requirements for patient safety in intraoperative settings, ensuring that medical applications (e.g., one or more of the applications A279a) can continue to operate effectively even under adverse conditions.

[0204] In certain embodiments, the cloud computing environment A279 may also provide access to other computing nodes / environments (not shown) that may provide additional functionality, applications, hardware, and / or other components that may contribute to the resources available as part of the distributed computing environment A271. This accessibility may enable the distributed computing environment A271 to facilitate scalability and collaboration with external entities that may, for example, receive access to lower-level functionalities of the environment A271, which may be managed and / or otherwise influenced by the module A279b.

[0205] For example, the module A279b may generally enable entities (e.g., product / hardware vendors) to access lower-level hardware functionalities, such as for optimizing data transfers, minimizing latency, and / or achieving real-time processing. The module A279b may provide vendors with access to gain direct control over memory operations, PCIe configurations, and / or vendorspecific operations like atomic operations and cache management in certain portions of the environment A279. This access enables the cloud computing environment A279 to customize data paths, fine-tune transfer characteristics, and / or manage concurrency in multi-device environments, to enhance the performance of medical imaging applications (e.g., as part of the set of applications A279).

[0206] As another example, the module A279b may facilitate scalability within the cloud computing environment A279 to ensure the system remains scalable and adaptable to future hardware and software advancements. The module A279b may implement a modular and / or layered architecture, which may allow for easy integration of new hardware components or software updates without significant overhauls. The module A279b may also utilize various APIs and / or hardware abstraction layers (HAL) to facilitate communications between the cloud computing environment A279 and external compute resources (e.g., A273a, A273b, A273c), enabling the environment A279 to leverage additional computational power from connected devices (e.g., a 4th cart per system and / or a dedicated compute box in an operating room). Moreover, as discussed herein, the module A279b may implement dynamic resource allocation and / or virtualization techniques to efficiently distribute workloads based on available resources. This may include modeling multiple discrete devices as a single larger device and / or automatically partitioning work to take advantage of newer, higher-density devices.

[0207] Although not shown in FIG. 2D, it should be understood that each local computing environment A273a, A273b, and A273c may include a networking interface to enable the local computing environments A273a, A273b, and A273c to communicate with the cloud computing environment A279. More specifically, such networking interfaces may enable the local computing environments A273a, A273b, and A273c to communicate with each component of the cloud computing environment A279 acrossAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONa network A272 through their respective networking interfaces. The networking interfaces may support one or more of the communication / network protocols implemented by the network A272. Moreover, the network A272 may be a single communication network, or may include multiple communication networks of one or more types (e.g., one or more wired and / or PANs or LANs, and / or one or more WANs such as the Internet). In some embodiments, the network A272 includes multiple, entirely distinct networks.

[0208] FIG. 2E depicts a flow diagram representing an example computer-implemented method A280 for optimizing intraprocessor data transfer and processing, in accordance with various embodiments described herein. Generally, the method A280 may be implemented by one or more processors of the various computing devices / environments described herein, such as the GPU A236 and / or the FPGA A234 of the downstream nodes A225, for example.

[0209] The method A280 includes receiving a sequence of sub-frame chunks of a frame captured by a sensor device, each sub-frame chunk of the sequence of sub-frame chunks representing a stripe / line of the frame (block A281). The method A280 may further include determining one or more tasks associated with at least one application of one or more applications to which the sequence of sub-frame chunks correspond (block A282). The method A280 may further include causing at least one GPU or at least one other programmable logic device to analyze one or more sub-frame chunks of the sequence of sub-frame chunks based on the one or more tasks (block A283).

[0210] The method A280 may further include iteratively compiling the frame based on sequential analysis of the at least one GPU or the at least one other programmable logic device of the one or more sub-frame chunks (block A284). The method A280 may further include determining an outcome of the one or more tasks during the iterative compiling of the frame (block A285). The outcome may be associated with operation of a computer-assisted system, such as a robotic medical / surgical system.

[0211] In certain embodiments, the method A280 further includes determining an expected position of a feature of interest within the frames captured by the sensor device, the feature of interest to be analyzed as part of an image processing algorithm that is a task of the one or more tasks; chunking one or more subsequent captured frames to create one or more priority chunks that include portions of the subsequent captured frames corresponding to the expected position of the feature of interest; determining the image processing algorithm associated with the one or more priority chunks; and causing the at least one GPU or the at least one other programmable logic device to analyze the one or more priority chunks based on the image processing algorithm and the expected position of the feature of interest.

[0212] In certain embodiments, the method A280 further includes receiving data corresponding to at least one of (I) a model of lumens for an endoscope or (II) kinematic data of the endoscope; and determining the expected position of the feature of interest based on at least one of (I) the model of lumens or (II) the kinematic data.

[0213] In certain embodiments, the method A280 further includes analyzing one or more sub-frame chunks by: determining one or more processing requirements of the one or more tasks and processing requirements of the at least one application; and allocating the one or more tasks between the at least one GPU and the at least one other programmable logic device based on respective processing capabilities of the at least one GPU and the at least one other programmable logic device compared to the one or more processing requirements of the one or more tasks and the at least one application.

[0214] In certain embodiments, the one or more processing requirements may correspond to a task of a medical procedure or a phase of the medical procedure.

[0215] In certain embodiments, the sensor device is a medical device, and the frame and the sequence of sub-frame chunks are medical images.Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION

[0216] In certain embodiments, the method A280 further includes extending margins of each sub-frame chunk to include a margin of pixels from adjacent portions included in the sequence of sub-frame chunks; and merging extended margins of adjacent sub-frame chunks during the iterative compiling of the frame to avoid duplications or inconsistencies within the frame.

[0217] In certain embodiments, the method A280 further includes modifying one or more processing parameters for an analysis algorithm executed by the at least one GPU or the at least one other programmable logic device near boundaries of a subframe chunk; and causing the at least one GPU or the at least one other programmable logic device to analyze the sub-frame chunk in accordance with the one or more modified processing parameters for the analysis algorithm.

[0218] In certain embodiments, the outcome of the one or more tasks is displayed on a collaboration interface for enabling collaboration with one or more external entities to optimize task scheduling and data transfer.

[0219] In certain embodiments, the compute node is communicatively coupled with a cloud-based application server.

[0220] Of course, it is to be appreciated that the actions of the method A280 may be performed any suitable number of times, and that the actions described in reference to the method A280 may be performed in any suitable order.

[0221] FIG. 2F depicts a flow diagram representing an example computer-implemented method A286 for optimizing compute resource management, in accordance with various embodiments described herein. Generally, the method A286 may be implemented by one or more processors of the of the various computing devices / environments described herein, such as the CPU A263 and / or the GPU A264 of the second node A243, for example.

[0222] The method A286 may include determining, by executing a compute partitioning coordinator, (i) a set of tasks for one or more applications to be executed by a compute node, (ii) a task prioritization level for each task of the set of tasks, and (ill) compute resource requirements for each task of the set of tasks (block A287). The method A286 may further include partition, based on (i)-(iii), the set of tasks into a task schedule that specifies a respective processing core of a CPU or a GPU of the compute node to perform execution of one or more of tasks of the set of tasks (block A288). The method A286 may further include causing the GPU and the CPU to execute the partitioned tasks in accordance with the task schedule (block A289).

[0223] In certain embodiments, the compute node comprises at least one edge compute node, the application server is a cloud server, and the compute node is connected to the cloud server.

[0224] In certain embodiments, the compute node receives compute tasks from the cloud-based application server to execute as part of one or more of the applications accessible by the compute node.

[0225] In certain embodiments, the compute partitioning coordinator schedules at least one task for execution by the application server.

[0226] In certain embodiments, the method A286 may further include determining the task schedule for each application based on the task schedule for every other application to be executed by the compute node.

[0227] In certain embodiments, the compute node may further comprise one or more FPGAs, and the method A286 may further include determining one or more high priority tasks based on the task prioritization level of one or more tasks in the set of tasks; and partitioning the one or more high priority tasks for execution on the core of the GPU or the one or more FPGAs.

[0228] In certain embodiments, the method A286 may further include partitioning the set of tasks based on (i)-(iii) and consolidating computation onto one or more particular GPUs that each satisfy a processing threshold.Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION

[0229] In certain embodiments, the method A286 may further include creating one or more virtual sandboxes within the GPUs, each sandbox isolating a respective task to minimize negative interactions between concurrent tasks.

[0230] In certain embodiments, at least one virtual sandbox of the one or more virtual sandboxes corresponds to performing operations corresponding to a medical procedure requiring high timing consistency.

[0231] In certain embodiments, at least one of the tasks is executed as part of a kernel across one or more threads of the one or more GPU cores.

[0232] In certain embodiments, the method A286 may further include dynamically adjusting allocation of GPU resources based on the task prioritization levels and the compute resource requirements.

[0233] In certain embodiments, at least one application of the one or more applications corresponds to a medical procedure, and the set of tasks correspond to processing tasks of medical image data acquired by an endoscope.

[0234] In certain embodiments, the compute node receives a sequence of sub-frame chunks of a frame captured by a sensor device, each sub-frame chunk of the sequence of sub-frame chunks representing a portion of the frame, and the method A286 may further include determining (i)-(iii) associated with the sequence of sub-frame chunks; partitioning the set of tasks associated with the sequence of sub-frame chunks into the task schedule; and causing the GPU or at least one other programmable logic device to execute the partitioned tasks in accordance with the task schedule.

[0235] In certain embodiments, the method A286 may further include iteratively compiling the frame based on sequential analysis of the at least one GPU or the at least one other programmable logic device of the sub-frame chunks; and determining an outcome of the one or more tasks during the iterative compiling of the frame, wherein the outcome is associated with operation of a computer-assisted system.

[0236] In certain embodiments, the method A286 may further include determining an expected position of a feature of interest within the frames captured by the sensor device, the feature of interest to be analyzed as part of an image processing algorithm that is a task of the one or more tasks; chunking one or more subsequent captured frames to create one or more priority chunks that include portions of the subsequent captured frames corresponding to the expected position of the feature of interest; determining the image processing algorithm associated with the one or more priority chunks; and causing the at least one GPU or the at least one other programmable logic device to analyze the one or more priority chunks based on the image processing algorithm and the expected position of the feature of interest.

[0237] In certain embodiments, a first compute node and a second compute node request access to a first application of the one or more applications, and the method A286 may further include determining a respective processing environment configuration for the first compute node and the second compute node indicating (i) a first compute processing capacity of the first compute node, (ii) a second compute processing capacity of the second compute node, (iii) a first compute processing hardware configuration of the first compute node, and (iv) a second compute processing hardware configuration of the second compute node; and comparing the respective processing environment configurations of the first compute node and the second compute node to application processing configuration requirements for the first application, to provision access for the first compute node or the second compute node.

[0238] In certain embodiments, the method A286 may further include provisioning access to the first application for the first compute node based on the first compute processing capacity and the first compute processing hardware configuration satisfying the processing configuration requirements for the first application; and denying access to the first application for the secondAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONcompute node based on the second compute processing capacity or the second compute processing hardware configuration failing to satisfy the processing configuration requirements for the first application.

[0239] In certain embodiments, the method A286 may further include selectively offering different services based on the determined configuration of the plurality of compute nodes.

[0240] In certain embodiments, the method A286 may further include partitioning processing power of the compute nodes into multiple virtual sandboxes to isolate respective tasks and prevent negative interactions between different tasks.

[0241] In certain embodiments, a first application of the one or more applications may include a machine learning algorithm, a second application of the one or more applications may include an image processing algorithm, and the task prioritization level for tasks of the image processing algorithm may include real-time processing needs.

[0242] Of course, it is to be appreciated that the actions of the method A286 may be performed any suitable number of times, and that the actions described in reference to the method A286 may be performed in any suitable order.

[0243] FIG. 2G depicts a flow diagram representing an example computer-implemented method A290 for hardware-aware application management, in accordance with various embodiments described herein. Generally, the method A290 may be implemented by one or more processors of the of the various computing devices / environments described herein, such as one or more processors of the cloud computing environment A279, for example.

[0244] The method A290 includes connecting to a plurality of compute nodes requesting access to the one or more applications, wherein a first compute node and a second compute node request access to a first application of the one or more applications (block A291). The method A290 may further include determining a respective processing environment configuration for the first compute node and the second compute node indicating (i) a first compute processing capacity of the first compute node, (ii) a second compute processing capacity of the second compute node, (iii) a first compute processing hardware configuration of the first compute node, and (iv) a second compute processing hardware configuration of the second compute node (block A292).

[0245] The method A290 may further include comparing the respective processing environment configurations of the first compute node and the second compute node to application processing configuration requirements for the first application (block A293). The method A290 may further include performing at least one of: provisioning access to the first application for the first compute node based on the first compute processing capacity and the first compute processing hardware configuration satisfying the processing configuration requirements for the first application (block A294), and / or denying access to the first application for the second compute node based on the second compute processing capacity or the second compute processing hardware configuration failing to satisfy the processing configuration requirements for the first application (block A295).

[0246] In certain embodiments, the method A290 may further include selectively offering different services based on the determined configuration of the plurality of compute nodes.

[0247] In certain embodiments, one or more compute nodes of the plurality of compute nodes includes at least one of an FPGA and a GPU.

[0248] In certain embodiments, the method A290 may further include prioritizing tasks between compute processing hardware of the plurality of compute nodes based on application processing configuration requirements and real-time processing needs.Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION

[0249] In certain embodiments, the applications include one or more machine learning algorithms and the method A290 may further include determining (i) tasks of the machine learning algorithms, (ii) corresponding processing resource requirements of the tasks, or (iii) predicted real-time processing needs of the tasks; and wherein the application processing configuration requirements correspond to minimum hardware requirements to perform the tasks of the machine learning algorithm.

[0250] In certain embodiments, the method A290 may further include partitioning processing power of the compute nodes into multiple virtual sandboxes to isolate respective tasks and prevent negative interactions between different processing tasks.

[0251] In certain embodiments, the method A290 may further include offloading processing tasks of the one or more applications to a cloud environment or a hybrid cloud environment based on processing demands and existing hardware capabilities of the compute nodes.

[0252] In certain embodiments, the method A290 may further include dynamically adjusting provisioning of the one or more applications to respective compute nodes and processing task allocations based on (i) updated application processing configuration requirements, (ii) updated compute processing capacity of a compute node of the one or more compute nodes, or (iii) updated compute hardware configurations of the compute node of the one or more compute nodes.

[0253] In certain embodiments, the one or more applications are configured to assist as part of a medical procedure, and the method A290 may further include maintaining data security and data privacy of data transmitted to applications hosted on the server.

[0254] In certain embodiments, the method A290 may further include receiving a sequence of sub-frame chunks of a frame captured by a sensor device, each sub-frame chunk of the sequence of sub-frame chunks representing a portion of the frame; determining one or more tasks associated with at least one application of the one or more applications to which the sequence of sub-frame chunks correspond; causing at least one GPU or at least one other programmable logic device to analyze one or more sub-frame chunks of the sequence of sub-frame chunks based on the one or more tasks; iteratively compiling the frame based on sequential analysis of the at least one GPU or the at least one other programmable logic device of the one or more sub-frame chunks; and determining an outcome of the one or more tasks during the iterative compiling of the frame, wherein the outcome is associated with operation of a computer-assisted system.

[0255] In certain embodiments, the method A290 may further include determining an expected position of a feature of interest within the frames captured by the sensor device, the feature of interest to be analyzed as part of an image processing algorithm that is a task of the one or more tasks; chunking one or more subsequent captured frames to create one or more priority chunks that include portions of the subsequent captured frames corresponding to the expected position of the feature of interest; determining the image processing algorithm associated with the one or more priority chunks; and causing the at least one GPU or the at least one other programmable logic device to analyze the one or more priority chunks based on the image processing algorithm and the expected position of the feature of interest.

[0256] In certain embodiments, the method A290 may further include determining, by executing a compute partitioning coordinator, (i) a set of tasks for one or more applications to be executed by a compute node, (ii) a task prioritization level for each task of the set of tasks, and (iii) compute resource requirements for each task of the set of tasks; partitioning, based on (i)-(iii), the set of tasks into a task schedule that specifies a respective processing core of a CPU or a GPU of the compute node to perform execution of one or more of tasks of the set of tasks; and causing the GPU and the CPU to execute the partitioned tasks in accordance with the task schedule.Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION

[0257] Of course, it is to be appreciated that the actions of the method A290 may be performed any suitable number of times, and that the actions described in reference to the method A290 may be performed in any suitable order.OMNISCHEDULER

[0258] In addition to the processing unit- level improvements to ensure efficient task execution, the systems and methods described herein may also provide a number of system-level improvements through the use a computing device (such as a data connector 20) configured to manage scheduling for various nodes in an environment such as a hospital. In particular, by activating and / or deactivating nodes based on a predicted likelihood of being used, the instant techniques reduce the power used from such nodes, which may otherwise be constantly on in case of need. For example, in a hospital environment, there may be maximum power draw requirements. By determining precisely when a device will be needed and / or used (e.g., based on procedural details being entered into the system and / or steps of a procedure being performed), the instant techniques may allow for such devices to be temporarily deactivated until needed while still complying with power restrictions. Similar to power restrictions, there may be a cost associated with utilizing various equipment and / or applications (e.g., power consumption, cloud computing IOPS, wages for personnel, etc.). Accordingly, the various efficient equipment management described below may consider these additional factors when orchestrating the various tasks associated with a hospital and / or a procedure.

[0259] Similarly, the processing power / resource usage for systems and devices implementing the instant techniques may similarly be reduced, in that improving and simplifying the infrastructure to a particular node managing scheduling reduces the complex processing power needs from scheduling and redundant determinations.

[0260] Moreover, in some implementations, the instant techniques as described herein may further decrease the resource usage of nodes associated with the computing device. For example, by removing the redundancy in transmitting and / or analyzing data at each individual node, the overall latency and bandwidth requirements are reduced. Furthermore, the instant techniques improve the efficiency of usage by scheduling when the processing occurs to improve overall resource usage across a location and / or system associated with the scheduling node(s) and / or computing device(s).

[0261] It will be understood that such improvements do not constitute an exhaustive list, and other improvements will be clear according to the various examples discussed herein. For example, the above-described processing-unit level improvements may be implemented at different layers in the system architecture to implement efficient task processing throughout the entire processing environment.

[0262] FIG. 3A illustrates a block diagram of an exemplary system B300A for receiving local node data (e.g., data regarding various nodes of a hospital, such as surgical device data) and performing autonomous equipment scheduling and / or task scheduling operations. Depending on the implementation, the exemplary system B300A may be, be part of, or include the data connector 20, cloud 50, etc. of FIGS. 1A-1D, the robotic controller architecture A200 of FIG. 2A, the robotic controller architecture A220 of FIG. 2B, the compute resource partitioning scheme A240 of FIG. 2C, the distributed computing environment of FIG. 2D, and / or any other computing component and / or system as described herein. It should be appreciated that while the term “local” node is used herein, in some embodiments, a “local” node may be located remote from an operating room (e.g., when the instant techniques are implemented in a teleoperation and / or remote guidance system).

[0263] The system B300A includes a number of local nodes (e.g., local nodes B312, B314A, and B314B). In some implementations, at least one local node B312 is a node currently in use by a user. Depending on the implementation, the local node B312 may be or include a computing device associated with the user (e.g., a computer, mobile device, wearable device, etc.), a computing device associated with a subject (e.g., a pacemaker, an external wearable device, a computer, etc.), a computingAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WG PATENT APPLICATIONdevice associated with and / or part of a medical device (e.g., a surgery assistance / enhancement device, an internal imaging device, a noninvasive imaging device, etc.), a computing device associated with the local environment (e.g., a surgical workstation 16, a planning workstation, a nurse workstation computer, a computer communicatively coupled to a hospital network, etc.), and / or any other such computing device as described herein.

[0264] For example, the local node B312 may be or include a robotic controller 13 that implements the robotic controller architecture A200 and / or A220, as described in more detail above with regard to FIGS. 2A and 2B. As such, the local node B312 may capture image data (e.g., via an endoscope A202) and may transfer the image data to the local distribution node B315 and / or data connector (B320) via the ESC A204.

[0265] In further implementations, the local node B312 may additionally include and / or be replaced by a remote device (e.g., a remote surgeon’s computer workstation, a tele-med computing device, a patient wearable device, etc.) and / or a device functioning as a temporary local device (e.g., via a VPN, temporary ID, etc.).

[0266] In further implementations, the system B300A includes additional local nodes B314A and / or B314B not actively in use by the user. For example, the additional local nodes B314A and / or B314B may be or include nodes that were previously used as part of a procedure; nodes including additional information related to a procedure, user, procedure target, etc.; nodes storing historical data; and / or any other such node. It will be understood that, although FIG. 3A depicts three local nodes, any arbitrary number of local nodes may be envisioned.

[0267] Depending on the implementation, the local node B312 provides information (e.g., surgical data, medical data, etc.) to a local distribution node B315A for analysis (e.g., by a data connector B320). For example, the local node B312 may generate data related to a current step of a procedure (e.g., a surgical procedure, a diagnostic procedure, a preventative care procedure, etc.), such as image data (e.g., an x-ray image, a CT scan image, an MRI image, etc.), subject operation data (e.g., EKG data, catheterization lab data, neurological activity data, etc.), administrative data (e.g., past procedure data, performed procedure step data, guideline / standards data, etc.), and / or any other such form of data for distribution and analysis.

[0268] In some implementations, the local distribution node B315A determines that the additional local nodes B314A and / or B314B include additional data and / or data associated with the provided information. In such implementations, the local distribution node B315A can query the local nodes B314A and / or B314B to retrieve the additional data. In further implementations, the data connector B320 does so instead. Depending on the implementation, the data connector B320 may query the local nodes B314A and / or B314B through the local distribution node, through a direct query to the respective local nodes B314A and / or B314B, through a query to a separate node to function as a relay, etc. In further implementations, the local nodes B312, B314A, and / or B314B receive instructions and / or an indication to perform an action unsuited to and / or unable to be performed by the corresponding node (s). In some such implementations, the corresponding node(s) transmit the data and / or a request to the local distribution node B315A and / or the data connector B320 to request task scheduling performed using the data included. In further implementations, the local distribution node B315A and / or the data connector B320 request data responsive to scheduling the operation and / or transmits an indication of a node to which the corresponding node(s) should transmit the data.

[0269] Depending on the implementation, the local distribution node B315A and / or data connector B320 may be or include the downstream nodes A205 and / or be communicatively coupled to the downstream nodes A205 (e.g., which may then be any one of the local nodes B312, B314A, and / or B314B). Depending on the implementation, the local nodes B312, B314A, and / or B314B; the local distribution node B315A; and / or the data connector B320 may additionally perform functionalities as described with regard to FIG. 2A (e.g., pre-processing data, enhancing pre-processed image data to improve visibility and / or clarity, render image data for real-time display, integrate processed image data with patient data and / or surgical information, encoding data for storage, etc.).Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION

[0270] The data connector B320 receives the data from the local distribution node B315A at a communication interface B322, configured to communicate with local nodes and / or local distribution nodes. In some implementations, the data connector B320 receives the data from the local node B312 (and / or local nodes B314A and / or B314B) directly, bypassing the local distribution node B315A (e.g., as described in more detail below with regard to FIG. 3B).

[0271] The data connector B320 also includes a data analysis module B324 that receives and analyzes the data to generate analysis data. Depending on the implementation, the data analysis module B324 includes a trained machine learning model. In some implementations, the trained machine learning model is or includes a neural network, a large language model, a small language model, a teacher-student trained model, a generative Al model, a linear regression model, an unsupervised model, a supervised model, a semi-supervised model, and / or any other model as described herein. Depending on the implementation, the trained machine learning model may be or include an attention-like model and / or other such model capable of analyzing and / or receiving multi-modal data (e.g., a multimodal transformer model, etc.). In some such implementations, the data analysis module B324 may analyze and / or organize multimodal data (e.g., video data, image data, audio data, text data, etc.). For example, the data analysis module B324 may receive and analyze image data from a local node B312 and audio data from a surgeon speaking to a mobile device (e.g., functioning as an additional local node B314A and / or B314B).

[0272] The trained machine learning model may be trained with historical data regarding one or more medical procedures (e.g., surgical procedures, imaging procedures, data analysis procedures, etc.), such as the historical procedure data 62. The trained machine learning model may additionally or alternatively be trained on subsets of the historical procedure data filtered using one or more indexable fields, such as personnel performing the procedure (e.g., physician, nurse practitioner, administrator, etc.), the procedure type, the robotic equipment configuration, the hospital, etc. As such, the trained machine learning model may be configured to determine a next step and / or next device to use in a procedure based on past data for similar procedures. In particular, the trained machine learning model may determine which device will likely be used next and for what purpose. Similarly, the trained machine learning model may determine whether user preference would indicate that a different node / device will be likely to be used.

[0273] In some implementations, the data analysis module B324 additionally generates mapping data associated with the received data. For example, the data analysis module B324 may link and / or indicate related data, determine how / where data is processed, determine a priority for processing data, etc. In further implementations, the data analysis module B324 may predict, based on the received data, one or more further actions that personnel will take next and / or would prefer to take next (e.g., based on user preference data, based on other past procedures, etc.). Additionally or alternatively, the data analysis module B324 may predict a course of action and present the prediction to the user (e.g., as a recommendation).

[0274] In further implementations, the data connector B320 (e.g., via the communication interface B322 and / or the data analysis module B324) receives user customization and / or historical user information. The data connector B320 can modify an output of the data analysis module B324 and / or other such modules based on the user customization and / or historical user information. For example, the data connector B320 (e.g., via an additional module and / or the data analysis module B324) may determine that a user always orders an EKG as part of a procedure that does not require an EKG. The data connector B320 may then automatically generate an order for an EKG and / or activate a device in accordance with such an order when the user in question is performing the procedure.

[0275] The data analysis module B324 provides the analysis data to an equipment scheduling module B326 and / or a task scheduling module B328. The equipment scheduling module B326 may determine, using the analysis data, a likelihood for one or more future nodes of whether the procedure will use the future nodes. In some implementations, the equipment scheduling moduleAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONB326 determines the likelihood using and / or based on the trained machine learning model of the data analysis module B324. In further implementations, the equipment scheduling module B326 uses a separately trained machine learning module instead. The equipment scheduling module B326 may determine whether the likelihood (e.g., a likelihood metric) meets a predetermined likelihood threshold. Depending on the implementation, the equipment scheduling module B326 may be a part of and / or a functionality implemented by the data analysis module B324.

[0276] In some alternate implementations, the data analysis module B324 determines the likelihood for the one or more future nodes and transmits an indication of such to the equipment schedule module B326, which may in turn generate a request for the node(s) for transmission to a local distribution node B315B. The equipment scheduling module B326 then transmits an indication and / or request to activate one or more local nodes to the local distribution node B315B (e.g., the same as or different from the local distribution node B315A) and / or directly to the local nodes in question.

[0277] The task scheduling module B328 may determine, using the analysis data, one or more data packages to be used in the surgical procedure. In some implementations, the data packages include data to be pre-loaded to a component and / or device (e.g., node) in the system. In further implementations, the data packages include one or more programs, models, applications, and / or other executables. In some implementations, the data packages may be or include information generated by the local nodes 312, 314A, and / or 314B for use by a user (e.g., in a procedure) and / or to be stored for later use (e.g., as described with regard to the ESC A204, A224, etc. and / or one or more downstream nodes A205, A225, etc. above). For example, the data packages may be or include raw image data (e.g., the raw image frames A203), pre-processed image data (e.g., the pre-processed image frames A207), enhanced image data (e.g., to improve visibility and / or clarity), three dimensional data prepared for rendering on monitors and / or heads-up displays, image data integrated with other patient and / or procedure data, sub-frame chunks A227a, and / or any other such data as described herein. In further implementations, the data packages may be or include data retrieved from a cloud network device (such as the (local) cloud 50). For example, the data packages may be or include a model trained at and / or by a model stored at a cloud network device (such as the Al models 64), training data for a model from a database (such as the historical procedure data 62), data with which to pre-configure a node, etc. Depending on the implementations, similarly to the equipment scheduling module B326, the task scheduling module B328 may be a part of and / or a functionality implemented by the data analysis module B324.

[0278] The task scheduling module B328 may transmit the data packages and / or an indication of the data packages to a local distribution node B315C (e.g., the same as or differentfrom the local distribution node(s) B315A and / or B315B) and / or directly to the local nodes in question. Depending on the implementation, the data packages may be particular data to be used in the next step of a procedure, such as subject operation data (e.g., EKG data, catheterization lab data, neurological activity data, etc.), image data (e.g., MRI data, CT scan data, x-ray data, etc.), device data (e.g., pacemaker data, blood sugar monitor data, fitness data, etc.). In some implementations, the data package may be or include data analyzed by the data analysis module B324. In further implementations, the data package may be or include data separate from that analyzed by the data analysis module B324.

[0279] In some implementations, the equipment scheduling module B326 and / or a local distribution node B315B and / or B315C may directly take action on behalf of a user to organize the next steps of a procedure. In some such implementations, the equipment scheduling module B326 and / or the local distribution node B315B and / or B315C may update an inventory management record to reserve the equipment (e.g., via the inventory management application 26), generate a phone call, broadcast information to team members, change settings in a device, activate a device, etc.

[0280] In further implementations, the data analysis module B324, the equipment scheduling module B326, and / or the task scheduling module B328 may function as an LLM orchestrating other agents. For example, the data analysis module B324, theAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONequipment scheduling module B326, and / or the task scheduling module B328 may receive multimodal data (e.g., video data, audio data, and text data), may determine a course of action based on some of the multimodal data (e.g., audio data including instructions from a surgeon), and may organize and / or distribute a remainder of the multimodal data to other modules functioning as machine learning agents (e.g., a module trained to process video data, a module trained to process text data, etc.). In some such implementations, the various modules are stored on the same computing device (e.g., the data connector B320) as the data analysis module B324, the equipment scheduling module B326, and / or the task scheduling module B328. In further implementations, the various modules are stored on other computing devices (e.g., a local node B312, B314A, B314B; a local distribution node B315A, B315B, B315C; n cloud computing device; an edge computing device, etc.). Further inventory management techniques that may be implemented by an LLM of the equipment scheduling module B326 are disclosed in U.S. Application serial no. 63 / 708,145 entitled “SYSTEMS AND METHODS FOR GENERATING PRE-OPERATIVE AND INTRA OPERATIVE GUIDANCE USING ADAPTIVE ARTIFICIAL INTELLIGENCE,” and filed on October 16, 2024, the entire disclosure of which is hereby incorporated by reference.

[0281] The local distribution node(s) B315B and / or B315C may distribute the received instructions, data packages, indications, etc. to the appropriate nodes, devices, etc. In some implementations, the local distribution nodes B315B and / or B315C (or the local nodes B316A and / or B316B as described below) are and / or are communicatively coupled to one or more user computing devices. In some such implementations, the practitioner, administrator, and / or other user can veto the scheduled activation and / or pre-loading prior to performance of such.

[0282] In some implementations, the data analysis module B324 and / or another module of the data connector B320 may additionally or alternatively generate a visual representation of a determined schedule for distribution and display to a user. For example, the data analysis module B324 may determine a plurality of future nodes and an order for the plurality of future nodes. The data analysis module B324 may then determine expected time period(s) in which the respective node(s) are expected to be used. The data analysis module B324 may then generate a visual schedule for review, verification, and / or approval by a user. In some such implementations, the data analysis module B324 may determine that a step of the procedure is taking longer or shorter than expected, and may modify the schedule and / or visual representation accordingly. The data analysis module B324 may then transmit the approved schedule to the data connector B320 for synchronization with a hospital schedule system, such as the scheduling system 28.

[0283] Depending on the implementation, operations and / or functionalities of the data analysis module B324 may be performed by a CPU and / or GPU (e.g., CPU A263 and / or GPU A264) of the data connector B320. Depending on the implementation, the CPU and / or GPU may include a plurality of CPU cores (e.g., CPU cores A265-A267) and / or GPU cores (e.g., GPU cores A268-A270). In further implementations, the CPU, GPU, and / or cores of the CPU and / or GPU may be spread amongst a plurality of devices and / or nodes, or may be part of a computing pack to be plugged into the data connector B320. In some implementations, the task scheduling module B328 may utilize the multiple cores of the CPU and / or GPU, or the task scheduling module B328 may utilize multiple cores of CPUs and / or GPUs of other nodes and / or devices to schedule tasks to be performed appropriately as described above with regard to FIG. 2C. In some such implementations, the task scheduling module B328 may be communicatively coupled to the coordinator A262 of FIG. 2C. In further implementations, the task scheduling module B328 generates and / or configures multiple virtual machines (VMs) to perform assigned tasks and transmits configurations for such and / or instructions to configure such to the appropriate node and / or device to perform the corresponding task(s).

[0284] In some such implementations, the data analysis module B324, equipment scheduling module B326, and / or task scheduling module B328 may determine (i) a set of tasks for the one or more applications to be executed by the compute node, (ii)Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONa task prioritization level for each task of the set of tasks, and (ill) compute resource requirements for each task of the set of tasks, partition, based on (i)-(iii), the set of tasks into a task schedule that specifies a respective processing core of the CPU or the GPU to perform execution of one or more of tasks of the set of tasks, and cause the GPU and the CPU to execute the partitioned / designated tasks in accordance with the task schedule, as described above with regard to FIG. 2C. In further implementations, the data analysis module B324, equipment scheduling module B326, and / or task scheduling module B328 may instead transmit parameters and / or configurations to an individual node / device to perform such determinations on an internal level. Depending on the implementation, the modules in question may perform such using priority-based deadline-aware scheduling, adaptive batching, and CUDA graphs for predictable execution, combined with dynamic stream partitioning and monitoring to meet strict timing guarantees with minimal overhead.

[0285] In further implementations, the user computing devices include remote devices from one or more remote users (e.g., remote surgeons, remote practitioners, remote physicians, etc.). Depending on the implementation, the user computing device may load information from the data connector B320 as part of the scheduling process.

[0286] In further implementations, the remote devices may be part of or may be one of the local nodes B312, B314A, and / or B314B and / or may be queried by such. As such, the remote devices may automatically upload information to the data connector B320. In some such implementations, the remote user provides an indication of such. In further implementations, the data connector B320 determines that the remote node includes the information and automatically queries such.

[0287] FIG. 3B illustrates a block diagram of an exemplary system B300B for receiving local node data (e.g., data regarding various nodes of a hospital, such as surgical device data) and performing autonomous equipment scheduling and / or task scheduling operations on another local node. Similar to FIG. 3A, depending on the implementation, the exemplary system B300B may be, be part of, or include the data connector 20, cloud 50, etc. of FIGS. 1A-1 D, the robotic controller architecture A200 of FIG.2A, the robotic controller architecture A220 of FIG. 2B, the compute resource partitioning scheme A240 of FIG. 2C, the distributed computing environment of FIG. 2D, and / or any other computing component and / or system as described herein.

[0288] The system B300B may be generally similar to system B300A, but may instead take place entirely at a local node (e.g., the local orchestration node B330). As such, the system B300B may utilize edge computing techniques to analyze data via a machine learning model, determine and / or configure future nodes for future steps of a procedure, transmit intra-operative data, generate a schedule, and / or otherwise perform operations as described herein. Additionally or alternatively, the system B300B may utilize local computing with a local model trained via teacher-student machine learning techniques via connection to a teacher model stored at a cloud network device. For example, the data analysis module B334, equipment scheduling module B336, and / or task scheduling module B338 may be or include a model trained by a teacher model. In particular, models stored at the local orchestration node B330 may be initially trained by a teacher model stored at the data connector B320 and / or a cloud network device, but may later be updated and / or tuned using data specific to the local orchestration node (e.g., particularly hospital data, particular unit data, particular device data, particular user data, etc.).

[0289] As such, in some such implementations, the local orchestration node B330 may be one of multiple such orchestration nodes that are periodically updated (e.g., trained) by the cloud server. Similarly, the local orchestration node B330 may therefore store results in a memory (not shown) to be transmitted to the cloud server B320 to update the teacher models as well.

[0290] In some implementations, the communication interface B332, data analysis module B334, equipment scheduling module B336, and / or task scheduling module B338 may be similar to the corresponding modules as described above (e.g., the communication interface B322, data analysis module B324, equipment scheduling module B326, and / or task scheduling module B328, respectively).Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WG PATENT APPLICATION

[0291] Moreover, depending on the implementation, the local orchestration node B330 may be and / or include the local distribution node B315A as described with regard to FIG. 3A above. As such, the local orchestration node B330 may be communicatively coupled to and may directly communicate with local nodes as described above. Further, the local orchestration node B330 may directly configure and / or directly load data onto the local node(s) in question.

[0292] As such, the equipment scheduling module B336 and / or task scheduling module B338 may transmit the data packets and / or instructions directly to the local node(s) B316A and / or B316B as described above.

[0293] In some implementations, an environment (e.g., a hospital, a bank, a research center, etc.) in which the local orchestration node B330 may choose to keep data isolated from other environments. As such, the local orchestration node B330 may alternatively train models such as models used by the analysis module B334, equipment scheduling module B336, and / or task scheduling module B338 using only local data, and may avoid uploading data used to train the respective modules to a cloud computing device.

[0294] Similarly, in further implementations, various nodes and / or environments may have hardware limitations (e.g., due to a device that has not yet been updated to include newer hardware modules and / or processing capabilities). In such implementations, the local orchestration node B330 may similarly train models for such nodes using data obtained from such limited systems while training models for updated nodes using the techniques described above with regard to FIG. 3A. As a result, the training data for such models better reflects the limited capabilities associated with said systems. In some implementations, the historical procedure data (e.g., kinematics data, event data, log data, etc.) retrieved from such a conventional system and / or node may be weighted differently than system data from updated nodes in training models for updated nodes such that the conventional systems still have some influence in the performance of trained model that is implemented across a plurality of system architectures.

[0295] Depending on the implementation, the various nodes and / or environment may function under various environment rules. For example, in implementations in which the environment is or includes a hospital, the hospital may have environmentpreferred rules regarding data ownership and / or privacy. As such, the environment and / or a node associated with the environment may set one or more environment and / or node rulesets, such as refraining from transmitting particular types of data to a cloud computing device, refraining from transmitting any data from a cloud computing device, refraining from transmitting data from particular nodes to a cloud computing device, refraining from training models using data from the cloud computing device, etc. In further implementations, the environment rules may be or include region rules (e.g., rules to prevent exportation of data from a region, rules to obfuscate particular types of data within certain regions, rules to clean data exiting an environment within a region, etc.). As such, the instant system B300B (and / or other such systems as described herein, such as systems B300A, B300C, etc.) may enforce regional, environmental, and / or other such preference-based controls while routing data and / or scheduling tasks.

[0296] FIG. 3C illustrates a block diagram of an exemplary system B300C for receiving local node data (e.g., data regarding various nodes of a hospital, such as surgical device data) and performing autonomous equipment scheduling and / or task scheduling operations in an edge computing environment. Similar to FIGS. 3A and 3B, depending on the implementation, the exemplary system B300B may be, be part of, or include the data connector 20, cloud 50, etc. of FIGS. 1 A-1 D, the robotic controller architecture A200 of FIG. 2A, the robotic controller architecture A220 of FIG. 2B, the compute resource partitioning scheme A240 of FIG. 2C, the distributed computing environment of FIG. 2D, and / or any other computing component and / or system as described herein.

[0297] The system B300C may generally be similar to the system B300A and / or the system B300B, but may instead take place partially at a local node (e.g., the local orchestration node B330) and partially at a data connector B320 (e.g., in an edge computing environment). As such, similar to the systems B300A and B300B of each of FIGS. 3A and 3B, the system B300C mayAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WG PATENT APPLICATIONutilize cloud computing techniques, edge computing techniques, local computing techniques, and / or other such techniques as described herein. In some implementations, the data connector B320 may be or include a computing pack to be plugged into the local orchestration node B330. Additionally or alternatively, the local orchestration node may be a computing pack to be plugged into one or more local distribution node(s) B315A, B315B, B315C and / or other local nodes, as described herein. As such, older devices and nodes may be incorporated into the scheduling system as described herein without requiring a complete replacement and / or overhaul of the system devices.

[0298] Furthermore, depending on the implementation, similar to FIG. 3B, the local orchestration node B330 may be and / or include the local distribution node B315A as described with regard to FIG. 3A above. As such, it will be understood that the instant techniques as described with regard to FIGS. 3A-3C may be or include a cloud computing device communicatively coupled to one or more additional nodes, a local on-premises node including one or more modules and / or models trained using one or more modules and / or models stored at a cloud computing device, an edge-computing node communicatively coupled to a cloud computing device, a computing pack including one or more modules and / or hardware to support the one or more modules to be plugged into local nodes, and / or any other such implementations as described herein.

[0299] Even though the local orchestration node B330 is depicted as including the data analysis module B334 and the data connector B320 is depicted as including the equipment scheduling module B326 and the task scheduling module B328, it will be understood that alternative arrangements are also envisioned. For example, the local orchestration node B330 may include the data analysis module B334 and a task scheduling module (e.g., task scheduling module B338 of FIG. 3B) while the data connector B320 may include the equipment scheduling module B326. It should be appreciated that the term “task” may refer to either a task performed in furtherance of a medical procedure or a processing task associated with executing a particular command or function. Accordingly, the task scheduling module B338 may assign tasks to be performed by a particular computing system and / or a particular processing unit thereof.

[0300] Similar to FIG. 3B, the data analysis module B334 may be trained by a teacher and / or training machine learning model stored at the data connector B320 (e.g., a data analysis module B324) using teacher student machine learning model training techniques.

[0301] In some implementations, the local distribution node B315B and / or B315C may be or may include the local orchestrion node B330. Additionally or alternatively, the equipment scheduling module B326 and / or local task scheduling module B328 may transmit the data packet(s) and / or instructions directly to the local nodes (e.g., local nodes 116A and / or 116B as depicted in FIG. 3B).

[0302] FIG. 3D illustrates a block diagram of an exemplary local node B350, configured to perform autonomous task scheduling operations using data gathered at the node (e.g., data regarding various nodes of a hospital, such as surgical device data). Depending on the implementation, the local node B350 maybe or include a local node B312, B314A, and / or B314B of FIGS.3A-3C. Similarly, the local node B350 may be or include a local distribution node B315A, B315B, and / or B315C of FIGS. 3A-3C. Moreover, the local node B350 may be or include components of the data connector 20, cloud 50, etc. of FIGS. 1A-1D, the robotic controller architecture A200 of FIG. 2A, the robotic controller architecture A220 of FIG. 2B, the compute resource partitioning scheme A240 of FIG. 2C, the distributed computing environment of FIG. 2D, and / or any other computing component and / or system as described herein.

[0303] The local node B350 includes a number of device component (e.g., device components B362, B364A, B364B, and B366). In some implementations, at least one device component B362 may be in use and / or have previously been used by a user (e.g., a physician, a nurse practitioner, an administrator, etc.). Depending on the implementation, additional device componentsAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONB364A and / or B364B may be or include additional components that may store and / or access additional data (e.g., historical data, procedural data, user preference data, etc.), may communicate with a user device, may query additional data and / or prompt a user, and / or any other such component as described herein.

[0304] The device component B362 transmits data (e.g., surgical data) to an orchestration component B380 of the local node B350. In further implementations, additional device components B364A and / or B364B may provide additional data to the orchestration component B380. In some implementations, the local node B350 is or includes a surgical device. In some such implementations, the device component B362 is the surgical device and / or a component of the surgical device B362 (e.g., a data generation component, a data gathering component, etc.). In some implementations, the orchestration component may be or include a computing pack to be plugged into the local node B350. Depending on the implementation, the orchestration component B380 may be disposed close to a controller and / or other hardware implementing the local orchestration module B360 and / or cloud orchestration module B370 to ensure quick access and processing (e.g., for intra-operative data transmission, as described below with regard to FIG. 3F).

[0305] In further implementations, the additional device components B364A and / or B364B may be additional components of a same device as device component B362, components of other devices (e.g., if the local node includes a plurality of devices), components communicatively coupled to other devices, and / or any other such device components as described herein.

[0306] In some implementations, the local node B350 may include an orchestration module, such as local orchestration module B360 and / or cloud orchestration module B370. Depending on the implementation, the cloud orchestration module B370 may receive instructions from a data connector (e.g., data connector B320 of FIGS. 3A-3C), and the orchestration component B380 may perform an operation and / or task scheduling responsive to receiving the instructions.

[0307] In further implementations, the orchestration component B380 may receive data from the cloud orchestration module B370 (and / or from another cloud communication module (not shown) in the local node B350), and the orchestration component B380 may analyze the data (e.g., via data analysis module B384) and / or determine whether to act on instructions from the cloud orchestration module B370.

[0308] In some implementations, the orchestration component B380 additionally or alternatively receives instructions from a local orchestration module B360 (e.g., from one or more additional local nodes (not shown) in a system). In further implementations, the orchestration component B380 may receive data from the local orchestration module B360, similar to the cloud orchestration module B370. The orchestration component B380 may include a communication interface B382, a data analysis module B384, a task scheduling module B388, and / or any other modules as discussed herein.

[0309] In some implementations, the communication interface B382, the data analysis module B384, and / or the task scheduling module B388 are similar to the communication interfaces B322 and / or B332, data analysis modules B324 and / or B334, and task scheduling modules B328 and / or B338, respectively.

[0310] In some implementations, the orchestration component B380 (e.g., via the task scheduling module B388) transmits one or more data packets to a device component B366. Depending on the implementation, the device component B366 may be a component of a device in the local node B350 (e.g., the device component B362, the device component B364A and / or B364B, and / or another device component). In further implementations, the orchestration component B380 (e.g., via the task scheduling module B388) organizes and / or configures particular computes and / or tasks. For example, the task scheduling module B388 may transmit instructions to execute a particular command at a particular processor or other component (e.g., in addition to or in place of the data packets). Similarly, the task scheduling module may prioritize and / or consider factors for various components and / orAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONcomputes to perform various actions in scheduling a task (e.g., stability of components and / or functionalities, execution speed, priority of task, impact on procedure outcome, etc.). Depending on the implementation, the orchestration component B380 may adhere to and / or determine a prioritization scheme for node and / or component resources before transmitting data packets and / or instructions to the corresponding node and / or component. For example, a task with a high priority (e.g., an urgent surgery-related task) may be allocated resources that have higher execution speed, isolation, and / or other desirable compute traits than a low priority task.

[0311] In further implementations, the device component B366 may be communicatively coupled to another device (e.g., outside of local node B350), and may similarly transmit the data packets and / or instructions to another local node and / or device. In still further implementations, the device component B366 may be another device (e.g., associated with the local node B350). In some such implementations, the orchestration component B380 may implement load balancing techniques and / or other such load optimizations (e.g., as described herein) to ensure timely processing of tasks orchestrated by the orchestration component B380.

[0312] It should be appreciated that while the foregoing sets forth various modules configured to perform the described functionality, in some embodiments, these modules are implemented as an Al agent 41 (including an LLM Al agent) within a runtime environment hosted at the corresponding computing device that executes the module.

[0313] FIG. 3E illustrates a flow chart of an exemplary method B400 for orchestrating efficient usage of a plurality of local surgical nodes via a cloud network. Depending on the implementation, the method B400 may be implemented by any one of the systems 100, A200, or B300; one or more components associated with the hospital 10; one or more alternative systems; and / or any other such components as described herein.

[0314] At block B402, one or more processors of a computing device (e.g., the cloud computing device B320, the local orchestration node 330, a local orchestration component B380, etc.) receives procedure data from one or more local computing nodes of a plurality of computing nodes. In some implementations, the one or more local computing nodes are used to support a procedure (e.g., a surgical procedure). In some implementations, the local computing nodes may include a remote node (e.g., a remote user) functioning as a temporary local node (e.g., via a VPN, temporary ID, etc.).

[0315] At block B404, the computing device may analyze the procedure data via a trained machine learning model to generate procedure analysis data. In some implementations, the computing device further trains the machine learning model. In some such implementations, the machine learning model is a local machine learning model, and the computing device trains the local machine learning model using a teacher machine learning model (e.g., stored at a cloud database). In further implementations, the computing device receives historical data associated with a procedure to be performed and trains the machine learning model using the historical data. The historical data may include user preference data associated with a procedure, practice guidelines for a procedure, environment-specific data for a procedure, etc. The trained machine learning model may, in some implementations, be stored and used at an edge-computing device.

[0316] At block B406, the computing device may predict, based on the procedure analysis data, a likelihood of at least one node being a future node to be used as part of the procedure. In some implementations, the computing device receives an indication from a user of a preferred next step of the procedure. In some implementations, the computing device predicts the likelihood at least in part based on the indication. Alternatively, the computing device may update the likelihood based on the indication. The indication may be or include a direct indication from the user (e.g., choosing one of multiple options, indicating a particular preference, checking a particular box at the beginning of the procedure, etc.), an indirect indication (e.g., a history of preferring one over node over another, a use of a node in a current procedure that would require a different order from a standardAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONorder, etc.), and / or any other such indication. As described in more detail above, the computing device may train the machine learning model based on the indication.

[0317] At block B408, the computing device may configure the at least one node based on the predicted likelihood. In some implementations, configuring the at least one node includes activating the at least one node. For example, the computing device may transmit an indication to the at least one node including instructions to activate at a predetermined time and to be deactivated until such a time. As another example, the computing device may wait until a predetermined time to transmit a wakeup message to the at least one node. In some such implementation, the computing device transmits periodic messages to keep the at least one node activated until the predetermined time and / or a predetermined ending time.

[0318] In further implementations, the computing device further configures the at least one node by deactivating the at least one node (e.g., after the procedure no longer uses the at least one node). In some such implementations, the computing device determines a predetermined wakeup time and a predetermined sleep time and subsequently transmits an initial wakeup message to the at least one node and later transmits a sleep message to turn the at least one node off.

[0319] In further implementations, configuring the at least one node includes transmitting an indication to a user that the at least one node is to be activated and / or deactivated. Depending on the implementation, the indication may include a scheduled time and / or time period from a current moment (e.g., in 10 minutes, in 1 hour, on July 2nd, etc.). In further implementations, the indication may include a device identifier (e.g., a device name, device serial number, device ID, relevant room number, etc.) and a confirmation prompt for the user to verify the schedule. The computing device may then activate and / or deactivate the device responsive to a confirmation from the user. In some such implementations, the computing device provides the confirmation prompt only when activating and / or deactivating a node. In alternative implementations (e.g., when the computing device is pre-loading data onto a local node and / or pre-configuring a local node with tailored models for a procedure), the computing device may automatically activate the device without waiting for user confirmation and / or after a timeout of waiting for a user confirmation occurs.

[0320] In further implementations, configuring the at least one node includes pre-loading one or more data packages to the at least one node. For example, the data package(s) may include historical procedure data generated by one or more previously accessed computing nodes, a priority indication for the data package(s) (e.g., an order in which the data packages are to be used), etc. In further implementations, configuring the at least one node includes pre-configuring the at least one node with one or more tailored models for performing a procedure.

[0321] In still further implementations, configuring the at least one node includes generating a mapping for data associated with the procedure indicative of which future nodes with which each respective portion of the data is associated. The mapping may, depending on the implementation, include indications of relations between data segments of the data (e.g., indications of how data is related to the current subject, indications of how data was previously used, indications of what data was used to generate what other data, etc.). Depending on the implementation, the computing device may configure the at least one node responsive to determining that the likelihood meets a predetermined likelihood threshold value.

[0322] In some implementations, the computing device may consider a priority associated with the data when configuring the at least one node. For example, if data with a higher priority is determined to be needed by the local node (e.g., indicating an emergency that requires the node), the computing device may overwrite the previously configured data with the new data. In further implementations, the computing device may determine whether to overwrite data based on the determined schedule including one or more rules. Depending on the implementation, the one or more rules may include rules to avoid overwriting data for a node thatAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONhas already begun a procedure, avoid overwriting data for a node within a predetermined timespan before or after performing a procedure, overwrite data responsive to confirmation / verification from a human, etc.).

[0323] In some implementations, the instant techniques as described by the method B400 decrease the power consumption of nodes associated with the computing device. For example, by activating and / or deactivating nodes based on the predicted likelihood, the instant techniques reduce the power used from such nodes, which may otherwise exceed a power limit for an operating room.

[0324] Similarly, the processing power / resource usage may similarly be reduced, in that improving and simplifying the infrastructure to a particular node managing scheduling reduces the complex processing power needs from scheduling and redundant determinations.

[0325] FIG. 3F illustrates a flow chart of another exemplary method B500 for orchestrating efficient usage of a plurality of local surgical nodes via a cloud network. Depending on the implementation, the method B500 may be implemented by any one of the systems 100, 200, or B300; one or more components of anyone of the systems 100, A200, or B300; one or more components outside of the systems 100, 200, or B300; one or more alternative systems; and / or any other such components as described herein.

[0326] At block B502, one or more processors of a computing device (e.g., the cloud computing device B320, the local orchestration node B330, a local orchestration component B380, etc.) receives surgical procedure data from one or more local surgical nodes of a plurality of computing nodes. In some implementations, the one or more local surgical nodes are used as part of a surgical procedure. In further implementations, block B502 is performed similarly to block B402.

[0327] At block B504, the computing device may analyze the surgical procedure data via a trained machine learning model to generate procedure analysis data. In further implementations, block B504 is performed similarly to block B404.

[0328] At block B506, the computing device may predict, based on the procedure analysis data, one or more destination surgical nodes of the plurality of computing nodes. In some implementations, the computing device receives an indication from a user of a preferred next step of the procedure. In some implementations, the computing device predicts the destination surgical node(s) at least in part based on the indication. Alternatively, the computing device may update the prediction based on the indication. The indication may be or include a direct indication from the user (e.g., choosing one of multiple options, indicating a particular preference, checking a particular box at the beginning of the procedure, etc.), an indirect indication (e.g., a history of preferring one over node over another, a use of a node in a current procedure that would require a different order from a standard order, etc.), and / or any other such indication. As described in more detail above, the computing device may train the machine learning model based on the indication.

[0329] At block B508, the computing device may transmit the surgical procedure data to the one or more predicted destination surgical nodes for processing. In some implementations, the surgical procedure data may be or include data used to determine the destination nodes, additional information, information indicated by the user, etc. In further implementations, the computing device may additionally or alternatively include update data to the surgical nodes to update an operation methodology, programming, etc. Additionally, the computing device may facilitate communication between individual surgical nodes such that the computing device may ensure that a compute is properly scheduled and synchronized for communication based on the predicted destination node.

[0330] In some implementations, the instant techniques as described by the method B500 decrease the resource usage of nodes associated with the computing device. For example, by removing the redundancy in transmitting and / or analyzing data at each individual node, the overall latency and bandwidth requirements are reduced.Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION

[0331] Moreover, the instant techniques improve the efficiency of usage by scheduling when the processing occurs to improve overall resource usage across a location and / or system associated with the scheduling node(s) and / or computing device(s).DIGITAL TWINS

[0332] While the foregoing set forth a hardware tech stack that enables efficient routing of tasks and / or data, these improved efficiencies enable digital tech stacks to leverage this efficiency to enable services and / or applications that may be too intensive for conventional hardware arrangements. That is, while the following describes the application-layer improvements, it is envisioned that these application-layer techniques are implemented using the hardware systems described above. For example, the various functionality described with respect to the following figures may be orchestrated and / or coordinated using a data connector (and / or an omnischeduler implemented thereby) using the improved hardware processing techniques described above.

[0333] With the foregoing considerations in mind, FIG. 4A depicts an example computing environment C100 in which methods and systems for presenting a digital twin may be implemented, according to embodiments. The presentation of a digital twin may be viewed as task managed by an orchestration node (such as the local orchestration node 330). Accordingly, in some embodiments, a data analysis module (such as the data analysis module 324) may analyze the digital twins as an additionally contextually-relevant data stream to perform any of the various efficient task generation and / or routing described above.

[0334] The computing environment C100 may include a network C105 communicatively coupling a remote viewing system C110 (e.g., the remote viewing system 25), a surgical workstation C115 (e.g., the surgical workstation 16) coupled to robotic control systems (such as the robotic control system 13) for robotic equipment (such as the robotic equipment 12), a remote client device C120, and a planning workstation C125. It should be appreciated that the digital twins described herein may be accessed preoperatively (e.g., for planning in anticipation of an upcoming procedure), intraoperatively (e.g., to support remote viewing and / or for practicing an upcoming phase of the procedure), and / or postoperatively (e.g., to review a past procedure and simulate how different decisions may have resulted in different outcomes). In the intraoperative scenarios, the various techniques for efficient scheduling of data may be applied to the processing systems at which the digital twins are implemented to configure the available processing resources in a manner that supports “real-time” display of digital twins of objects located in an operating room (such as the operating room 11). Although FIG. 4A depicts certain entities, components, equipment, and / or devices, it should be appreciated that additional, fewer, and / or alternate entities, components, equipment, and / or devices are envisioned.

[0335] The network C105 may be and / or include one or more wired networks (e.g., data bus, Ethernet, LAN, WAN), wireless networks (e.g., cellular, Wi-Fi, WLAN), local networks, remote networks, or any combinations thereof, and / or any other suitable means of transmitting data throughout the computing environment C100. In at least some embodiments, the remote viewing system C110 communicatively couples or otherwise connects multiple data connectors 20, 320 for transmission of data (e.g., multimodal data streams C104) among the data connectors 20, 320.

[0336] The remote viewing system C110 may include one or more computing devices configured to provide a virtual environment (VE). As one example, the remote viewing system 110 may host an NVIDIA Omniverse virtual environment. In some embodiments the virtual environment may provide or otherwise include a simulation of augmented information overlaid onto real-world objects via a viewer device (e.g., augmented reality surgical information overlaid on a patient and presented via an augmented reality headset). In some embodiments, the VE may be entirely simulated (e.g., a simulated surgical procedure performed using digital twins of a surgical robot and a surgical patient, and presented via a virtual reality headset). While the following generallyAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONprovides examples for a VE solution, any such description envisions an equivalent implementation in an AR system. That is, after the digital twin model is generated, the digital twin model may be presented within any suitable virtual environment.

[0337] The remote viewing system C110 may generate, update, import and / or implement one or more digital twins (also referred to at times as “models”) of various objects located within the operating room (such as procedure personnel, a subject, robotic and / or other equipment used perform the procedure, etc.). As it is used herein, a digital twin or model may be a virtual object that reflects a plurality of characteristics of a corresponding real-world object. For example, the digital twin may model the dimensions of the object, the appearance of the object, the material of the object (e.g., by including physics models that indicate how different portions respond to external stimuli (e.g., how a tissue responds to a force applied by a catheter, a CAD model of how mechanical components interact with one another), and / or a function of the object (e.g., blood flow through internal vessels, the various drives and / or other mechanisms that implement robotic motion). Accordingly, the digital twin may simulate the look, feel, texture, characteristic and / or properties of the digital twin, such as simulating the hardness, elasticity, resistivity, impedance, dimensions, contours, and texture of tissue, bone, cartilage, muscle, and / or other anatomy of the patient via one or more models.

[0338] The remote viewing system C110 may store one or more digital twins on a memory for inclusion in the VE. As described above, a data connector (such as the data connector 20) may pre-load the memory with the digital twins prior to a procedure, for example by implementing the method B500. Accordingly, when initiating the VE, the remote viewing system C110 may retrieve a digital twin for inclusion in the VE.

[0339] The remote viewing system C110 may generate and / or obtain the digital twin pre-operatively and / or intraoperatively. For example, the remote viewing system C110 may generate both an internal and external digital twin of a subject (e.g., a digital twin generated from external image and / or depth data to represent the external model and a digital twin of anatomy that will undergo a surgical procedure as derived, for example, via preoperative data C102 and / or endoscopic intra-operative imaging). Accordingly, the surgical workstation C115 may generate portions of the subject’s digital twin in the VE based on the multi-modal data streams C104 and / or import other portions of the subject’s digital twin into the VE based on the per-operative data C102. For example, at the onset of the procedure, the remote viewing system may analyze depth data to determine a position of the subject in the operating room and align a preoperative model of the subject anatomy (e.g., a model derived based on preoperative image data) with a model of the subject derived from the depth data.

[0340] During the procedure, the surgical workstation C115 may update the digital twin based upon the multimodal data streams C104 or other suitable data. For example, the surgical workstation C115 may update a digital twin of the subject’s anatomy based upon using intraoperative data associated with the subject’s anatomy. For example, the surgical workstation C115 may update an external texture map (e.g., a “skin”) of the model based on intraoperative data. As another example, the surgical workstations C115 may update a pose of the model based on kinematic data of the real-world equivalent object. Similarly, the surgical workstation C115 may detect changes in pose of the various robotic equipment in the operating room (e.g., via the kinematic data) to generate corresponding pose updates for the digital twins that correspond to the robotic equipment. The surgical workstation C115 may synchronize the updates to the remote viewing system C110 such that the VE is correspondingly updated to reflect the updated model information.

[0341] The surgical workstation C115 may include one or more user input devices for manipulating robotic equipment located within the operating room. Accordingly, the surgical workstation C115 may include a display device configured to present intraoperative image data (e.g., endoscopic image data). In at least some embodiments, a user (e.g. surgeon) of the surgical workstation C115 may switch to a VE mode in which the surgical workstation C115 presents the VE on the display device (e.g., using one or more digital twins of surgical devices and a subject surgical patient) instead of the intraoperative image data. In thisAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONmode, the user inputs provided via the user input devices may be converted to equivalent actions within the VE to virtual implement the commands. For example, a surgeon may utilize the VE to simulate one or more surgical steps on a digital twin of the patient using a digital twin surgical robot before performing the actual surgical steps in the real-world. As described above, the surgical workstation 0115 may update the digital twin of the endoscope as it traverses the subject anatomy. It should be appreciated that the particular simulated surgical steps may be derived based upon steps performed in historical procedures and / or output from a task generation machine learning model processing multimodal data generated within the VE. Accordingly, when the surgical workstation 0115 switches to the VE mode, the display device may present the field of view of the digital twin of the endoscope in the VE.

[0342] One or more remote client devices 0120 may also connect to the VE (e.g., via the network 105) for their respective users to observe the simulated surgical procedure (e.g., for teleoperation, training, guidance), among other things. It should be appreciated that in some embodiments, it may be faster to update the VE to reflect a digital twin of the real-world than it is to transmit the real-world data to the remote client devices C120. As a result, the remote client devices C120 may be able to view the state of the real-world procedure in a lower latency manner than conventional solutions.

[0343] In at least some embodiments, the remote viewing system C110 and / or other suitable devices (e.g., the surgical workstation C115, the remote client device C120, surgical robots, etc.) may implement the various processing that supports the VE using a dedicated VE GPU C112, such as the NVIDIA OVX® reference architecture, implement general data processing using a general-purpose GPU 0114, such as the NVIDIA DGX® reference architecture. The GPUs 0112, 0114 may provide accelerated artificial intelligence and graphics performance via high-bandwidth GPUs, advanced physics engine(s), and / or low-latency network interfaces, among other things. The GPUs 0112, 0114 may include one or more data connectors XXX. The GPU 0112 may provide the remote viewing system 110 with superior performance when providing a virtual environment, generating and executing digital twins, generating, training, and / or executing models (e.g., machine learning models), receiving, processing, and / or transmitting multimodal datastreams 0104, etc., as compared to a conventional GPU. In at least some embodiments, the surgical workstation 0115 may host the virtual environment via the VE GPU 0112 while processing one or more of the multimodal data streams 0104 (e.g., for intraoperative purposes such as surgical device control, updating a digital twin, providing guidance, remote viewing via remote client devices 0120) via the general-purpose GPU 0114.

[0344] As described herein, the digital twins (e.g., 3D models) may simulate and / or otherwise model surgical devices, equipment and / or robots, e.g., that are used to perform a real-world procedure within the VE. The surgical workstation 0115 may generate the surgical device digital twin based upon data from the surgical equipment manufacturer (e.g., device specification information), usage data associated with the actual surgical equipment (e.g., data from historical surgical procedures), and / or any other suitable data associated with surgical equipment. In one example, the manufacturer or otherwise provider of the surgical equipment may provide a digital twin of the corresponding surgical equipment. In some embodiments, the remote viewing system 0110 may obtain an indication of the robotic equipment configuration in advance of the procedure (e.g., via facility scheduling data) to import the appropriate digital twin into the VE. In other embodiments, the remote viewing system 0110 may derive robotic equipment types from event data received from the robotic equipment during an initial portion of the procedure.

[0345] The surgical device digital twin may simulate performance of a corresponding real-world robotic surgical device. The physics engine of the remote viewing system 0110 may simulate the control, operation, responsiveness and / or other characteristics of components of the surgical device (e.g., controllers, powered joints, actuators, end effectors, etc.). In some embodiments, one or more control scripts may set values for the control parameters that control simulated operation of the surgical device digital twinAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONin the VE. In other embodiments, a robotic transformer model may obtain virtual multimodal data and a task to be implemented to generate the values for the various control parameters to implement the corresponding task.

[0346] Additionally or alternatively, the remote viewing system C110 may use and / or receive control data or otherwise input from the surgical workstation 115 to generate one or more control values for the control parameters of the digital twin. For example, the surgical workstation 0115 may include user input devices (e.g., handheld controllers) for generating control signals for controlling a surgical robot performing a surgical operation on a patient. The remote viewing system 0110 may receive the control signals from the surgical workstation 0115 (e.g., via the network 0105), and use the input to control a digital twin of the surgical robot in the VE to perform a corresponding simulated operation for the digital twin of the robotic equipment. The surgical equipment model(s) may allow a VE user to simulate using the associated surgical equipment in the VE, for example to train a surgical student to use the surgical equipment, to plan and / or simulate a surgical procedure on a simulated patient (e.g., to evaluate performance of the surgical device and outcome of the surgical procedure before it is performed), etc.

[0347] Additionally, one or more digital twins may simulate and / or otherwise model a subject. The subject digital twin may be based upon preoperative model data C102 of the patient, such as preoperative images, electronic health record data, and / or other suitable data. For example, the remote viewing system C110 may generate multiple digital twins that collectively simulate subject anatomy (e.g., one digital twin of ribs, another digital twin of lungs airways), including the surgical work site and / or the various channels between the surgical work site and an access port.

[0348] For example, in some embodiments, the digital twin of the lung airways may enable a user to simulate how to navigate to a particular work site within the VE. More particularly, the preoperative data may identify a work site for performing a biopsy or treating a polyp associated with the lungs. In addition to simulating how to reach the work site, the simulation may also be used to determine whether a particular positioning of the robotic equipment provides a kinematic motion envelop that permits the work site to be reached in the first place. In these embodiments, an optimal location of the robotic equipment may be determined based on the simulation such that preoperative guidance can be provided to medical personnel to position the real-world robotic equipment in the determined position.

[0349] During the procedure, the surgical workstation C115 may generate and / or update (e.g., in the VE) a digital twin model based upon multimodal data streams C104, preoperative data C102, and / or other suitable data. In at least some embodiments, the preoperative data C102 of a subject may be used to generate a digital twin of the subject which is updated based upon multimodal data streams C104 associated with the subject (e.g. intraoperative image data of patient anatomy). Similarly, a digital twin of a surgical device may be updated in the VE based upon multimodal data streams C104 associated with the surgical device (e.g., force data, procedure video, etc.). Accordingly, the digital twin may be updated in the VE substantially in real-time such that the VE generally-reflects the state of the real-world procedure. This may enable the remote client devices C120 to remotely view the state of the procedure with lower latency than conventional approaches where the multimodal data C104 is transmitted to the remote client devices C120.

[0350] Updating the digital twin of the subject may include texture mapping (e.g., via the one or more image warping and / or transposition techniques being applied to endoscopic data and / or corresponding depth data), generating or updating no-fly zones to restrict the virtual positioning of the surgical device digital twin respective to the patient digital twin, etc. In one example, the positioning of simulated actuators of a digital twin surgical robot in the VE may be updated using real-time sensor data of the multimodal data streams C104 corresponding to the pose of the physical actuators of the physical surgical robot. In another example, a digital twin of a subject’s anatomy in the VE may be updated based upon the multimodal data streams 104 providing real-time intraoperative data (e.g., image data) of the subject’s actual anatomy.Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION

[0351] In some embodiments, the surgical workstation 0115 may be configured to provide image registration to correlate the objects in the image data with the corresponding object of the digital twin. Registration maybe based upon fiducials, landmarks (e.g., bony structures, strictures, carinas, etc.), and / or any other suitable means of registering image data and / or model data. The image registration may include registering a coordinate system of the digital twin in the VE with a coordinate system of the intraoperative image data of multimodal data streams C104. In one example, registration may cause updates to the subject’s anatomy captured in the intraoperative image data to be reflected in the digital model of the anatomy in the VE. In another example, the remote viewing system C110 may receive endoscope sensor data as a data stream of the multimodal data streams C104 indicating a physical pose of a robotic endoscope to update the pose of the digital twin of the robotic endoscope in the VE.

[0352] The surgical workstation C115 may be or include one or more computing devices configured to control a surgical device. The surgical workstation C115 may include one or more user interfaces (e.g., surgical display, AR / VR headset, keyboard, mouse, hand controllers, etc.) and / or otherwise be configured for controlling one or more physical surgical devices, controlling one or more simulated digital twin surgical devices, viewing the VE, and / or generally interacting with the VE. The surgical workstation C115 may be configured to access, update, configure, and / or view the virtual environment via the remote viewing system C110. In at least some embodiments, the user interface of the surgical workstation C115 may allow the user (e.g., surgeon) to view surgical data such as the multimodal data streams C104 and the preoperative data C102. The surgical workstation C115 may be configured to simultaneously access the VE to perform a simulated surgical procedure while also performing the corresponding surgical procedure on an actual patient using actual surgical equipment.

[0353] The remote client device C120 may be or include one or more computing devices configured to access the virtual environment via the remote viewing system C110. The remote client device C120 may provide or otherwise include one or more user interfaces (e.g., surgical display, AR / VR headset, keyboard, mouse, hand controllers, etc.) for viewing and / or interacting with the VE (e.g., during a simulated surgical procedure) and / or view an actual surgical procedure (e.g., via the multimodal data streams C104). For example, the remote client device C120 may be configured for real-time observation (e.g., for surgical procedure training, surgical procedure guidance, surgical device training) of an actual or simulated surgical procedure performed by a surgeon operating the surgical workstation C115.Preoperative Surgical Procedure Planning

[0354] In at least some embodiments, the computing environment C100 may support preoperative surgical planning. The preoperative planning may include generating digital twins, planning the order of surgical steps to perform, generating surgical guidance, etc.

[0355] The planning workstation C125 may include one or more computing devices configured to execute software to facilitate planning and / or preparation for a surgical procedure. The planning workstation C125 may be, or include, the surgical workstation C115 and / or remote client device C120. In at least some embodiments, the planning workstation C125 may receive or otherwise access the preoperative data C102. The preoperative data C102 may include electronic medical record data of a subject of a surgical procedure, preoperative image, a surgical plan, surgical notes, digital twins (e.g., of a patient and / or surgical equipment), and / or any other suitable preoperative data associated with the subject. The preoperative data C102 may include one or more images from computed tomography, tomosynthesis, cone-beam computed tomography, and / or other suitable images, such as images of tissue, bones, tumors, anatomy, etc. of a region of the subject that will undergo surgery.

[0356] The planning workstation C125 may allow the user to view, generate and / or edit the preoperative data C102, such as indicating no-fly zones, navigational pathways, and surgical planes for a surgical robot digital twin, annotating anatomicalAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONstructures (e.g., with surgical guidance information), generating a surgical plan (e.g., of ordered steps to perform). In at least some embodiments, the user of the planning workstation 0125 may generate and / or edit the preoperative data 0102 via the VE. For example, the user (e.g., surgeon) of the planning workstation 0125 located at a hospital may execute a surgical planning application to view a model of the patient in the VE to become familiarized with the expected subject anatomy, practice the procedure, and / or otherwise prepare for the procedure through interactions within the VE. The user may annotate patient models with notes and / or other information that the use wants to appear when performing the procedure.

[0357] In at least some embodiments, the VE may be used to view or modify a digital twin associated with the patient (e.g., of patient anatomy). For example, the user may select or otherwise determine attributes of the digital twin, such as selecting a type of tissue to model, identifying a location of a lesion, a cancerous region, or other tissue condition, etc. Based on the indicated changes, the VE hosted by the surgical workstation C115 may modify the various physical characteristics of the digital twin are simulated in the VE (e.g., by applying a different set of physics parameters associated with the indicated type of tissue and / or condition).

[0358] In some embodiments, the user of the planning workstation C125 may executing one or more surgical simulations in the VE using patient models and / or surgical device models. In some embodiments, the simulated VE is generated in a manner that is privacy preserving while enabling simulation of different scenarios. Such techniques are described in U.S. Application serial no. 63 / 595,244 entitled “SYSTEMS AND METHODS FOR SENSOR DATA GENERATION IN MEDICAL ENVIRONMENTS,” and filed on November 1, 2023, the entire disclosure of which is hereby incorporated by reference. The simulations may determine steps of the surgical procedure, evaluate the performance of digital twins, generate information (e.g., instructions, navigational pathways, warnings, guidance, no-fly zones, surgical planes, annotations) for presentation intraoperatively (e.g., to the surgeon, a surgical procedure viewer) during an actual corresponding surgical procedure, evaluate and analyze corresponding simulated outcomes, generate scripts for a surgical devices to execute during the actual surgical procedure, and / or other suitable purposes. It should be appreciated that the simulation may implement stochastic modeling techniques to identify the most likely outcomes for the procedure. If the planning station C125 detects any complications in the simulated procedure, the planning workstation C125 may generate guidance to avoid the corresponding source for the complication. In some embodiments, the guidance is provided by modifying how one or more machine learning models control autonomous operation of robotic equipment (e.g., by modifying a constitution input into the model, by selecting a particular model version, etc.) to avoid the identified potential sources of complications.Intraoperative Surgical Procedure

[0359] In at least some embodiments, the computing environment C100 may be configured to perform a surgical procedure. As previously described, the surgical workstation C115 may control one or more surgical devices (e.g., robots) used to perform a surgical procedure, display or otherwise provide preoperative and intraoperative guidance (e.g., no fly-zones, navigational guidance, surgical planes, etc.), collect and / or transmit real-time surgical data (e.g., the multimodal data streams C104), etc. In operation, the remote viewing system C110 and / or surgical workstation C115 may receive (e.g., via the data connector 20, B320) multimodal data streams C104 from one or more devices (e.g., surgical devices, imaging devices, etc.), such as described in U.S. Patent Application serial no. 63 / 706,355 entitled “SYSTEMS AND METHODS FOR GENERATING TRAINEE GUIDANCE USING ADAPTIVE ARTIFICIAL INTELLIGENCE,” and filed on October 11, 2024 (the “’355 Application”), the contents of which are incorporated herein in its entirety by reference.

[0360] As illustrated, the multimodal data streams C104 may include force data indicating forces exerted on surgical instruments and / or repositionable structures. The multimodal data streams C104 may include event data indicating events suchAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONas instrument installs, insertions, removals, etc. The multimodal data streams 0104 may include kinematics data indicating the pose and / or movements of surgical instruments and / or repositionable structures. The multimodal data streams 0104 may include external video (e.g., via surgical room imaging devices) and procedure video (e.g., from a flexible surgical instrument such as a catheter), and / or any other suitable data. In some embodiments, the remote viewing system 0110 may include a multi-modal transformer type architecture that receives and processes the multimodal data streams 0104 (e.g., to update the VE to reflect information provided by the multimodal data streams 0104), although it should be appreciated that this architecture may be substituted or supplemented with other architectures including, but not limited to, convolutional neural network (CNN) architectures, visual transformers (VITs), recurrent / recursive neural network (RNN) architectures, sorting / clustering architectures, etc.

[0361] The surgical workstation 0115 may provide the multimodal data streams 0104 and / or preoperative data 0102 to the user (e.g., surgeon) intraoperatively. In at least some embodiments, the surgical workstation 0115 may access the VE (e.g., via the remote viewing system 0110) for simulating at least a portion of the surgical procedure. The remote viewing system 0110, may render and / or update the virtual environment and / or one or more digital twins based upon the preoperative data 0102 and the multimodal data streams 0104 respectively. The remote viewing system 0110 may simulate or otherwise depict one or more no-fly zones, surgical planes, surgical guidance, etc., via the VE. For example, the remote viewing system 0110 may fuse force data, external video data and procedural video data of the multimodal data streams 0104 with preoperative data 0102, and present the fused data to a user of the surgical workstation 0115. In such an example, this may allow the user of the surgical workstation 0115 to simulate a surgical step in the VE, and / or provide situational awareness and guidance intraoperatively during the actual surgical procedure. For example, the remote viewing system 0110 and / or the surgical workstation 0115 may assess the virtual performance of the procedural action within the VE to provide guidance for how to improve when performing the same procedural action in the real-world. To this end, the guidance may indicate a more optimal approach to the target work site, a more optimal amount of force to apply, an improved sequencing of steps, a set of common issues that may arise during the procedure, etc. Examples of how the remote viewing system C110 and / or the surgical workstation C115 may generate the guidance, including the content and / or modality of said guidance, are described in the ’355 Application.

[0362] In some embodiments, the guidance is provided by a virtual avatar that communicates the guidance in a manner that replicates an expert mentor that remotely observes the procedure. In these embodiments, the guidance generation module may be or include an LLM that is configured to accept a constitution that set forth a set of predefined personality traits to deliver the guidance in a manner tailored to the user. As described in the ’355 Application, a continuous learning process may be implemented to adapt the constitution to the user. In some embodiments, if the user still does not exhibit improved results, the remote viewing system C110 and / or the surgical workstation C115 may switch to a conventional remote mentoring system to ensure effective user learning.

[0363] In some embodiments, the remote viewing system C110 may present and / or update the virtual environment in response to a user input received at the surgical workstation C115. The user input may be received, for example, ata user interface and / or a handheld user input device communicative coupled to the surgical workstation C115. The user input device may be associated with one or more robotic systems (e.g., a robotic arm, an instrument coupled thereto, a robotic catheter, etc.). The surgical workstation C115 may convert input signals received via the user input device into one or more control signals for controlling the associated robotic equipment. Additionally, the surgical workstation C115 may provide the user input signals and / or the control signal to the remote viewing system C110 to update the VE in accordance therewith.

[0364] As described above, the surgeon may transition provide input data to the surgical workstation C115 to switch between real-world operation of the robotic equipment and virtual operation of robotic equipment within the VE hosted by the remote viewingAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONsystem C110. For example, the surgeon may transition from operating in the actual surgical environment to operating in the simulated VE via a GUI presented by the surgical workstation 0115, a physical switch or toggle associated with a user input, via a gaze tracking user interface, and / or any other suitable means. When switching modes, the surgical workstation 0115 may identify which streams of image data are currently presented on a display and cause the VE to initialize in a state that presents virtual equivalents of the displayed views. It should be appreciated because the surgical workstation 0115 may be continuously updating the remote viewing system 0110 with the data needed to keep the VE reflective of the state of the real-world procedure, switching between modes may occur with relatively low latency.

[0365] In some embodiments, the remote viewing system 0110 may periodically execute automated simulations of upcoming procedural steps within a copy of the VE, for example, at a dedicated simulation VE-specific chipset coupled to the remote viewing system 0110. In some embodiments, a compute partitioning coordinator (such as the compute partitioning coordinator A262) may ensure that processing tasks for supporting the VE do not appreciably affect processing tasks for performing the medical procedure. In some embodiments, the remote viewing system 0110 may include a chipset expansion module via which additional hardware units can be coupled to support the disclosed automated simulation techniques. The remote viewing system 0110 may execute the automated simulations, for example, when detecting the completion of a particular phase of the procedure, when detecting a prior automated simulation completed, and son. By executing the automated simulations in parallel to the hosted VE, the remote viewing system 0110 is able to validate that the real-world procedure is still on track of a positive patient outcome, identify any potential complications that might interfere with said positive patient outcome, and provide guidance on how to mitigate and / or avoid such complications.

[0366] In some embodiments, the automatic execution of the upcoming phases of the procedure is adapted based on the multimodal data C104 associated with the prior phases of the procedure. For example, the remote viewing system C110 may detect characteristics of how the user interacted with the robotic equipment during the prior phases (e.g., by detecting forces exerted thereupon, rates of advancement, sentimental states of procedure personnel, etc.) and cause the simulation of the upcoming phases to be performed under similar conditions that reflect how the surgical team has actually performed during the procedure to simulate the upcoming phases in a manner that is more likely to occur. As a result, the complications can be detected with greater accuracy and the guidance can be tailored to the specific personnel to improve likelihood of mitigating any potential complications.

[0367] For example, a controller of the VE may include a controller that assesses the virtual intraoperative multimodal data to autonomously generate tasks to be implemented by the robotic controllers. An example of such a controller for a real-world robotic controller is described in, U.S. Patent Application serial no. 63 / 663,539 entitled “MULTI-MODAL AND TASK-ADAPTIVE SYSTEM ARCHITECTURE FORAN INTELLIGENT SURGICAL ROBOT,” and filed on June 24, 2024 (the “’539 Application”). The planning workstation 0125 may implement similar techniques as described in the ’539 Application at a virtual controller for the robotic equipment within the VE. Accordingly, as the operator of the procedure performs the procedure, the remote viewing system 0110 may adapt the instructions provided to the virtual controller such that the virtual controller better reflects the actions that are likely to be performed in furtherance of the current medical procedure. As a result, the automated simulations are even further tailored to the specific procedure being performed, resulting in greater predictive performance for models that analyze the outcomes of the simulated procedures (such as those described elsewhere in herein).

[0368] While the foregoing set out preoperative and intraoperative usage of digital twins, it is also envisioned that the digital twin techniques can be utilized postoperatively as well. For example, at the completion of a procedure, a postoperative scan of the patient may be performed to generate a post-operative digital twin. Accordingly, the pre-operative digital twin can then be comparedAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONto the postoperative digital twin to, for example, evaluate the procedure and / or build a searchable database of digital twins for developing training materials. In these embodiments, the digital twins may be analyzed by various classification models to generate indexable labels to build a library of digital twins associated with the corresponding classes.Example Method for Presenting a Digital Twin

[0369] FIG. 4B depicts a flowchart of an example method C200for presenting a digital twin, according to embodiments. The method 400 may be performed, for example, via one or more processors of a computing device of the computing environment C100 (e.g., the surgical workstation C115 and / or a data connector integrated therewith).

[0370] The method C200 may include obtaining preoperative image data (e.g., the preoperative data C102) of a subject (block C210). The preoperative data may be retrieved or otherwise obtained from one or more sources, such as electronic medical records, a memory of the planning workstation C125, etc.

[0371] Based on the preoperative image data, the method C200 may include generating a three-dimensional (3D) model (e.g., digital twin) of the subject (block C220). In some embodiments, generating the 3D model includes transmitting the preoperative image data to an application hosted in a cloud (such an application 66, 68 of the cloud 50) and obtaining the model generated by the application. In other embodiments, the model is generated locally at the one or more processors.

[0372] The method C200 may include importing the 3D model into a virtual environment (block C230). The virtual environment includes a physics engine for modeling at least a portion of one or more of the 3D model or robotic surgical equipment.

[0373] The method C200 may include presenting the virtual environment to a viewer device (e.g., the surgical workstation C115, the remote client device C120, the planning workstation C125) (block C240).

[0374] In at least some embodiments, the method C200 may include presenting the virtual environment at planning computing device (e.g., the planning workstation C125); simulating an upcoming surgical procedure in the virtual environment; and annotating at least one model and / or task for presentation intraoperatively during the surgical procedure in a non-simulated environment. In some such embodiments, simulating the surgical procedure may include determining a potential task ordering for implementing the surgical procedure; simulating the potential task ordering; and providing an analysis of predicted outcomes based upon the simulated potential task ordering. As one example, the potential task ordering may be derived by analyzing historical procedure data to predict the potential task ordering. As another example, the potential task ordering is determined by obtaining intraoperative data (either real or virtual within the VE) of an intraoperative environment and inputting the intraoperative data into a task prediction model configured to analyze the intraoperative data to predict a subsequent task of the potential task ordering. For example, the virtual controller may include the task generation and / or task selection prediction model configured in the manner described in the ’395 application.

[0375] In at least some embodiments, the method C200 may include obtaining intraoperative data of an intraoperative environment; and updating the virtual environment based upon the intraoperative data. In some such embodiments, the method C200 may include presenting the intraoperative environment to one or more remote clients (e.g., the remote client device C120). In some such embodiments, the method C400 may include presenting the virtual environment at an operator workstation (e.g., the remote viewing system C110) in response to a user input. In some such embodiments, the method 400 may include converting an input of a user input device to one or more control signals (e.g., within the VE). In some embodiments, presenting the virtual environment includes presenting guidance, such as one or more of a no-fly zone, navigational guidance, a surgical plane, or surgical guidance.Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION

[0376] In at least some embodiments, the presenting the virtual environment includes generating an external model of the subject based on depth data included in the intraoperative data; aligning the 3D model within the external model of the subject; and presenting the virtual environment such that both the external model and the 3D model are viewable. In some embodiments, aligning the 3D model within the external model of the subject includes registering the 3D model to a coordinate system associated with the external model of the subject. To perform the registration, in some embodiments, the method C200 includes analyzing the depth data to detect an external feature of the subject; and registering the 3D model relative to the external feature of the subject. In some additional embodiments, the registration is performed by registering the preoperative image data to intraoperative 3D scan data of the patient; and registering the 3D model based on the registration of the preoperative image data. In still additional embodiments, the registration is performed by detecting an internal feature of the subject within the endoscopic image data; and registering the 3D model based on the location of the internal feature. In any of the approaches, the features the registration are based on may include bony features of the subject (e.g., a rib), a stricture, a carina, etc.

[0377] In at least some embodiments, the method C200 may include hosting the virtual environment on a VE GPU (e.g., the VE GPU 112) while processing multi-modal data (e.g., the multimodal data streams C104) for intraoperative control via a general-purpose GPU (e.g., the general-purpose GPU C114).

[0378] It should be understood that not all blocks of the exemplary method 0200 of FIG. 4B are required to be performed. Additionally, the method 0200 may include fewer, additional, and / or other steps than those depicted in FIG. 4B, including those described with respect to the other methods described herein.SYNTHETIC DATA

[0379] In addition to the above-described techniques for presenting digital twins in furtherance of a surgical procedure, the remote viewing system 0110 may also be configured to host a VE for the purpose of executing simulations for the purpose of training a classifier and / or an autonomous control policy. For example, a classifier may be trained to predict whether a procedure is likely to result in a successful patient outcome based on a current state of a pending procedure. Training such a classifier requires a significant amount of training data, often more than what can conventionally be obtained using historical data. Furthermore, different phases of a procedure may have different impacts on the likelihood of success. Accordingly, it may be beneficial to generate significantly more training data for these determinative portions of the procedure without necessarily having to perform a complete procedure. Hence, being able to identify and simulate these determinative portions of the procedure within the VE under different sets of conditions may enable the development of a training set of data that more accurately predicts an outcome of a given procedure. Furthermore, because the classifier may utilize an architecture that consumes the time-series relationship of intraoperative data, the classifier may be deployed to predictively detect and / or mitigate circumstances that may have otherwise led to less successful procedure outcome.

[0380] As another example, the various robotic equipment may be configured to interpret control instructions to enact an indicated response by controlling the setting of the various controlled motors, drives, and / or other electromechanical components. As equipment is updated to include new control parameters and / or equipment, the particular controls that implement the same control instruction may change. As a result, there is a need to develop updated control policies to autonomously convert the control instructions into the desired kinematic responses. Similarly, as higher levels of robotic autonomy are implemented in the robotic equipment controller, there may be a need to generate higher level autonomous control policies that issue the various control instructions sent to lower-level control components. Accordingly, the Ves described herein provide an environment where unsupervised learning techniques can be deployed to develop optimized control policies without causing subject harm during theAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATIONtraining process. Accordingly, as new robotic equipment is obtained (or as older robotic equipment is repaired), the VE may enable the develop of autonomous control policies faster and more accurately than conventional techniques.

[0381] It should be appreciated that while the VE generally replicates the real-world as close as possible, there may nonetheless be some minor differences in how the digital twins of robotic models physically respond to the same control commands as their real-world equivalents. Accordingly, to accountfor these differences, the planning workstation C125 may still implement a fine-tuning phase using training dummies and / or other training equipment to adapt the virtually-trained control policy to one that more closely responds to actual conditions. It should be appreciated that because of the general closeness of the VE at simulating the robotic equipment, the control policy generally needs a significantly shorter real-world training period than if the training were to solely rely on the training equipment.

[0382] Furthermore, in some aspects, some conventional digital twin techniques may only model external structures of objects within a monitored environment. However, in an medical context, it is important to model patient internal anatomy to derive medical context to, for example, generate tasks to treat the patient, determine how to navigate to a worksite, how to arrange equipment in the monitored environment, etc. Accordingly, by integrating the above-described digital twins of internal patient anatomy into a VE, the additional context is provided to support the improved generation of synthetic data. As a result, the improved efficiencies in execution of tasks described above (e.g., with respect to the omnischeduler) provide processing capabilities that support the improved generation of synthetic data.

[0383] The planning workstation C125 may access historical procedure data (such as the historical procedure data 62) to facilitate the disclosed training techniques. For example, after each procedure, the surgical workstation C115 that implemented the procedure may compile, process, and / or upload the set of multimodal data C104 for that procedure to a central server for storage and / or analysis. Accordingly, the disclosed techniques may utilize the historical procedure data to, for example, determine common actions performed with respect to different portions of a procedure, compare the actions to procedure outcome to identify which actions are most determinative on procedure outcome, generate scripts that implement actions derived from historical procedures within a virtual environment, and so on. As a result, the VE is able to be configured in a manner to generate training data that more accurately reflects real-world usage of the simulated components.

[0384] To increase the breadth of the training scenarios, the planning workstation C125 may combine different portions of the historical procedure data to create hybrid sets of historical data and / or utilize generative Al to introduce new scenarios into a set of historical procedure data. For example, the planning workstation C125 may wantto create awide array of different anatomical pathways, tissue conditions, subject diagnoses, equipment characteristics and / or combinations, and / or other characteristics that may change when performing the same procedure on a different subject and / or at a different location. This enables the trained classifiers and / or control policies to be able to process wider ranges of scenarios, resulting in improve trained models. When utilizing the generative Al to create new scenarios, the planning workstation C125 may generate event sequences that are similar to historical event sequences, generate 3D models of anatomical features and / or any abnormal features associated therewith (e.g., a lesion, a cancerous mass, a polyp, etc.). As a result, even if there is no exact set of historical data included in the database for a particular scenario, the planning workstation C125 may synthetically create several variations of the scenario in the VE to generate simulated procedure data associated therewith.Classifier TrainingAttorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION

[0385] In at least some embodiments, the planning workstation C125 may generate synthetic data for surgical procedure steps or portions most influential on the surgical procedure outcome. FIG. 4C depicts a block diagram C500 for training a classifier for predicting procedure outcome using synthetically generated data (e.g., data generated within a VE), according to embodiments.

[0386] The block diagram C500 begins when the planning workstation C125 or other suitable computing device (e.g., machine learning model training / operation server) receive, accesses, or otherwise obtain historical surgical procedure data C510. As described herein, the historical procedure data C510 may maintained at a cloud storage system (such as the (local) cloud 50) to which collections of historical procedure data are uploaded for storage. Each set of historical procedure data C510 may include various types of data, such as historical patient medical data (e.g., EMR, surgical outcomes), preoperative patient data, intraoperative multimodal data, personnel data (e.g., surgical performance, specializations, etc.), surgical equipment data (e.g., type of equipment, surgical robot scripts), and or other suitable data describing a procedure. It should be appreciated that the planning workstation C125 may train different classifiers for different procedure types. Accordingly, when obtaining the historical procedure data C510, the planning workstation C125 may filter the collection of historical procedure data based on an indication of procedure type associated therewith.

[0387] Based on the set of historical surgical procedure data, the planning workstation C125 may determine the portions of the surgical procedure that is most closely associated with an outcome of the procedure. It should be appreciated, that for some procedures, the success of the procedure may be determined at some point after the procedure has completed and the patient has progressed along a rehabilitation plan. Accordingly, the historical procedure data C510 may also include data indicative of a subject’s progress along their rehabilitation plan to be able to accurately assess procedure outcome.

[0388] Regardless, the planning workstation C125 may analyze the historical procedure data C510 to identify patterns and / or correlations between the sets of historical procedure data C510 that had successful patient outcomes and those that did not. In some embodiments, the unsuccessful procedures are further subdivide...

Claims

1. Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION CLAIMS2.What is claimed is:

1. A compute node configured to optimize intra-processor data transfer and processing, the compute node comprising:4.one or more processors, including at least one Graphics Processing Unit (GPU) and at least one other programmable logic device; and5.one or more memories storing one or more applications and instructions thereon that, when executed by the one or more processors, cause the compute node to:6.receive a sequence of sub-frame chunks of a frame captured by a sensor device, each sub-frame chunk of the sequence of sub-frame chunks representing a portion of the frame,7.determine one or more tasks associated with at least one application of the one or more applications to which the sequence of sub-frame chunks correspond,8.cause the at least one GPU or the at least one other programmable logic device to analyze one or more subframe chunks of the sequence of sub-frame chunks based on the one or more tasks,9.iteratively compile the frame based on sequential analysis of the at least one GPU or the at least one other programmable logic device of the one or more sub-frame chunks, and10.determine an outcome of the one or more tasks during the iterative compiling of the frame, wherein the outcome is associated with operation of a computer-assisted system.

2. The compute node of claim 1, wherein the instructions, when executed, further cause the compute node to: determine an expected position of a feature of interest within the frames captured by the sensor device, the feature of interest to be analyzed as part of an image processing algorithm that is a task of the one or more tasks;12.chunk one or more subsequent captured frames to create one or more priority chunks that include portions of the subsequent captured frames corresponding to the expected position of the feature of interest;13.determine the image processing algorithm associated with the one or more priority chunks; and14.cause the at least one GPU or the at least one other programmable logic device to analyze the one or more priority chunks based on the image processing algorithm and the expected position of the feature of interest.

3. The compute node of claim 2, wherein the instructions, when executed, further cause the compute node to: receiving data corresponding to at least one of (I) a model of lumens for an endoscope or (II) kinematic data of the endoscope; and16.determining the expected position of the feature of interest based on at least one of (I) the model of lumens or (II) the kinematic data.

4. The compute node of claim 1, wherein the instructions, when executed, further cause the compute node to cause the at least one GPU or the at least one other programmable logic device to analyze the one or more sub-frame chunks by:18.determining one or more processing requirements of the one or more tasks and processing requirements of the at least one application; and Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION19.allocating the one or more tasks between the at least one GPU and the at least one other programmable logic device based on respective processing capabilities of the at least one GPU and the at least one other programmable logic device compared to the one or more processing requirements of the one or more tasks and the at least one application.

5. The compute node of claim 4, wherein the one or more processing requirements correspond to a task of a medical procedure or a phase of the medical procedure.

6. The compute node of claim 1, wherein the sensor device is a medical device, and the frame and the sequence of sub-frame chunks are medical images.

7. The compute node of claim 1, wherein the instructions, when executed, further cause the compute node to: extend margins of each sub-frame chunk to include a margin of pixels from adjacent portions included in the sequence of sub-frame chunks; and23.merge extended margins of adjacent sub-frame chunks during the iterative compiling of the frame to avoid duplications or inconsistencies within the frame.

8. The compute node of claim 1, wherein the instructions, when executed, further cause the compute node to: modify one or more processing parameters for an analysis algorithm executed by the at least one GPU or the at least one other programmable logic device near boundaries of a sub-frame chunk; and25.cause the at least one GPU or the at least one other programmable logic device to analyze the sub-frame chunk in accordance with the one or more modified processing parameters for the analysis algorithm.

9. The compute node of claim 1, wherein the outcome of the one or more tasks is displayed on a collaboration interface for enabling collaboration with one or more external entities to optimize task scheduling and data transfer.

10. The compute node of claim 1, wherein the compute node is communicatively coupled with a cloud-based application server.

11. A system configured to optimize intra-processor data transfer and processing, the system comprising: one or more processors, including at least one Graphics Processing Unit (GPU) and at least one other programmable logic device; and29.one or more memories storing one or more applications and instructions thereon that, when executed by the one or more processors, cause the system to:30.receive a sequence of sub-frame chunks of a frame captured by a sensor device, each sub-frame chunk of the sequence of sub-frame chunks representing a portion of the frame,31.determine one or more tasks associated with at least one application of the one or more applications to which the sequence of sub-frame chunks correspond,32.cause the at least one GPU or the at least one other programmable logic device to analyze one or more subframe chunks of the sequence of sub-frame chunks based on the one or more tasks,33.iteratively compile the frame based on sequential analysis of the at least one GPU or the at least one other programmable logic device of the one or more sub-frame chunks, and Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION34.determine an outcome of the one or more tasks during the iterative compiling of the frame, wherein the outcome is associated with operation of a computer-assisted system.

12. The system of claim 11, further comprising a cloud-based application server that is communicatively coupled with a compute node that includes the one or more processors and the one or more memories.

13. The system of claim 12, wherein one or more applications are accessible by the compute node from the cloud-based application server, each application having associated tasks with different compute resource requirements, and the system further comprises:37.a compute partitioning coordinator configured to:38.determine (i) a set of tasks for the one or more applications to be executed by the compute node, (ii) a task prioritization level for each task of the set of tasks, and (ill) compute resource requirements for each task of the set of tasks,39.partition, based on (i)-(iii), the set of tasks into a task schedule that specifies a respective processing core of the GPU or the at least one other programmable logic device to perform execution of one or more of tasks of the set of tasks, and40.cause the GPU and the at least one other programmable logic device to execute the partitioned tasks in accordance with the task schedule.

14. The system of claim 12, wherein the compute node comprises at least one edge compute node, the cloudbased application server is a cloud server, and the compute node is connected to the cloud server.

15. The system of claim 13, wherein the compute node receives compute tasks from the cloud-based application server to execute as part of one or more of the applications accessible by the compute node.

16. The system of claim 13, wherein the computer partitioning coordinator is further configured to:44.create one or more virtual sandboxes within the at least one GPU, each sandbox isolating a respective task to minimize negative interactions between concurrent tasks.

17. The system of claim 16, wherein at least one virtual sandbox of the one or more virtual sandboxes corresponds to performing operations corresponding to a medical procedure requiring high timing consistency.

18. The system of claim 13, wherein the cloud-based application server is configured to:47.connect to a plurality of compute nodes requesting access to the one or more applications, wherein a first compute node and a second compute node request access to a first application of the one or more applications;48.determine a respective processing environment configuration for the first compute node and the second compute node indicating (i) a first compute processing capacity of the first compute node, (ii) a second compute processing capacity of the second compute node, (iii) a first compute processing hardware configuration of the first compute node, and (iv) a second compute processing hardware configuration of the second compute node; and49.compare the respective processing environment configurations of the first compute node and the second compute node to application processing configuration requirements for the first application, and perform at least one of: Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION50.provision access to the first application for the first compute node based on the first compute processing capacity and the first compute processing hardware configuration satisfying the processing configuration requirements for the first application, and51.deny access to the first application for the second compute node based on the second compute processing capacity or the second compute processing hardware configuration failing to satisfy the processing configuration requirements for the first application.

19. The system of claim 18, wherein a processing environment analysis module stored in the cloud-based application server is configured to selectively offer different services based on the determined configuration of the plurality of compute nodes.

20. A method for optimizing intra-processor data transfer and processing, comprising:54.receiving, at one or more processors, a sequence of sub-frame chunks of a frame captured by a sensor device, each sub-frame chunk of the sequence of sub-frame chunks representing a portion of the frame;55.determining, by the one or more processors, one or more tasks associated with at least one application of one or more applications to which the sequence of sub-frame chunks correspond;56.causing, by the one or more processors, at least one GPU or at least one other programmable logic device to analyze one or more sub-frame chunks of the sequence of sub-frame chunks based on the one or more tasks;57.iteratively compiling, by the one or more processors, the frame based on sequential analysis of the at least one GPU or the at least one other programmable logic device of the one or more sub-frame chunks; and58.determining, by the one or more processors, an outcome of the one or more tasks during the iterative compiling of the frame, wherein the outcome is associated with operation of a computer-assisted system.

21. A non-transitory computer-readable medium storing instructions for optimizing intra-processor data transfer and processing that, when executed, cause one or more processors of a compute node to perform the method of claim 20.

22. A system for optimizing compute resource management, the system comprising:61.a compute node including a central processing unit (CPU) and a graphics processing unit (GPU), the CPU and the GPU having a plurality of processing cores configured to execute computer-executable instructions;62.one or more applications accessible by the compute node from an application server, each application having associated tasks with different compute resource requirements; and63.a compute partitioning coordinator configured to:64.determine (i) a set of tasks for the one or more applications to be executed by the compute node, (ii) a task prioritization level for each task of the set of tasks, and (ill) compute resource requirements for each task of the set of tasks,65.partition, based on (i)-(iii), the set of tasks into a task schedule that specifies a respective processing core of the CPU or the GPU to perform execution of one or more of tasks of the set of tasks, and66.cause the GPU and the CPU to execute the partitioned tasks in accordance with the task schedule. Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION23. The system of claim 22, wherein the compute node comprises at least one edge compute node, the application server is a cloud-based server, and the compute node is connected to the cloud-based server.

24. The system of claim 22, wherein the compute node receives compute tasks from the application server to execute as part of one or more of the applications accessible by the compute node.

25. The system of claim 22, wherein the compute partitioning coordinator schedules at least one task for execution by the application server.

26. The system of claim 22, wherein the compute partitioning coordinator is further configured to: determine the task schedule for each application based on the task schedule for every other application to be executed by the compute node.

27. The system of claim 22, wherein the compute node further comprises one or more field-programmable gate arrays (FPGAs), and the compute partitioning coordinator is further configured to:72.determine one or more high priority tasks based on the task prioritization level of one or more tasks in the set of tasks; and73.partition the one or more high priority tasks for execution on the core of the GPU or the one or more FPGAs.

28. The system of claim 22, wherein the compute partitioning coordinator is configured to:75.partition the set of tasks based on (i)-(iii) and consolidating computation onto one or more particular GPUs that each satisfy a processing threshold.

29. The system of claim 22, wherein the compute partitioning coordinator is further configured to:77.create one or more virtual sandboxes within the GPU, each sandbox isolating a respective task to minimize negative interactions between concurrent tasks.

30. The system of claim 29, wherein at least one virtual sandbox of the one or more virtual sandboxes corresponds to performing operations corresponding to a medical procedure requiring high timing consistency.

31. The system of claim 22, wherein at least one of the tasks is executed as part of a kernel across one or more threads of the one or more GPU cores.

32. The system of claim 22, wherein the compute partitioning coordinator is further configured to dynamically adjust allocation of GPU resources based on the task prioritization levels and the compute resource requirements.

33. The system of claim 22, wherein at least one application of the one or more applications corresponds to a medical procedure, and the set of tasks correspond to processing tasks of medical image data acquired by an endoscope.Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION34. The system of claim 22, wherein the compute node receives a sequence of sub-frame chunks of a frame captured by a sensor device, each sub-frame chunk of the sequence of sub-frame chunks representing a portion of the frame, and the compute partitioning coordinator is further configured to:83.determine (i)-(iii) associated with the sequence of sub-frame chunks;84.partition the set of tasks associated with the sequence of sub-frame chunks into the task schedule; and85.cause the GPU or at least one other programmable logic device to execute the partitioned tasks in accordance with the task schedule.

35. The system of claim 34, wherein the task schedule further comprises:87.iteratively compile the frame based on sequential analysis of the at least one GPU or the at least one other programmable logic device of the sub-frame chunks; and88.determine an outcome of the one or more tasks during the iterative compiling of the frame, wherein the outcome is associated with operation of a computer-assisted system.

36. The system of claim 34, wherein the task schedule further comprises:90.determine an expected position of a feature of interest within the frames captured by the sensor device, the feature of interest to be analyzed as part of an image processing algorithm that is a task of the one or more tasks;91.chunk one or more subsequent captured frames to create one or more priority chunks that include portions of the subsequent captured frames corresponding to the expected position of the feature of interest;92.determine the image processing algorithm associated with the one or more priority chunks; and93.cause the at least one GPU or the at least one other programmable logic device to analyze the one or more priority chunks based on the image processing algorithm and the expected position of the feature of interest.

37. The system of claim 22, wherein a first compute node and a second compute node request access to a first application of the one or more applications, and the system further comprises a processing environment analysis module configured to:95.determine a respective processing environment configuration for the first compute node and the second compute node indicating (i) a first compute processing capacity of the first compute node, (ii) a second compute processing capacity of the second compute node, (iii) a first compute processing hardware configuration of the first compute node, and (iv) a second compute processing hardware configuration of the second compute node; and96.compare the respective processing environment configurations of the first compute node and the second compute node to application processing configuration requirements for the first application, to provision access for the first compute node or the second compute node.

38. The system of claim 37, wherein the processing environment analysis module is further configured to perform at least one of:98.provision access to the first application for the first compute node based on the first compute processing capacity and the first compute processing hardware configuration satisfying the processing configuration requirements for the first application; and Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION99.deny access to the first application for the second compute node based on the second compute processing capacity or the second compute processing hardware configuration failing to satisfy the processing configuration requirements for the first application.

39. The system of claim 37, wherein the processing environment analysis module is further configured to selectively offer different services based on the determined configuration of the plurality of compute nodes.

40. The system of claim 37, wherein the compute partitioning coordinator is configured to partition processing power of the compute nodes into multiple virtual sandboxes to isolate respective tasks and prevent negative interactions between different tasks.

41. The system of claim 22, wherein a first application of the one or more applications includes a machine learning algorithm, a second application of the one or more applications includes an image processing algorithm, and the task prioritization level for tasks of the image processing algorithm include real-time processing needs.

42. A method for optimizing compute resource management, comprising:104.determining, by one or more processors executing a compute partitioning coordinator, (i) a set of tasks for one or more applications to be executed by a compute node, (ii) a task prioritization level for each task of the set of tasks, and (ill) compute resource requirements for each task of the set of tasks;105.partition, by the one or more processors and based on (i)-(iii), the set of tasks into a task schedule that specifies a respective processing core of a CPU or a GPU of the compute node to perform execution of one or more of tasks of the set of tasks; and106.cause the GPU and the CPU to execute the partitioned tasks in accordance with the task schedule.

43. A non-transitory computer-readable medium storing instructions for optimizing compute resource management that, when executed, cause one or more processors to perform the method of claim 42.

44. A cloud-based application server for hardware-aware application management, the cloud-based application server comprising:109.one or more processors; and110.one or more memories storing one or more applications and a processing environment analysis module that, when executed by the one or more processors, cause the cloud-based application server to:111.connect to a plurality of compute nodes requesting access to the one or more applications, wherein a first compute node and a second compute node request access to a first application of the one or more applications, determine a respective processing environment configuration for the first compute node and the second compute node indicating (i) a first compute processing capacity of the first compute node, (ii) a second compute processing capacity of the second compute node, (iii) a first compute processing hardware configuration of the first compute node, and (iv) a second compute processing hardware configuration of the second compute node, and compare the respective processing environment configurations of the first compute node and the second compute node to application processing configuration requirements for the first application, and perform at least one of: Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION112.provision access to the first application for the first compute node based on the first compute processing capacity and the first compute processing hardware configuration satisfying the processing configuration requirements for the first application, and113.deny access to the first application for the second compute node based on the second compute processing capacity or the second compute processing hardware configuration failing to satisfy the processing configuration requirements for the first application.

45. The cloud-based application server of claim 44, wherein the processing environment analysis module is further configured to selectively offer different services based on the determined configuration of the plurality of compute nodes.

46. The cloud-based application server of claim 44, wherein one or more compute nodes of the plurality of compute nodes includes at least one of a Field Programmable Gate Array (FPGA) and a Graphics Processing Unit (GPU).

47. The cloud-based application server of claim 44, further comprising a compute partitioning coordinator configured to prioritize tasks between compute processing hardware of the plurality of compute nodes based on application processing configuration requirements and real-time processing needs.

48. The cloud-based application server of claim 47, wherein the applications include one or more machine learning algorithms and the server is further configured to:118.determine (i) tasks of the machine learning algorithms, (ii) corresponding processing resource requirements of the tasks, or (iii) predicted real-time processing needs of the tasks;119.wherein the application processing configuration requirements correspond to minimum hardware requirements to perform the tasks of the machine learning algorithm.

49. The cloud-based application server of claim 47, wherein the compute partitioning coordinator is configured to partition processing power of the compute nodes into multiple virtual sandboxes to isolate respective tasks and prevent negative interactions between different processing tasks.

50. The cloud-based application server of claim 44, wherein the processing environment analysis module, when executed, further cause the cloud-based application server to:122.offload processing tasks of the one or more applications to a cloud environment or a hybrid cloud environment based on processing demands and existing hardware capabilities of the compute nodes.

51. The cloud-based application server of claim 44, wherein the processing environment analysis module, when executed, further cause the cloud-based application server to:124.dynamically adjust provisioning of the one or more applications to respective compute nodes and processing task allocations based on (i) updated application processing configuration requirements, (ii) updated compute processing capacity of a compute node of the one or more compute nodes, or (iii) updated compute hardware configurations of the compute node of the one or more compute nodes. Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION52. The cloud-based application server of claim 44, wherein the one or more applications provisioned by the cloud-based server are configured to assist as part of a medical procedure, and the cloud-based application server is configured to maintain data security and data privacy of data transmitted to applications hosted on the server.

53. A method for hardware-aware application management, comprising:127.connecting, by one or more processors, to a plurality of compute nodes requesting access to the one or more applications, wherein a first compute node and a second compute node request access to a first application of the one or more applications;128.determining, by the one or more processors, a respective processing environment configuration for the first compute node and the second compute node indicating (i) a first compute processing capacity of the first compute node, (ii) a second compute processing capacity of the second compute node, (iii) a first compute processing hardware configuration of the first compute node, and (iv) a second compute processing hardware configuration of the second compute node; and comparing, by the one or more processors, the respective processing environment configurations of the first compute node and the second compute node to application processing configuration requirements for the first application, and perform at least one of:129.provisioning access to the first application for the first compute node based on the first compute processing capacity and the first compute processing hardware configuration satisfying the processing configuration requirements for the first application, and130.denying access to the first application for the second compute node based on the second compute processing capacity or the second compute processing hardware configuration failing to satisfy the processing configuration requirements for the first application.

54. The method of claim 53, further comprising:132.selectively offering, by the one or more processors, different services based on the determined configuration of the plurality of compute nodes.

55. The method of claim 53, wherein one or more compute nodes of the plurality of compute nodes includes at least one of a Field Programmable Gate Array (FPGA) and a Graphics Processing Unit (GPU).

56. The method of claim 53, further comprising:135.prioritizing, by the one or more processors, tasks between compute processing hardware of the plurality of compute nodes based on application processing configuration requirements and real-time processing needs.

57. The method of claim 56, wherein the applications include one or more machine learning algorithms and the method further comprises:137.determining, by the one or more processors, (i) tasks of the machine learning algorithms, (ii) corresponding processing resource requirements of the tasks, or (iii) predicted real-time processing needs of the tasks;138.wherein the application processing configuration requirements correspond to minimum hardware requirements to perform the tasks of the machine learning algorithm.

58. The method of claim 53, further comprising:Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION140.partitioning, by the one or more processors, processing power of the compute nodes into multiple virtual sandboxes to isolate respective tasks and prevent negative interactions between different processing tasks.

59. The method of claim 53, further comprising:142.offloading, by the one or more processors, processing tasks of the one or more applications to a cloud environment or a hybrid cloud environment based on processing demands and existing hardware capabilities of the compute nodes.

60. The method of claim 53, further comprising:144.dynamically adjusting, by the one or more processors, provisioning of the one or more applications to respective compute nodes and processing task allocations based on (i) updated application processing configuration requirements, (ii) updated compute processing capacity of a compute node of the one or more compute nodes, or (iii) updated compute hardware configurations of the compute node of the one or more compute nodes.

61. The method of claim 53, wherein the one or more applications are configured to assist as part of a medical procedure, and the method further comprises:146.maintaining, by the one or more processors, data security and data privacy of data transmitted to applications hosted on a server.

62. The method of claim 53, further comprising:148.receiving, at the one or more processors, a sequence of sub-frame chunks of a frame captured by a sensor device, each sub-frame chunk of the sequence of sub-frame chunks representing a portion of the frame;149.determining, by the one or more processors, one or more tasks associated with at least one application of the one or more applications to which the sequence of sub-frame chunks correspond;150.causing, by the one or more processors, at least one GPU or at least one other programmable logic device to analyze one or more sub-frame chunks of the sequence of sub-frame chunks based on the one or more tasks;151.iteratively compiling, by the one or more processors, the frame based on sequential analysis of the at least one GPU or the at least one other programmable logic device of the one or more sub-frame chunks; and152.determining, by the one or more processors, an outcome of the one or more tasks during the iterative compiling of the frame, wherein the outcome is associated with operation of a computer-assisted system.

63. The method of claim 62, further comprising:154.determining, by the one or more processors, an expected position of a feature of interest within the frames captured by the sensor device, the feature of interest to be analyzed as part of an image processing algorithm that is a task of the one or more tasks;155.chunking, by the one or more processors, one or more subsequent captured frames to create one or more priority chunks that include portions of the subsequent captured frames corresponding to the expected position of the feature of interest;156.determining, by the one or more processors, the image processing algorithm associated with the one or more priority chunks; and157.causing, by the one or more processors, the at least one GPU or the at least one other programmable logic device to analyze the one or more priority chunks based on the image processing algorithm and the expected position of the feature of interest. Attorney Docket: 33685 / 70820 / PC Intuitive Docker: P07030-WO PATENT APPLICATION64. The method of claim 53, further comprising:159.determining, by the one or more processors executing a compute partitioning coordinator, (i) a set of tasks for one or more applications to be executed by a compute node, (ii) a task prioritization level for each task of the set of tasks, and (ill) compute resource requirements for each task of the set of tasks;160.partitioning, by the one or more processors and based on (i)-(iii), the set of tasks into a task schedule that specifies a respective processing core of a CPU or a GPU of the compute node to perform execution of one or more of tasks of the set of tasks; and161.causing the GPU and the CPU to execute the partitioned tasks in accordance with the task schedule.

65. A non-transitory computer-readable medium storing instructions for hardware-aware application management that, when executed, cause one or more processors to perform the method of any of claims 53-64.

Images