Machine learning based detection of bleeding induced by an electrosurgical device from video data

The online medical system synchronizes and analyzes electrosurgical device data to detect bleeding events using machine-learning, enhancing surgical skill assessment and patient care through effective bleeding event monitoring.

WO2026234640A1PCT designated stage Publication Date: 2026-11-12CILAG GMBH INTERNATIONAL
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/IB2026/054588
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-09
Filing Date
2026-05-08
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Existing electrosurgical devices lack effective monitoring for bleeding events during medical procedures, which are critical for surgical skill assessment and patient care.

Method used

An online medical system that synchronizes operational data and video data to detect bleeding events using machine-learning models, analyzing pre- and post-activation frames to determine bleed metrics and classify whether device activation caused bleeding.

Benefits of technology

Accurately identifies bleeding events, providing valuable feedback for surgical skill improvement and patient care by reducing the likelihood of future bleeding occurrences.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure IB2026054588_12112026_PF_FP_ABST
    Figure IB2026054588_12112026_PF_FP_ABST
Patent Text Reader

Abstract

An online medical system receives operational data of an electrosurgical device during a medical procedure and video data including at least a portion of the electrosurgical device during the medical procedure. The online medical system temporally synchronizes the operational data and the video data, then determines an activation time and a deactivation time of the electrosurgical device from the operational data. Based on the activation time and the deactivation time, the online medical system extracts a pre-activation set of frames and a post- activation set of frames from the video data. Based on characterization of blood in frames of the pre-activation set and in frames of the post-activation set by a machine-learning model, the online medical system generates one or more metrics for determining whether activation of the electrosurgical device caused bleeding.
Need to check novelty before this filing date? Find Prior Art

Description

MACHINE LEARNING BASED DETECTION OF BLEEDING INDUCED BY AN ELECTROSURGICAL DEVICE FROM VIDEO DATACROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 802,957 filed on May 9, 2026, which is incorporated by reference herein.BACKGROUNDTECHNICAL FIELD

[0002] The described embodiments relate to analyzing video data depicting operation of an electrosurgical device during a medical procedure to determine whether activation of the electrosurgical device caused bleeding.DESCRIPTION OF THE RELATED ART

[0003] Electrosurgical devices are surgical instruments used in various medical procedures. An electrosurgical device uses high frequency electric current to heat tissue for purposes of cutting tissue, ablating tissue, to desiccating tissue, or performing other actions. An electrosurgical device includes a generator generating high frequency electric current that is transmitted to an active electrode. The active electrode contacts tissue of the patient to locally heat a portion of the patient’s tissue contacting the active electrode.

[0004] Adverse events, such as bleeding or tissue damage, may occur from inadequate application of the electrosurgical device during a medical procedure. Monitoring electrosurgical device-induced bleeding during one or more medical procedures may provide important information for surgical skill assessment, improving device application, and contributing to patient care.BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure (FIG.) 1 is an example embodiment of a computing environment capturing data describing operation of an electrosurgical device.

[0006] FIG. 2 is an example block diagram of an online medical system determining whether operation of an electrosurgical device during a medical procedure caused bleeding.

[0007] FIG. 3 is a flowchart of an example embodiment of a method for determining whether 1 JNJ-046WO / VRB5194WOPCT1operation of an electrosurgical device during a medical procedure caused bleeding.

[0008] FIG. 4 is an example timing diagram of an online medical system extracting a set of preactivation frames and a set of post-activation frames of video data captured during a medical procedure to evaluate operation of an electrosurgical device.DETAILED DESCRIPTION

[0009] The Figures (FIGS.) and the following description describe certain embodiments by way of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein. Reference will now be made to several embodiments, examples of which are illustrated in the accompanying figures. Wherever practicable, similar or like reference numbers may be used in the figures and may indicate similar or like functionality.

[0010] An electrosurgical device uses high frequency electric current applied to tissue of a patient to heat the tissue of the patient during a medical procedure for tissue dissection and sealing. To evaluate operation of the electrosurgical device during the medical procedure, an online medical system obtains operational data (e.g., energy usage data or other activation / deactivation indicators) of the electrosurgical device and obtains video data of the procedure performed using the electrosurgical device. The operational data may include current, voltage, power, or other electrical characteristics of the electrosurgical device at different times during the procedure. Alternatively, the operational data may include encoded activation or deactivation signals generated by the electrosurgical device. The video data comprises video depicting the surgical area and operation of the electrosurgical device. For example, the video data includes a tip of an electrode of the electrosurgical device. The online medical system temporally synchronizes the operational data and the video data to align their respective time indices.

[0011] Based on the operational data, the online medical system determines an activation time of the electrosurgical device and determines a deactivation time of the electrosurgical device. For example, the online medical system determines the activation time as a time when an electrical current used by the electrosurgical device equals or exceeds a threshold value associated with device activation. Similarly, the online medical system determines the deactivation time as a time when the electrical current used by the electrosurgical device is less than the threshold value. To generate one or more metrics to evaluate operation of the electrosurgical device, the online medical system extracts a pre-activation set of frames from the video data based on the activation time. The pre-activation set of frames are frames of the video data corresponding to a 2 JNJ-046WO / VRB5194WOPCT1specific time interval before the activation time. The online medical system similarly extracts a post-activation set of frames from the video data based on the deactivation time. The postactivation set of frames are frames of the video data corresponding to a particular time interval after the deactivation time.

[0012] Through application of one or more machine-learning models to the set of pre-activation frames, the online medical system determines one or more pre-activation bleeding metrics. In various embodiments, a pre-activation bleeding metric is based on a number of blood pixels in the pre-activation frames identified as depicting blood by a machine -learning model. Similarly, the online medical system determines one or more post-activation bleeding metrics by applying the one or more machine-learning models to the set of post-activation frames. In various embodiments a post-activation bleeding metric is based on a number of blood pixels in the postactivation frames identified as depicting blood by the machine-learning model. Based on the one or more pre-activation bleeding metrics and the one or more post-activation bleeding metrics, the online medical system determines one or more bleed event metrics. In various embodiments, a bleed event metric comprises a ratio of a post-activation bleeding metric to a pre-activation bleeding metric. The online medical system determines multiple bleed event metrics based on different pre-activation bleeding metrics and post-activation bleeding metrics in some embodiments.

[0013] Based on one or more bleed event metrics, the online medical system generates a classification result of whether activation of the electrosurgical device caused bleeding (e.g., based on a classifier). For example, the online medical system may store a value, a flag, a likelihood, or other information associated with the activation event of the electrosurgical device characterizing the inference of whether or not the electrosurgical device caused a bleeding event. Further, the one or more bleed event metrics may be used to subsequently determine configuration settings or to modify a control algorithm of a surgical robot using an electrosurgical device to reduce a likelihood of the surgical robot causing bleeding when operating the electrosurgical device Other information may be generated based on the one or more bleed event metrics in various embodiments.

[0014] FIG. 1 illustrates an example embodiment of a computing environment 100 in which an online medical system 130 captures data describing operation of an electrosurgical device 105. The online medical system 130 may include one or more servers coupled by a network 120 to an electrosurgical device 105 and to an image capture device 115. The online medical system 130 may be coupled to additional devices, such as one or more client devices or one or more third party systems via the network 120 in various embodiments. One or more third party systems may provide various services, such as electronic healthcare records (EHR) services, medical3 JNJ-046WO / VRB5194WOPCT1telepresence services, operating room scheduling, data analytics services, etc. in various embodiments.

[0015] The online medical system 130 may be implemented using on-site computing or storage systems, cloud computing or storage systems, or a combination thereof. The online medical system 130 may be implemented using local or cloud-based servers, which may include physical or virtual machines, or a combination thereof. Cloud-based servers may include private cloud systems, public cloud systems, hybrid public / private cloud systems, or a combination thereof. Accordingly, the online medical system 130 may be local, remote, and / or distributed relative to the medical environments where medical procedures are performed and relative to one or more client devices providing user access. Furthermore, different portions of the online medical system 130 may execute on different remote servers and various system elements of the online medical system 130 may be communicatively coupled over the network 120.

[0016] The electrosurgical device 105 uses high frequency electric current to heat tissue of a patient for purposes of cutting and / or coagulating tissue. In various embodiments, the electrosurgical device 105 includes a generator, an active electrode, and a return electrode. The generator generates high frequency electric current from a power source that is transmitted to the active electrode, which contacts tissue of the patient. High frequency electric current passes from the active electrode through the contacted portion of the patient’s tissue to the return electrode. Current passing through the portion of the patient’s tissue contacting the active electrode heats the patient’s contacted tissue. Heating may be controlled in various ways to effectuate cutting tissue, ablating tissue, desiccating tissue, or performing other actions.

[0017] In various embodiments, the generator generates high frequency current for the active electrode in response to receiving an activation signal. For example, the generator continuously generates high frequency current transmitted to the active electrode while the generator receives the activation signal, and the generator stops generating high frequency current when the activation signal is no longer received. As another example, the generator generates high frequency current in response to receiving the activation signal and stops generating high frequency current in response to receiving a separate deactivation signal.

[0018] A current monitor 110 coupled to the electrosurgical device captures energy usage data of the electrosurgical device 105. In various embodiments, the energy usage data comprises associations between amounts of energy (e.g., amounts of current) used by the electrosurgical device 105 and different times. For example, the energy data comprises amounts of current used by the electrosurgical device 105 at various times. In various embodiments, the current monitor 110 comprises an ammeter coupled to the generator of the electrosurgical device 105. For example, the current monitor 110 is a current clamp that is clamped around wiring providing 4 JNJ-046WO / VRB5194WOPCT1power to the generator from a power source or around other wiring of the electrosurgical device 105. The current clamp detects a magnetic field generated by current passing through the wiring around which the current clamp is clamped to determine an amperage within the wiring. For example, the current monitor 110 captures an amperage generated by the electrosurgical device 105 at different times. In other embodiments, the current monitor 110 is included in a generator of the electrosurgical device, or is otherwise included in the electrosurgical device 105.

[0019] In some embodiments, the current monitor 110 is coupled to the network 120 and transmits the energy usage data to the online medical system 130 via the network 120. For example, the current monitor 110 transmits the energy usage data to the online medical system 130 via the network 120 in response to a specific input from a user, such as a medical practitioner. Alternatively, the current monitor 110 continuously transmits the energy usage data to the online medical system 130 via the network 120. In other embodiments, the current monitor 110 is coupled to a client device; the client device receives the energy usage data from the current monitor 110 and transmits the energy usage data to the online medical system 130. The energy usage data includes a procedure identifier to uniquely identify a medical procedure during which the energy usage data was captured in various embodiments. In other embodiments, the electrosurgical device 105 directly generates an activation signal upon activating and a deactivation signal upon deactivating (e.g., encoded as operational data, such as, telemetry data associated with the electrosurgical device 105). For example, activation and deactivation signals may be directly sensed from the trigger mechanism without necessarily monitoring energy usage. The electrosurgical device 105 may identify an activation time and a deactivation time in operational data captured or generated by the electrosurgical device 105 during operation in various embodiments.

[0020] Additionally, an image capture device 115 captures video data of the medical procedure. The image capture device 115 may be positioned to capture a tissue region where the electrosurgical device 105 is applied, for example, by tracking a tip of the active electrode of the electrosurgical device 105. In various embodiments, the image capture device 115 comprises a laparoscopic camera or an endoscope positioned to have a field of view including a tip of the active electrode of the electrosurgical device 105 and surrounding tissue proximate to the tip of the active electrode. The video data captured by the image capture device 115 comprises a plurality of frames, each frame corresponding to a different time.

[0021] In various embodiments, the image capture device 115 captures video data during a medical procedure where the electrosurgical device 105 is used and transmits the video data to the online medical system 130 via the network 120. In some embodiments, the image capture device 115 is coupled to a client device or to a storage device, and the client device or the storage 5 JNJ-046WO / VRB5194WOPCT1device transmits the video data to the online medical system 130 via the network 120. The image capture device 115 may transmit the video data to the online medical system 130 via the network 120 in response to receiving a specific input from a user or may continuously transmit the video data to the online medical system 130 in various embodiments. The video data includes a procedure identifier to uniquely identify a medical procedure during which the video data was captured in various embodiments.

[0022] The network 120 comprises communication pathways for communication between the electrosurgical device 105, the image capture device 115, and the online medical system 130. Further, the network 120 enables communication between the online medical system 130 and other computing devices, such as client devices or third party systems, in various embodiments. The network 120 may include one or more local area networks and / or one or more wide area networks (including the Internet). The network 120 may also include one or more direct wired or wireless connections (e.g., Ethernet, WiFi, cellular protocols, WiFi direct, Bluetooth, Universal Serial Bus (USB), or other communication link).

[0023] The online medical system 130, further described below in conjunction with FIG. 2, receives the energy usage data from the current monitor 110 and the video data from the image capture device 115 and generates one or more metrics based on the energy usage data and the video data. As further described below in conjunction with FIGS. 2-4, the online medical system 130 determines an activation time of the electrosurgical device 105 based on the energy usage data and determines a deactivation time of the electrosurgical device 105 based on the energy usage data. Based on the activation time, the online medical system 130 extracts a pre-activation set of frames of the video data that comprises frames of video data earlier than the activation time (e.g., frames of video data corresponding to a specific time interval earlier than the activation time). Similarly, the online medical system 130 extracts a post-activation set of frames from the video data based on the deactivation time. In various embodiments, the post-activation set of frames are frames of the video data later than the deactivation time (e.g., frames of video data corresponding to a particular time interval after the deactivation time).

[0024] In further embodiments, the electrosurgical device 105 may be communicatively coupled to the online medical system 130 and / or the image capture device 115 and may directly output operational data, such as telemetry data or energy usage data, encoding the activation time or deactivation time. In one such implementation, the operational data may be received separately by the online medical system 130 and synchronized to identify video frames corresponding to the activation and deactivation times. In other embodiments, the electrosurgical device 105 and the image capture device 115 may be communicatively coupled to enable capturing of video data and operational data in a synchronized manner. The synchronized video data and operational data 6 JNJ-046WO / VRB5194WOPCT1may be received by the online medical system 130 as a set of synchronized signals or as a single signal that encodes both the video and telemetry data in various embodiments.

[0025] The online medical system 130 applies one or more machine-learning models to the preactivation set of frames to determine one or more pre-activation bleeding metrics. Similarly, the online medical system 130 applies one or more machine-learning models to the post-activation set of frames to determine one or more post-activation bleeding metrics. As further described below in conjunction with FIGS. 2-4, the pre-activation bleeding metric is based on a number of pixels in one or more frames of the pre-activation set of frames that the machine-learning model characterizes as depicting blood. Similarly, the post-activation bleeding metric is based on a number of pixels in one or more frames of the post-activation set of frames that the machine-learned model characterizes as depicting blood. Hence, the pre-activation bleeding metric provides a measure of an amount of blood included in one or more frames of the video data captured before the electrosurgical device 105 was activated, and the post-activation bleeding metric provides a measure of an amount of blood included in frames of the video data captured after the electrosurgical device 105 was deactivated.

[0026] As further described below in conjunction with FIGS. 2-4, based on one or more pre-activation bleeding metrics and one or more post-activation bleeding metrics, the online medical system 130 determines one or more bleed event metrics corresponding to activation of the electrosurgical device between the activation time of the electrosurgical device 105 and the deactivation time. In various embodiments, a bleed event metric comprises an indication of whether a greater amount of blood was included in frames of the video data after deactivation of the electrosurgical device 105 than before activation of the electrosurgical device 105. For example, a bleed event metric is a ratio of a post-activation bleeding metric to a pre-activation bleeding metric. The online medical system 130 uses one or more blood event metrics to determine whether activation of the electrosurgical device 105 at the activation time caused bleeding in the patient, as further described below in conjunction with FIGS. 2 and 3.

[0027] In various embodiments, the online medical system 130 selects and recommends training material for a medical practitioner to review in response to detecting bleeding events related to activation of the electrosurgical device 105 to educating the practitioner on possible causes of the bleeding events or techniques for reducing future occurrences of bleeding events. In various embodiments, the training material comprises video data, audio data, or text data describing operation or configuration of the electrosurgical device. The training material may be tailored to different causes of bleeding events, different types of medical procedure, associated with bleeding events, different types of electrosurgical devices 105 associated with the bleeding events, or other characteristics of the detected bleed events.7 JNJ-046WO / VRB5194WOPCT1

[0028] For example, the online medical system 130 applies a control model to the one or more bleed event metrics, or to a classification result whether activation of the electrosurgical device 105 caused bleeding. The classification model generates configuration information or modifications to a control process for a surgical robot using the electrosurgical device 105. The configuration information or modifications to the control process modify operation of the electrosurgical device 105 to reduce a likelihood of bleeding occurring from operation of the electrosurgical device 105 during a medical procedure. The online medical system 130 transmits the configuration information or the modified control process to the surgical robot for subsequent implementation in various embodiments. For example, the surgical robot modifies one or more configuration settings or modifies a stored control process based on the received information from the online medical system 130, modifying subsequent operation of the electrosurgical device 105 based on the information received from the online medical system 130.

[0029] Training material may include recommended configuration information for configuring the electrosurgical device 105 in a manner that may reduce future bleeding events from activation of the electrosurgical device 105. For example, training information specifies electrical current parameters for operating the electrosurgical device 105 or settings for configuring other aspects of the electrosurgical device 105. As another example, training information includes calibration instructions for calibrating the electrosurgical device 105. A user may transmit or store the configuration information to the electrosurgical device 105 to simplify subsequent configuration of the electrosurgical device 105 for performing a subsequent medical procedure. In another embodiment, the electrosurgical device 105 may be automatically reconfigured or recalibrated based on the training material identified by the online medical system 130 in response to detecting bleeding events from activation of the electrosurgical device 105.

[0030] FIG. 2 is a block diagram of an example architecture of an online medical system 130. In the example of FIG. 2, the online medical system 130 includes a synchronization module 205, a device activation detection module 210, a blood detection module 215, and a procedure evaluation module 220. Alternative embodiments may include more, fewer, or different components from those illustrated in FIG. 2, and the functionality of each component may be divided between the components differently from the description below. Additionally, each component may perform their respective functionalities in response to a request from a human, or automatically without human intervention in various embodiments.

[0031] The synchronization module 205 obtains operational data from the electrosurgical device 105. In various embodiments, the operational data comprises energy usage data from a current monitor 110 captured during operation of an electrosurgical device 105. Alternatively, the operational data is captured by the electrosurgical device 105 during operation. For example, the 8 JNJ-046WO / VRB5194WOPCT1operational data includes activation signals or deactivation signals received by the electrosurgical device during a medical procedure. The synchronization module 205 also obtains video data from an image capture device 115 captured during the medical procedure. As further described above in conjunction with FIG. 1, the video data comprises frames depicting operating of the electrosurgical device 105 at different times. For example, the synchronization module 205 receives video data and energy usage data both including a common identifier of a medical procedure and temporally synchronizes the video data and the energy usage data so times in the video data match times in the energy usage data. This correlates amounts of current provided to the electrosurgical device at a time with a frame of the video data corresponding to the time in various embodiments. In other embodiments, the synchronization module 205 synchronizes operational data from the electrosurgical device 105 and the video data so activation signals and deactivation signals are temporally synchronized with corresponding frames of the video data. The temporal synchronization enables identification of operational data (e.g., energy usage data) corresponding to one or more particular frames of video data and vice versa. The synchronization module 205 stores the temporally synchronized operational data (e.g., energy usage data) and video data for subsequent retrieval.

[0032] The device activation detection module 210 obtains the operational data, such as energy usage data, and determines an activation time of the electrosurgical device 105 from the operational data (e.g., energy usage data). Similarly, the device activation detection module 210 determines a deactivation time of the electrosurgical device 105 from the operational data (e.g., energy usage data). In embodiments where the operational data comprises energy usage data including a current amount (e.g., an amperage) associated with each of a plurality of times, the device activation detection module 210 maintains a threshold current. In some embodiments, the device activation detection module 210 stores different threshold currents in association with different types of electrosurgical devices 105 to account for variations in operation of different electrosurgical devices 105. For example, the device activation detection module 210 extracts an identifier of the electrosurgical device 105 from the energy usage data and retrieves a threshold current stored in association with the extracted identifier.

[0033] The device activation detection module 210 compares current amount (e.g., amperages) associated with each time in the energy usage data to the threshold current. In response to determining a current amount (e.g., an amperage) at a specific time equals or exceeds the threshold current and an amperage at a consecutive time earlier than the specific time is less than the threshold current, the device activation detection module 210 determines the specific time is an activation time of the electrosurgical device 105. The device activation detection module 210 stores the activation time. Similarly, in response to determining a current amount (e.g., an9 JNJ-046WO / VRB5194WOPCT1amperage) at a particular time is less than the threshold current while an amperage at a consecutive time earlier than the particular time equals or exceeds the threshold current, the device activation detection module 210 determines the particular time is a deactivation time of the electrosurgical device 105. The device activation detection module 210 stores the deactivation time. In various embodiments, the device activation detection module 210 stores the activation time and the deactivation time in association of an identifier of a medical procedure during which the energy usage data was captured.

[0034] In other embodiments, the device activation detection module 210 stores a threshold amount of current change in association with each of a set of identifiers of electrosurgical devices 105. The device activation module retrieves a threshold amount of current change stored in association with an identifier of the electrosurgical surgical device 105 included in the energy usage data. The device activation detection module 210 determines a change in current amount (e.g., amperage) between consecutive times in the energy usage data and compares the change in amperage to the threshold amount of current change. In response to the change in current amount being positive and having a magnitude equaling or exceeding the threshold amount of current change, the device activation detection module 210 determines a later time of the consecutive times as an activation time of the electrosurgical device 105. Similarly, in response to the change in current being negative and having a magnitude equaling or exceeding the threshold amount of current change, the device activation detection module 210 determines a later time of the consecutive times as a deactivation time of the electrosurgical device 105. The device activation detection module 210 stores the activation time and the deactivation time in association with an identifier of the medical procedure during which the energy usage data was captured.

[0035] In some embodiments, the device activation detection module 210 determines the activation time and the deactivation time from timing information included in the operational data. For example, the operational data includes a time corresponding to an activation signal and a time corresponding to a deactivation signal. The activation detection module 210 determines the activation time as a time corresponding to the activation signal the deactivation time as the time corresponding to the deactivation signal. In other embodiments, the device activation detection module 210 may determine the activation time and the deactivation time using one or more additional or alternative methods applied to the operational data.

[0036] Based on an activation time and a deactivation time from the device activation detection module 210, the blood detection module 215 obtains video data temporally synchronized with the operational data (e.g., energy usage data) from which the device activation detection module 210 determined the activation time and the deactivation time. The blood detection module 21510 JNJ-046WO / VRB5194WOPCT1identifies a video frame corresponding to the activation time from the video data and extracts a pre-activation set of frames from the video data based on the activation time. In various embodiments, the blood detection module 215 extracts frames corresponding to times earlier than the activation time (e.g., times within a specific time interval earlier than the activation time) as the pre-activation set of frames. For example, the blood detection module 215 maintains a predefined pre-activation window length and extracts frames of the video data corresponding to times within the pre-activation window length prior to the activation time.

[0037] Similarly, the blood detection module 215 identifies a video frame corresponding to the deactivation time from the video data and extracts a post-activation set of frames from the video data based on the deactivation time. In various embodiments, the blood detection module 215 extracts frames later than the deactivation time (e.g., times within a specific post-activation window length after the deactivation time) as the post-activation set of frames. For example, the blood detection module 215 maintains a predefined post-activation window length and extracts frames of the video data corresponding to times after the deactivation time within the postactivation window length. In various embodiments, the pre-activation window length equals the post-activation window length, while in other embodiments, the pre-activation window length and the post-activation window length are different. For example, the pre-activation window length is less than the post-activation window length, so the pre-activation set of frames includes frames from a smaller time interval than the post-activation set of frames.

[0038] For each frame in the pre-activation set of frames, the blood detection module 215 detects regions in a frame depicting blood. In various embodiments, the blood detection module 215 applies a trained machine-learning model to each frame, and the machine-learning model analyzes the frame to classify pixels in the frame as either blood pixels or non-blood pixels based on parameters from training on past training images including annotated pixels. Here, the blood pixels comprise pixels having sufficiently high likelihood (above a threshold likelihood) of depicting blood while the non-blood pixels comprise pixels without sufficiently high likelihood (below the threshold likelihood) of depicting blood. The machine-learning model may be a feature prediction model including a neural network or other machine leaming / parallel processing system to automatically detect features in data items, such as frames of video data. The machine-learning model may be a neural network having any number of layers in various embodiments. An example machine-learning model for computer vision is a residual neural network (ResNet) such as ResNet 18 with 18 layers. The machine-learning model may include one or more convolutional layers. The machine -learning model may include one or more feedforward paths or skips.11 JNJ-046WO / VRB5194WOPCT1

[0039] The machine -learning model includes a set of parameters used to process an input to generate an output. For example, a set of parameters for a linear regression model includes weights applied to each input variable in the linear combination comprising the linear regression model. Similarly, the set of parameters for a neural network may include weights and biases that are applied at each neuron in the neural network. In some embodiments, the blood detection module 215 generates the set of parameters (e.g., the particular values of the parameters) for a machine-learning model by “training” the machine -learning model. Once trained, the machinelearning model uses the set of parameters to transform inputs into outputs.

[0040] For example, the blood detection module 215 trains the machine -learning module based on a set of training examples. Each training example includes input data to which the machinelearning model is applied to generate an output. For example, each training example comprises an image in which blood pixels of the image are labeled. In these cases, the machine -learning model is trained by comparing its output when applied to a training example to the labels for the training example. In general, during training with labeled data, the set of parameters of the model may be set or adjusted to reduce a difference between the output for the training example (given the current parameters of the model) and the labels for the training example.

[0041] The blood detection module 215 may apply an iterative process to train the machinelearning model that updates parameter values of the machine -learning model based on each of the set of training examples. The training examples may be processed together, individually, or in batches. To train the machine-learning model based on a training example, the blood detection module 215 applies the machine -learning model to input data comprising a training example to generate an output based on a current set of parameter values. The blood detection module 215 scores the output from the machine -learning model using a loss function. A loss function is a function generating a score for the output of the machine -learning model so the score is higher when the machine-learning model performs poorly and lower when the machine-learning model performs well. In various embodiments, the loss function determines a difference between the set of pixels of a training example identified by the machine-learning model as blood pixels and the set of pixels of the training example labeled as blood pixels. Example loss functions include a mean square error function, a mean absolute error, a hinge loss function, and a cross entropy loss function. The blood detection module 215 updates the set of parameters for the machinelearning model based on the score generated by the loss function. For example, the blood detection module 215 may apply gradient descent to update the set of parameters.

[0042] In various embodiments, the machine-learning model comprises a random forest classifier (RFC) that detects blood pixels based on one or more color features. The machine -learning model uses a red, green, blue representation of color, a hue, saturation, value representation of 12 JNJ-046WO / VRB5194WOPCT1color, or a hue, saturation, light separation representation of color in various embodiments. For example, the machine-learning model determines features for a pixel of an image based on between different values (e.g., a red value, a blue value, a green value, a hue value, a saturation value, etc.). As an example, one or more features of a pixel comprise a ratio of a red value to a sum of a red value, a blue value, and a green value; a ratio of a green value to a red value; a ratio of a blue value to a sum of a red value, a blue value, and a green value; or other quantities based on one or more color values of the pixel. The machine-learning model receives one or more features of pixels of an image as input in various embodiments.

[0043] In other implementations, instead of characterizing individual pixels as blood pixels, the machine learning model may operate on regions within the image that may each include an array of pixels. For example, the machine learning model may classify each multi-pixel region as either a blood region or a non-blood region. In this implementation, the color values for the region may comprise an average pixel value, a median pixel value, or some other function of the pixel values within the region.

[0044] In various embodiments, the blood detection module 215 obtains the machine-learning model from a third party system or from a client device and stores the machine-learning model for subsequent application. Hence, the machine-learning model may be previously trained before being obtained and applied by the blood detection module 215. Further, the blood detection module 215 retrains the machine-learning model overtime in various embodiments.

[0045] The blood detection module 215 applies the machine-learning model to each frame of the pre-activation set of frames, and the machine-learning model identifies blood pixels in a frame. In some embodiments, the blood detection module 215 applies one or more computer vision models to a frame to determine a region of interest (ROI) of the frame and applies the machine learning model only to the region of interest. For example, a computer vision model identifies a tip of the electrosurgical device 105 in a frame and determines a ROI of the frame as pixels of the frame within a threshold distance of the tip of the electrosurgical device 105. In an embodiment, the ROI is determined based on detecting the tip of the electrosurgical device 105 and corresponding ROI in the frame corresponding to the activation time. In other embodiments, a ROI includes the entire frame.

[0046] Based on application of the machine -learning model to the ROI of frame of the pre-activation set of frames, the blood detection module 215 determines one or more pre-activation bleeding metrics. In some embodiments, a pre-activation bleeding metric comprises a mean percentage of blood pixels in the ROIs in the pre-activation set of frames. As another example, a pre-activation bleeding metric comprises a maximum of the respective frame-by-frame percentages of blood pixels in the ROIs of the pre-activation set of frames. In another example, a 13 JNJ-046WO / VRB5194WOPCT1pre-activation bleeding metric comprises a mean increase in number of blood pixels in the ROIs over the pre-activation set of frames. As another example, a pre-activation bleeding metric comprises a maximum increase in blood pixels in the ROIs between successive frames of the pre-activation set of frames. In a further example, a pre-activation bleeding metric comprises a ratio of a mean number of blood pixels in the ROIs of the pre-activation set of frames to a total number of pixels in the ROIs the pre-activation set of frames. In embodiments where the ROI comprises an entire frame, the blood detection module 215 determines one or more pre-activation bleeding metrics based on pixels within full frames, of the pre-activation set of frames.

[0047] Similarly, the blood detection module 215 applies the machine-learning model to each frame of the post-activation set of frames to detect blood pixels. In some implementations, the blood detection module 215 may identify the ROIs in each frame of the post-activation set of frames and detect blood pixels only within the respective ROIs. For example, the blood detection module 215 identifies coordinates in a frame corresponding to the ROI determined from the frame of the pre-activation set of frames and identifies the ROI in fames of the postactivation set based on the coordinates.

[0048] Based on application of the machine-learning model to the ROI of each frame of the postactivation set of frames, the blood detection module 215 determines one or more post-activation bleeding metrics. In some embodiments, a post-activation bleeding metric comprises a mean percentage of blood pixels in the ROIs of the post-activation set of frames. As another example, a post-activation bleeding metric comprises a maximum percentage of the respective frame-by-frame percentages of blood pixels in the ROIs of the post-activation set of frames. In another example, a post-activation bleeding metric comprises a mean increase in blood pixels in the ROIs of the post-activation set of frames. As another example, a post-activation bleeding metric comprises a maximum increase in blood pixels in the ROIs between successive frames of the post-activation set of frames. In a further example, a post-activation bleeding metric comprises a ratio of a mean number of blood pixels in the ROIs of the post-activation set of frames to a total number of pixels in the ROIs of the post-activation set of frames. In embodiments where the blood detection module 215 determines a ROIs of frames based on a location of a tip of the electrosurgical device 105, the blood detection module 215 determines the post-activation bleeding metric based on pixels within the ROIs. As further described above, the ROI comprises a full frame of the post-activation set of frames in various embodiments.

[0049] The procedure evaluation module 220 receives one or more pre-activation bleeding metrics and one or more post-activation bleeding metrics and generates one or more bleed event metrics based on the one or more pre-activation bleeding metrics and the one or more postactivation bleeding metrics. The one or more bleed event metrics characterize a change in 14 JNJ-046WO / VRB5194WOPCT1detected blood caused by activation of the electrosurgical device 105. For example, the bleed event metric may comprise a ratio of a post-activation bleeding metric to a pre -activation bleeding metric, a difference between the post-activation bleeding metric and the pre-activation bleeding metric, or other combinations of the post-activation and pre-activation bleeding metrics. In various embodiments, the one or more bleed event metrics comprises at least one of: (a) a ratio of a mean percentage of blood pixels in ROIs of the post-activation set of frames to a mean percentage of blood pixels in ROIs of the pre-activation set of frames; (b) a ratio of a maximum percentage of blood pixels in ROIs of the post-activation set of frames to a maximum percentage of blood pixels in ROIs of the pre-activation set of frames; (d) a difference between a (i) a ratio of a mean number of blood pixels in ROIs of the post-activation set of frames to a total number of pixels in the ROIs of the post-activation set of frames, and (ii) a ratio of a mean number of blood pixels in ROIs of the pre-activation set of frames to a total number of pixels in the ROIs of the pre-activation set of frames; (e) a ratio of a mean increase in blood pixels in ROIs of the postactivation set of frames to a mean increase in blood pixels in ROIs of the pre-activation set of frames; (f) a ratio of a maximum increase in blood pixels in ROIs of successive frames of the post-activation set of frames to a maximum increase in blood pixels in ROIs of successive frames of the pre-activation set of frames. The one or more bleed event metrics may also include one or more metrics that characterize the post-activation blood without necessarily considering the pre-activation blood. For example, a bleed event metric may be (g) a ratio of a mean number of blood pixels in ROIs of the post-activation set of frames to a total number of blood pixels in ROIs of the post-activation set of frames. However, different or additional bleed event metrics may be determined in various embodiments. The procedure evaluation module 220 determines multiple bleed event metrics in various embodiments.

[0050] The procedure evaluation module 220 outputs one or more bleed event metrics in various embodiments. Further, based on one or more bleed event metrics, the procedure evaluation module 220 determines whether activation of the electrosurgical device 105 at an activation time caused bleeding. For example, the procedure evaluation module 220 may perform a binary classification in which it stores or outputs a classification result comprising binary value indicative of whether or not the activation caused bleeding (e.g., a first specific value to indicate that the event caused bleeding and a second specific value to indicate that the event did not cause bleeding). The classification result may be stored in association with the activation time or the deactivation time to enable the event to be displayed on a timeline associated with the video. Alternatively, the procedure evaluation module 220 may output likelihood values or confidence values indicative of the determination.

[0051] In some embodiments, the procedure evaluation module 220 maintains a set of rules 15 JNJ-046WO / VRB5194WOPCT1corresponding to values of one or more bleed event metrics to determine whether activation of the electrosurgical device 105 caused bleeding. For example, a rule specifies a range of values for one or more bleed event metrics that map to a determination of a bleeding event or a nonbleeding event. In one implementation, a lookup table may map one or more bleeding event metrics to a determination of whether or not a bleeding event occurred.

[0052] Alternatively, the procedure evaluation module 220 maintains or obtains a classification model that receives one or more bleed event metrics as input and generates an output classifying activation of the electrosurgical device 105 at an activation time as causing bleeding or not causing bleeding based on the one or more bleed event metrics. In an embodiment, the classification model receives the set of six metrics (a)-(6) further described above or some subset of these metrics. For example, the classification model comprises a logistic regression model that receives one or more bleed event metrics and outputs a binary classification value indicating whether or not activation of the electrosurgical device 105 caused bleeding. Alternatively, the classification model may output a likelihood value or confidence value associated with the classification. The classification model may be iteratively trained through application to a set of training examples that each comprise training bleed event metrics and a label indicating whether bleeding was caused. A training process for the classification model comprises: applying the classification model to a training example, comparing an output of the classification model to the label associated with the training example, and updating weights comprising with the classification model through a back-propagation process based on the comparison. The weights may be stored on one or more computer-readable media and are used when applying the classification model to new data. Other types of models may be applied to one or more bleed event metrics to determine whether activation of the electrosurgical device 105 caused bleeding or did not cause bleeding in various embodiments.

[0053] FIG. 3 is a flowchart of an example embodiment of a method for determining whether operation of an electrosurgical device 105 during a medical procedure caused bleeding. An online medical system 130 obtains 305 operational data of an electrosurgical device 105 during a medical procedure. In various embodiments, the operational data comprises energy usage data of an electrosurgical device 105 during a medical procedure. The energy usage data identifies amounts of current provided to the electrosurgical device 105 at different times. As further described above in conjunction with FIG. 1, the online medical system 130 obtains 305 the operational data from a current monitor 110 coupled to, or included in, the electrosurgical device 105 in various embodiments. The operational data comprises different data in various embodiments. For example, the operational data includes an activation signal received by the electrosurgical device and a deactivation signal received by the electrosurgical device, and timing 16 JNJ-046WO / VRB5194WOPCT1information associated with the activation signal and with the deactivation signal. Different or additional information may be included in the operational data in various embodiments.

[0054] During the medical procedure where the operational data (e.g., energy usage data) was captured, an image capture device 115 captures video data including a portion of the electrosurgical device 105 and one or more portions of a patient on whom the medical procedure was performed. For example, the video data includes a tip of an active electrode of the electrosurgical device 105. The online medical system 130 obtains 310 the video data from the image capture device 115, or from a device communicatively coupled to the image capture device 115.

[0055] The online medical system 130 temporally synchronizes 315 the operational data (e.g., energy usage data) and the video data. In various embodiments, the online medical system 130 determines an identifier of a medical procedure in the operational data (e.g., energy usage data) and obtains 310 the video data including the identifier of the medical procedure. Similarly, the online medical system 130 may determine an identifier of a medical procedure in the video data and obtains 305 the operational data (e.g., energy usage data) including the identifier of the medical procedure. The online medical system 130 synchronizes 315 the operational data (e.g., energy usage data) and the video data so both have a common starting time and a common ending time. Synchronizing 315 the operational data (e.g., energy usage data) and the video data causes the energy usage data and the video data to have a common starting time and a common ending time, so times in the operational data (e.g., energy usage data) correspond to times in the video data. This allows retrieval of both video data and energy data corresponding to a specific time, so operational data (e.g., energy usage data) corresponding to one or more particular frames of video data may be retrieved by the online medical system 130.

[0056] In embodiments where the operational data comprises energy usage data, the online medical system 130 determines 320 an activation time of the electrosurgical device 105 and determines 325 a deactivation time of the electrosurgical device 105. In some embodiments, the online medical system 130 stores different threshold currents in association with different types of electrosurgical devices 105 to account for variations in operation of different electrosurgical devices 105. For example, the online medical system 130 extracts an identifier of the electrosurgical device 105 from the energy usage data and retrieves a threshold current stored in association with the extracted identifier. The online medical system 130 determines 320 the activation time as a specific time in the energy usage data where a current amount equals or exceeds the threshold current while a current amount at a consecutive time earlier than the specific time is less than the threshold current Similarly, the online medical system 130 determines 325 the deactivation time as a particular time where a current amount is less than the 17 JNJ-046WO / VRB5194WOPCT1threshold current a current amount at a consecutive time earlier than the particular time equals or exceeds the threshold current as the deactivation time of the electrosurgical device 105.Alternatively, the online medical system 130 determines 320 the activation time and determines 325 the deactivation time based on a change in amperage between consecutive times in the energy usage data, as further described above in conjunction with FIG. 2.

[0057] In other embodiments, the online medical system 130 determines 320 the activation time and determines 325 the deactivation time from the operational data. For example, the operational data includes an activation signal and an associated time and a deactivation signal and an associated time. The online medical system 130 determines 320 the activation time as the time associated with the activation signal. Similarly, the online medical system 130 determines 325 the deactivation time as the time associated with the deactivation signal. However, in other embodiments, the online medical system 130 may differently determine 320 the activation time or determine 325 the deactivation time based on the operational data.

[0058] Based on the activation time, the online medical system 130 extracts 330 a pre -activation set of frames of video data. In various embodiments, the online medical system 130 identifies a frame corresponding to the activation time from the video data and extracts 330 frames corresponding to times within a pre -activation time window earlier than the activation time as the pre-activation set of frames. For example, the online medical system 130 extracts 330 frames corresponding to times within a pre-activation time window ranging from a pre-activation amount (e.g., 2 seconds) earlier than the activation time to the activation time as the pre-activation set of frames. Hence, the pre-activation set of frames includes frames earlier than the activation time.

[0059] Similarly, the online medical system 130 extracts 335 a post-activation set of frames of video data based on the deactivation time. In various embodiments, the online medical system 130 identifies a frame corresponding to the deactivation time from the video data and extracts 335 frames corresponding to times within a post-activation window that is later than the deactivation time as the post-activation set of frames. For example, the online medical system 130 extracts 325 frames corresponding to times within a post-activation window of the deactivation time to the deactivation time increased by a post-activation amount (e.g., 4 seconds) as the post-activation set of frames. In various embodiments, the pre-activation window and the post-activation window are equal amounts of time. Alternatively, the pre-activation window and the post-activation window have different durations. For example, the post-activation window is longer than the pre-activation window.

[0060] The online medical system 130 determines 340 one or more pre-activation bleeding metrics based on the pre-activation set of frames. In various embodiments, the online medical 18 JNJ-046WO / VRB5194WOPCT1system 130 applies a trained machine-learned model to each frame of the set of pre -activation frames that characterizes pixels in a frame including blood or that characterizes pixels in a region of interest of a frame including blood. For example, the online medical system 130 applies one or more computer vision models to a frame of the set of pre-activation frames corresponding to the activation time to determine a region of interest based on a position of an operating component of the electrosurgical device 105 (e.g., tip of an active electrode) in the frame corresponding to the activation time. Subsequently, the online medical system 130 applies the machine-learning model to a set of frames in each frame of the pre-activation set corresponding to the region of interest to identify blood pixels in the region of interest determined as depicting blood, as further described above in conjunction with FIG. 2. The online medical system 140 determines 340 one or more pre-activation bleeding metrics based on blood pixels in frames of the pre-activation set of frames. As further described above in conjunction with FIG. 2, example pre-activation bleeding metrics include: a mean number of blood pixels in frames of the pre-activation set (or in the region of interest of frames of the pre-activation set), a maximum number of blood pixels in a frame of the pre-activation set (or in the region of interest of a frame of the pre-activation set), a mean increase in a number of blood pixels between consecutive frames of the pre-activation set (or in the region of interest of consecutive frames of the pre-activation set), a maximum increase in a number of blood pixels between consecutive frames of the pre-activation set (or in the region of interest of consecutive frames of the pre-activation set), or another value based on a number of blood pixels identified as in frames (or in the region of interest of frames) of the pre-activation set of frames.

[0061] Similarly, the online medical system 130 determines 345 one or more post-activation bleeding metrics based on the post-activation set of frames. In various embodiments, the online medical system 130 applies a trained machine-learned model to each frame of the set of postactivation frames that characterizes blood pixels in a frame (determined as depicting blood) or that identifies blood pixels in a region of interest of a frame, as further described above in conjunction with FIG. 2. In embodiments where the online medical system 130 identified a region of interest from the frame corresponding to the activation time, the online medical system 130 applies the machine-learning model to a set of pixels corresponding to the region of interest in each frame of the post-activation set to identify the blood pixels in the region of interest. The online medical system 140 determines 345 the one or more post-activation bleeding metrics based on the blood pixels in frames of the post-activation set of frames. As further described above in conjunction with FIG. 2, example post-activation bleeding metrics include: a mean number of blood pixels in frames of the post-activation set (or in the region of interest of frames of the post-activation set), a maximum number of blood pixels in a frame of the post-activation 19 JNJ-046WO / VRB5194WOPCT1set (or in the region of interest of a frame of the post-activation set), a mean increase in a number of blood pixels between consecutive frames of the post-activation set (or in the region of interest of consecutive frames of the post-activation set), a maximum increase in a number of blood pixels between consecutive frames of the post-activation set (or in the region of interest of consecutive frames of the post-activation set), or another value based on a number of blood pixels identified as in frames (or in the region of interest of frames) of the post-activation set of frames.

[0062] Based on the one or more pre -activation bleeding metrics and the one or more postactivation bleeding metrics, the online medical system 140 determines 350 one or more bleed event metrics for activation of the electrosurgical device 105 corresponding to the activation time and the deactivation time. In various embodiments, a bleed event metric comprises a ratio of a pre-activation bleeding metric to a post-activation bleeding metric. For example, a bleed event metric comprises a ratio of a mean number of blood pixels in frames of the post-activation set (or in the region of interest of frames of the post-activation set) to a mean number of blood pixels in frames of the pre-activation set (or in the region of interest of frames of the pre-activation set). As another example, a bleed event metric comprises a ratio of a maximum number of blood pixels in a frame of the post-activation set (or in the region of interest of a frame of the postactivation set) including blood to a maximum number of blood pixels in a frame of the pre-activation set (or in the region of interest of a frame of the pre-activation set). In another example, a bleed event metric comprises a difference between a ratio of a mean number of blood pixels including blood in the post-activation set of frames to a total number of pixels in frames of the post-activation set of frames and an additional ratio of a ratio of a mean number of blood pixels in the pre-activation set of frames to a total number of pixels in frames of the pre-activation set of frames. Additional examples of bleed event metrics are further described above in conjunction with FIG. 2. In various embodiments, the online medical system 130 determines 350 multiple bleed event metrics.

[0063] In various embodiments, the online medical system 130 determines whether activation of the electrosurgical device 105 at the activation time caused bleeding or did not cause bleeding based on one or more bleed event metrics. In some embodiments, the online medical system 130 maintains a set of rules corresponding to values of one or more bleed event metrics for classifying whether activation of the electrosurgical device 105 at the activation time caused bleeding. Alternatively, the online medical system 130 maintains or obtains a model that receives one or more bleed event metrics as input and generates an output comprising a classification result classifying activation of the electrosurgical device 105 at the activation time as causing bleeding or not causing bleeding based on the one or more bleed event metrics. The 20 JNJ-046WO / VRB5194WOPCT1online medical system 130 stores an indication whether the activation of the electrosurgical device 105 caused bleeding in association with a combination of the activation time and the deactivation time in various embodiments. For example, the indication has a specific value in response to the online medical system 130 determining activation of the electrosurgical device 105 at the activation time caused bleeding and has an alternative value in response to the online medical system 130 determining activation of the electrosurgical device 105 at the activation time did not cause bleeding.

[0064] As further described above in conjunction with FIG. 2, in various embodiments, the online medical system 130 selects training material to recommend to a medical practitioner in response to determining activation of the electrosurgical device 105 caused bleeding. For example, the online medical system 130 applies one or more trained models to the video data or the operational data (e.g., energy usage data) corresponding to times between a starting time of the pre-activation set of frames and an ending time of the post-activation set of frames. The one or more models infer a cause of the bleeding from the video data or from the energy usage data. For example, the inferred causes may comprise mistimed activation of the electrosurgical device 105, misplacement of the surgical tip of the electrosurgical device 105, improper calibration of the electrosurgical device 105, or other causes relating to non-optimal use of the electrosurgical device 105. Based on the inferred cause from the one or more models, the online medical system 130 selects training material corresponding to the cause and transmits information identifying the cause to a client device of the medical practitioner associated with the medical procedure. The training material may include techniques for avoiding or mitigating bleeding events in future surgeries, thereby leading to improved medical outcomes.

[0065] Further, training material may include configuration information for the electrosurgical device 105. For example, training information specifies current amounts for use by the electrosurgical device 105 or settings for configuring other aspects of the electrosurgical device 105. As another example, training information includes calibration instructions for the electrosurgical device 105. A user may transmit or store training material comprising training information to the electrosurgical device 105 to simplify subsequent configuration of the electrosurgical device 105 for performing a subsequent medical procedure. Alternatively, the electrosurgical device 105 may be automatically calibrated in a manner predicted to reduce bleeding events in response to receiving training information, such as calibration instructions, from the online medical system 130.

[0066] In another embodiment, image features in the pre-activation frames together with device data, patient characteristics or other information may be analyzed through a machine learning model to enable prediction of likelihood of bleeding events prior to activation. An alert system 21 JNJ-046WO / VRB5194WOPCT1may then issue an alert prior to activation (or upon activation) to warn a medical practitioner of the likelihood of a bleeding event. For example, bleeding prediction training examples including pre-activation frames labeled with an indication whether bleeding was caused by activation of the electrosurgical device 105 are generated based on the above determination. The online medical system 130 trains a bleeding prediction model through application to the bleeding prediction training examples. The bleeding prediction model receives a set of video frames as input and outputs a probability of activation of the electrosurgical device causing bleeding based on features of the set of video frames. The online medical system 130 trains the blood prediction model through application to bleeding prediction training examples. The online medical system 130 scores the output from the blood prediction model using a loss function generating a score for the output of the blood prediction model so the score is higher when the blood prediction model performs poorly and lower when the blood prediction model performs well. The loss function is based on a difference between a label applied to a blood prediction training example and a probability of bleeding occurring in response to activating the electrosurgical device 105 based on the blood prediction frames. The online medical system 130 updates the set of parameters for the blood prediction model, based on the score generated by the loss function. Subsequently, the online medical system 130 applies the trained blood prediction model to video data captured during a subsequent medical procedure using the electrosurgical device. In response to the blood prediction model generating at least a threshold likelihood of bleeding occurring from activation of the electrosurgical device 105 from frames of the video captured during the subsequent medical procedure, the online medical system 130 transmits a warning to the electrosurgical device 105 or to a client device associated with the subsequent medical procedure for presentation to a medical practitioner to provide a warning of a likely bleeding event if the electrosurgical device is activated.

[0067] In a further example, detected bleeding events for a set of procedures performed using a robot-assisted surgical platform may be analyzed (e.g., using a machine learning model) to predict when a bleeding event is likely to occur (prior to activation of the electrosurgical device 105) and a control signal may be used to the robot-assisted surgical platform to pause the procedure and / or alert an operator. Additionally, characterization of bleeding events together with telemetry data collected from the robot-assisted surgical platform during procedures may be utilized to determine control, configuration, and / or procedure planning settings for the robot-assisted surgical platform most likely to mitigate likelihood of bleeding events.

[0068] FIG. 4 is an example timing diagram of an online medical system 130 extracting a set of pre-activation frames and a set of post-activation frames of video data captured during a medical procedure to evaluate operation of an electrosurgical device 105. The example of FIG. 4 shows 22 JNJ-046WO / VRB5194WOPCT1operational data 400, such as energy usage data, captured during performance of the medical procedure and video data 405 captured during performance of the medical procedure. As further described above in conjunction with FIGS. 2 and 3, the online medical system 140 temporally synchronizes the operational data 400 and the video data 405. Temporally synchronizing the operational data 400 and the video data 405 causes the operational data 400 and the video data 405 to have a common starting time and a common ending time. In various embodiments, the operational data 400 comprises amounts of current corresponding to different times, and the online medical system 130 determines an activation time 410 of the electrosurgical device 105 and a deactivation time 415 of the electrosurgical device 105 based on amounts of currents associated with different times. For example, the online medical system 140 determines the activation time 410 as a time when the amount of current in the operational data 400 increases at least a threshold amount relative to an earlier consecutive time. Similarly, the online medical system 140 determines the deactivation time 415 as a time when the amount of current in the operational data 400 decreases at least a threshold amount relative to an earlier consecutive time. However, in other embodiments, the online medical system 130 uses different or additional criteria for determining the operational data 400 and the deactivation time 415 of the electrosurgical device 105, as further described above in conjunction with FIGS. 2 and 3.

[0069] Based on the activation time 410, the online medical system 130 extracts a pre-activation set of frames 420 of the video data 405. Because the video data 405 and the operational data 400 are temporally synchronized, the online medical system 130 identifies a frame of the video data 405 corresponding to the activation time 410. The online medical system 130 extracts frames of the video data 405 corresponding to times earlier than the activation time 410 (e.g., times within a specific time interval before the activation time 410) as the pre-activation set of frames 420. For example, the online medical system 130 extracts pre-activation frames 420in a pre-activation window of predefined length (e.g., 2 seconds) prior to the activation time 410. Hence the pre-activation set of frames 420 includes frames of the video data 405 corresponding to a time interval earlier than the activation time 410 and ending at the activation time 410.

[0070] Similarly, the online medical system 140 extracts a post-activation set of frames 425 of the video data 405 based on the deactivation time 415. Because the video data 405 and the operational data 400 are temporally synchronized, the online medical system 140 identifies a frame of the video data 405 corresponding to the deactivation time 415. The online medical system 130 extracts frames of the video data 405 corresponding to times after the deactivation time 415 (e.g., times within a specific time interval after the deactivation time 415) as the postactivation set of frames 425. For example, the online medical system 130 extracts in a postactivation window of predefined length (e.g., 4 seconds) after the deactivation time 415 as the 23 JNJ-046WO / VRB5194WOPCT1post-activation set of frames 425. Hence the post-activation set of frames 425 includes frames of the video data 405 corresponding to a time interval beginning at the deactivation time 415 and ending at a time later than the deactivation time 415.

[0071] As further described above in conjunction with FIG. 3, the online medical system 130 determines one or more pre -activation bleeding metrics from the pre -activation set of frames 420 based on detection of blood in pixels of frames of the pre-activation set of frames 420. Similarly, the online medical system 130 determines one or more post-activation bleeding metrics from the post-activation set of frames 425 based on detection of blood in pixels of frames of the postactivation set of frames 425. The online medical system 130 determines whether activation of the electrosurgical device 105 at the activation time 410 caused bleeding based on the one or more pre-activation bleeding metrics and the one or more post-activation bleeding metrics, as further described above in conjunction with FIG. 3.

[0072] The described embodiments incorporate multiple technical improvements in fields such as computer vision systems, electronic medical instrumentation, and robotic-assisted surgery by enabling more accurate and efficient evaluation of electrosurgical device use through captured video data of a medical procedure involving the electrosurgical device. For example, synchronizing operational data of an electrosurgical device during a medical procedure and video data during the medical procedure allows identification of specific sets of frames of video data corresponding to activation and deactivation of the electrosurgical device. Subsequently, a machine learning model may efficiently operate on a limited sets of frames of the video data, rather than the complete video data of the medical procedure, to evaluate operation of the electrosurgical device. This technique decreases computational resources and storage resources for evaluating video data to characterize use of the electrosurgical device. Additionally, analyzing specific sets of frames based on activation and deactivation of the electrosurgical device, rather than complete video data of a medical procedure, reduces an amount of time for evaluating operation of the electrosurgical device. Further, applying a trained machine-learning model to detect blood in frames of video data prior to activation of the electrosurgical device and in frames after deactivation of the electrosurgical device decreases an amount of time to identify the presence of blood in various frames relative to applying a blood detection algorithm across all frames, which reduces an amount of time to evaluate whether activation of the electrosurgical device caused bleeding.

[0073] The foregoing description of the embodiments has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the embodiments to the precise forms disclosed. Persons skilled in the relevant art can appreciate that many modifications and variations are possible in light of the above disclosure.24 JNJ-046WO / VRB5194WOPCT1

[0074] Some portions of this description describe the embodiments in terms of algorithms and symbolic representations of operations on information. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Furthermore, it has also proven convenient at times, to refer to these arrangements of operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combinations thereof.

[0075] Any of the steps, operations, or processes described herein may be performed or implemented with one or more hardware or software modules, alone or in combination with other devices. Embodiments may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, and / or it may include a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a tangible non-transitory computer readable storage medium or any type of media suitable for storing electronic instructions and coupled to a computer system bus. Furthermore, any computing systems referred to in the specification may include a single processor or may include architectures employing multiple processor designs for increased computing capability.

[0076] As used herein, unless expressly stated to the contrary, “or” refers to an inclusive “or” and not to an exclusive “or.” For example, a condition “A or B” is satisfied by any one of the following: A is true (or present) and B is false (or not present); A is false (or not present) and B is true (or present); and both A and B are true (or present). Similarly, a condition “A, B, or C” is satisfied by any combination of A, B, and C being true (or present). As a non-limiting example, the condition “A, B, or C” is satisfied when A and B are true (or present) and C is false (or not present). Similarly, as another non-limiting example, the condition “A, B, or C” is satisfied when A is true (or present) and B and C are false (or not present).

[0077] Finally, the language used in the specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the inventive subject matter. It is therefore intended that the scope is not limited by this detailed description, but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of the embodiments is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.25 JNJ-046WO / VRB5194WOPCT1

Claims

WHAT IS CLAIMED IS:

1. A method for characterizing bleeding events in video of a medical procedure that uses an electrosurgical device, the method comprising:obtaining, at an online medical system, operational data indicating an activation time and a deactivation time of the electrosurgical device;obtaining video data captured during the medical procedure;extracting a pre -activation set of frames from the video data based on the activation time, the pre-activation set of frames including frames earlier than the activation time; extracting a post-activation set of frames from the video data based on the deactivation time, the post-activation set of frames including frames later than the deactivation time;determining one or more pre-activation bleeding metrics based on application of a machine-learning model characterizing blood in frames of the pre-activation set of frames;determining one or more post-activation bleeding metrics based on application of the machine-learning model characterizing blood in frames of the post-activation set of frames;determining one or more bleed event metrics associated with operation of the electrosurgical device based on the one or more pre-activation bleeding metrics and the one or more post-activation bleeding metrics; andoutputting the one or more bleed event metrics.

2. The method of claim 1, further comprising:determining whether activation of the electrosurgical device at the activation time caused bleeding based on the one or more bleed event metrics.

3. The method of claim 2, wherein determining whether activation of the electrosurgical device at the activation time caused bleeding based on the one or more bleed event metrics comprises:applying a classification model to the one or more bleed event metrics, the classification model outputting a classification result characterizing likelihood of activation of the electrosurgical device at the activation time causing bleeding.

4. The method of any preceding claim, wherein extracting the pre-activation set of frames from the video data based on the activation time comprises:26 JNJ-046WO / VRB5194WOPCT1extracting frames from the video data corresponding to a predefined pre-activation time window before the activation time.

5. The method of claim 4, wherein extracting the post-activation set of frames from the video data based on the deactivation time comprises:extracting frames from the video data corresponding to a predefined post-activation time window after the deactivation time.

6. The method of claim 5, wherein a length of the predefined pre-activation time window is different from a length of the predefined post-activation time window.

7. The method of claim 6, wherein the length of the predefined post-activation time window is greater than the length of the predefined pre-activation time window.

8. The method of any preceding claim, wherein determining one or more pre-activation bleeding metrics based on application of the machine-learning model identifying blood in frames of the pre-activation set of frames comprises:determining a region of interest in each frame of the pre-activation set of frames based on a location of a cutting element of the electrosurgical device in a frame corresponding to the activation time; anddetermining one or more pre-activation bleeding metrics based on application of the machine-learning model identifying blood in the region of interest in frames of the pre-activation set of frames.

9. The method of claim 8, wherein determining one or more post-activation bleeding metrics based on application of the machine-learning model identifying blood in frames of the post-activation set of frames comprises:determining one or more post-activation bleeding metrics based on application of the machine-learning model identifying blood in the region of interest in frames of the post-activation set of frames.

10. The method of any preceding claim, wherein obtaining, at the online medical system, operational data indicating the activation time and the deactivation time of the electrosurgical device comprises:obtaining, at the online medical system, energy usage data for the electrosurgical device, the energy usage data comprising current amounts at different times during the medical procedure;determining an activation time of the electrosurgical device based on the energy usage data; and27 JNJ-046WO / VRB5194WOPCT1determining a deactivation time of the electrosurgical device based on the energy usage data.

11. The method of any preceding claim, further comprising:applying a control model to the one or more bleed event metrics to generate configuration information or modifications to a control process for a surgical robot using the electrosurgical device to reduce a likelihood of bleeding occurring from operation of the electrosurgical device by the surgical robot; andtransmitting the configuration information or the modifications to the control process to the surgical robot for execution to reduce the likelihood of bleeding occurring from operation of the electrosurgical device.

12. The method of any preceding claim, further comprising:obtaining video data captured during performance of a subsequent medical procedure using the electrosurgical device;applying a blood prediction model to frames of the video captured during the subsequent medical procedure, the blood prediction model trained based at least in part on the one or more bleed event metrics to determine a likelihood of bleeding occurring from activation of the electrosurgical device based on the frames of the video captured during the subsequent medical procedure; andtransmitting a warning to a client device of a medical practitioner associated with the subsequent medical procedure in response to determining the likelihood of bleeding occurring from activation of the electrosurgical device equaling or exceeding a threshold probability.

13. A non-transitory computer readable storage medium having instructions encoded thereon for detecting bleeding events in video of a medical procedure using an electrosurgical device, the instructions, when executed by one or more processors, cause the one or more processors to perform steps comprising:obtaining, at an online medical system, operational data indicating an activation time and a deactivation time of the electrosurgical device;obtaining video data captured during the medical procedure;extracting a pre-activation set of frames from the video data based on the activation time, the pre-activation set of frames including frames earlier than the activation time; extracting a post-activation set of frames from the video data based on the deactivation time, the post-activation set of frames including frames later than the deactivation time;28 JNJ-046WO / VRB5194WOPCT1determining one or more pre-activation bleeding metrics based on application of a machine-learning model characterizing blood in frames of the pre-activation set of frames;determining one or more post-activation bleeding metrics based on application of the machine-learning model characterizing blood in frames of the post-activation set of frames;determining one or more bleed event metrics associated with operation of the electrosurgical device based on the one or more pre-activation bleeding metrics and the one or more post-activation bleeding metrics; andoutputting the one or more bleed event metrics.

14. The non-transitory computer readable storage medium of claim 13 further having instructions encoded thereon that, when executed by the one or more processors, cause the one or more processors to perform steps comprising:determining whether activation of the electrosurgical device at the activation time caused bleeding based on the one or more bleed event metrics.

15. The non-transitory computer readable storage medium of claim 14, wherein determining whether activation of the electrosurgical device at the activation time caused bleeding based on the one or more bleed event metrics comprises:applying a classification model to the one or more bleed event metrics, the classification model outputting a classification result characterizing likelihood of activation of the electrosurgical device at the activation time causing bleeding.

16. The non-transitory computer readable storage medium of any one of claims 13 to 15, wherein extracting the pre-activation set of frames from the video data based on the activation time comprises:extracting frames from the video data corresponding to a predefined pre-activation time window before the activation time.

17. The non-transitory computer readable storage medium of claim 16, wherein extracting the post-activation set of frames from the video data based on the deactivation time comprises:extracting frames from the video data corresponding to a predefined post-activation time window after the deactivation time.

18. The non-transitory computer readable storage medium of claim 17, wherein a length of the predefined pre-activation time window is different from a length of the predefined post-activation time window.29 JNJ-046WO / VRB5194WOPCT119. The non-transitory computer readable storage medium of claim 18, wherein the length of the predefined post-activation time window is greater than the length of the predefined pre-activation time window.

20. The non-transitory computer readable storage medium of any one of claims 13 to 19, wherein determining one or more pre-activation bleeding metrics based on application of the machine-learning model identifying blood in frames of the pre-activation set of frames comprises:determining a region of interest in each frame of the pre-activation set of frames based on a location of a cutting element of the electrosurgical device in a frame corresponding to the activation time; anddetermining one or more pre-activation bleeding metrics based on application of the machine-learning model identifying blood in the region of interest in frames of the pre-activation set of frames.

21. The non-transitory computer readable storage medium of claim 20, wherein determining one or more post-activation bleeding metrics based on application of the machinelearning model identifying blood in frames of the post-activation set of frames comprises: determining one or more post-activation bleeding metrics based on application of the machine-learning model identifying blood in the region of interest in frames of the post-activation set of frames.

22. The non-transitory computer readable storage medium of any one of claims 13 to 21, wherein obtaining, at the online medical system, operational data indicating the activation time and the deactivation time of the electrosurgical device comprises: obtaining, at the online medical system, energy usage data for the electrosurgical device, the energy usage data comprising current amounts at different times during the medical procedure;determining an activation time of the electrosurgical device based on the energy usage data; anddetermining a deactivation time of the electrosurgical device based on the energy usage data.

23. The non-transitory computer readable storage medium of any one of claims 13 to 22 further having instructions encoded thereon that, when executed by the one or more processors, cause the one or more processors to perform steps comprising:applying a control model to the one or more bleed event metrics to generate configuration information or modifications to a control process for a surgical robot using the 30 JNJ-046WO / VRB5194WOPCT1electrosurgical device to reduce a likelihood of bleeding occurring from operation of the electrosurgical device by the surgical robot; andtransmitting the configuration information or the modifications to the control process to the surgical robot for execution to reduce the likelihood of bleeding occurring from operation of the electrosurgical device.

24. The non-transitory computer readable storage medium of any one of claims 13 to 23 further having instructions encoded thereon that, when executed by the one or more processors, cause the one or more processors to perform steps comprising: obtaining video data captured during performance of a subsequent medical procedure using the electrosurgical device;applying a blood prediction model to frames of the video captured during the subsequent medical procedure, the blood prediction model trained based at least in part on the one or more bleed event metrics to determine a likelihood of bleeding occurring from activation of the electrosurgical device based on the frames of the video captured during the subsequent medical procedure; andtransmitting a warning to a client device of a medical practitioner associated with the subsequent medical procedure in response to determining the likelihood of bleeding occurring from activation of the electrosurgical device equaling or exceeding a threshold probability.31 JNJ-046WO / VRB5194WOPCT1