Computing device and method for generating training data

Abstract

The invention relates to a method for generating training data, to a corresponding computing device and to a corresponding system. The method comprises at least the following steps: receiving (S100) sensor data (1) from at least one sensor (10; 20; 30) present in a clinical situation; feeding (S200) the sensor data into a first machine learning model (110); processing (S300) the first input data (11) in order to generate first output data (19) which represent an action recommendation, a control signal or information to a user; feeding (S400) the first output data (19) as second input data (21) into a second machine learning model (120); processing (S500) the second input data (21) by means of the second machine learning model (120) in order to generate second output data (29) which indicate whether the received sensor data (1) are to be used to improve the first machine learning model (110), and if this is the case: storing at least some of the received sensor data (1) as the basis for training data for the first machine learning model (110).

Description

[0001] Description

[0002] title

[0003] Computing setup and method for generating training data

[0004] Technical field of the invention

[0005] The invention relates to a computer-implemented method for generating training data for a machine learning model, and to a computing device which is configured to carry out such a method.

[0006] Another aspect relates to a method and a device for generating training data for a machine learning model in a clinical setting. Furthermore, the present invention relates to a method and a device for using a machine learning model in a clinical setting. Finally, the present invention relates to a system comprising a device for generating training data for a machine learning model and a device for using the machine learning model in a clinical setting.

[0007] Background of the invention

[0008] Machine learning models, such as artificial neural networks, k-means algorithms, or support vector machines, are typically trained using training data (training phase) and then deployed at their intended location (deployment phase). An update is generated periodically, for example, by adding newly collected training data to the existing training data, and the machine learning model is retrained or further trained with this supplemented data. In other variations, continuous retraining takes place using all data accumulated in the interim.

[0009] In both cases, these are relatively rigid processes whose timing depends neither on the quality of the previous training nor on the relevance of the data collected in the meantime.

[0010] In the medical field, systems are known that can provide support in clinical situations, such as during certain procedures. For example, a system-assisted system for minimally invasive surgery is known from WO 2024 / 008854 A1. The application of machine learning or artificial intelligence (AI) methods in clinical situations has the potential to improve support during procedures, diagnostic accuracy, efficiency, and patient care.

[0011] In endoscopies, for example, AI algorithms can be used to analyze endoscopic images in real time and detect anomalies. AI systems can also support the user in making a diagnosis, as well as provide real-time feedback and recommendations to highlight or confirm potential pathological findings.

[0012] Machine learning models are trained on large amounts of data to recognize patterns and features that are characteristic of certain disease states.

[0013] In the current stage of machine learning, generating training data requires considerable effort. Data must first be carefully sorted, evaluated, and weighted manually before it can be integrated into the training process. These manual steps are not only time-consuming but also carry the risk of errors and inconsistencies.

[0014] Summary of the invention

[0015] The present invention aims to provide an improved method for generating training data for a machine learning model, which in particular overcomes the aforementioned disadvantages of the prior art. Specifically, one object of the present invention is to provide devices and methods for generating training data for a machine learning model in a clinical setting, thereby simplifying the generation of training data. Furthermore, methods and devices for using a machine learning model in a clinical setting are to be provided. Finally, a system combining the two aforementioned devices is to be provided.

[0016] This problem is solved by the subject matter of the independent claims of the present invention. Advantageous embodiments are the subject matter of the dependent claims.

[0017] According to a first aspect, a computer-implemented procedure for generating training data for a machine learning model is provided, with the following steps:

[0018] Receiving sensor data from at least one sensor present in a clinical situation;

[0019] Feeding at least parts of the received sensor data as at least part of initial input data into an initial machine learning model; processing the initial input data by the initial machine learning model to generate initial output data, which represents a recommendation for action, a control signal for a medical device, or information for a user;

[0020] Feeding at least parts of the first output data and / or intermediate values ​​of the first machine learning model as at least part of the second input data into a second machine learning model;

[0021] Processing the second input data by the second machine learning model to generate second output data, which indicates whether the received sensor data should be used to improve the first machine learning model, and if so:

[0022] Storing and / or transmitting at least some of the received sensor data (especially that which was part of the initial input data) to a data storage device as a basis for (or as) training data for the first machine learning model.

[0023] The invention thus provides a method for checking whether the currently used (i.e., in the deployment phase) first machine learning model still delivers results of sufficient quality, or whether its improvement is already necessary and / or possible.

[0024] This not only results in a significant reduction in the required storage capacity and / or transmission bandwidth, but also in a significant improvement of the training data set or a reduction of additional training data to data that actually offers potential for improvement to the first machine learning model.

[0025] For example, the second machine learning model can be used to advantageously check whether information provided to the user with the initial output data (e.g., "Event X is imminent" or "Smoke extraction required") has subsequently proven to be correct. If not, there might be room for improvement at this point—that is, with regard to the sensor data that the first machine learning model processed to generate the incorrect initial output data. The corresponding sensor data could then be stored by the second machine learning model as future training data or transmitted to the data storage.

[0026] Within the scope of this invention, a clinical situation can be understood to mean an operation, a surgical procedure, the treatment of a patient, an external examination of a patient, or the like. In particular, medical or surgical instruments may be used.

[0027] The processing of the second input data by the second machine learning model preferably takes place immediately following, or partially overlapping, with the processing of the first input data by the first machine learning model.

[0028] An intermediate value of the first machine learning model is a value generated by the first machine learning model but not part of its output data. Such intermediate values ​​are also referred to as "hidden features." In an artificial neural network, for example, such intermediate values ​​are generated by intermediate layers, i.e., layers located between the input layer and the output layer. Preferably, the intermediate values ​​used as part of the second input data are taken from the penultimate or last layer before the output layer.

[0029] The stored and / or transmitted sensor data can themselves become part of the new (additional, future) training data. The second machine learning model can thus act as a kind of pre-filter for important, helpful, etc., training data for the existing, previously trained, first machine learning model. This allows the model to react to changing application situations after its market launch, for example, when a new clinical (especially medical or surgical) procedure is introduced.

[0030] According to some preferred embodiments, variants, or refinements of embodiments, the first and / or the second machine learning model comprises a support vector machine, an artificial neural network, and / or an extreme learning machine. These implementations have proven to be particularly advantageous for the functionality of the respective machine learning model. However, each of the machine learning models can also be implemented as a different type of artificial intelligence entity.

[0031] According to some preferred embodiments, variants or refinements of embodiments, the second machine learning model is configured such that the second output data includes (or is) a numerical value which is compared to a threshold, and a result of the comparison indicates whether the received sensor data (specifically the sensor data used in the first input data) should be used to improve the first machine learning model.

[0032] The numerical value could, for example, be a probability value representing how well the first machine learning model made a prediction. If this value falls below a predetermined threshold (e.g., 80%), this variant stipulates that the sensor data on which the prediction by the first machine learning model was based should be used to improve the first machine learning model.

[0033] In other words, this method automatically and continuously or regularly identifies, during the deployment phase, which sensor data (as initial input data) the first machine learning model struggles with, for example, being less accurate, less precise, or less reliable. Including precisely such sensor data in the generated training data thus enables particularly efficient and effective future (re-)training of the first machine learning model with the generated training data. According to some preferred embodiments, variants, or refinements of embodiments, the method also includes the dynamic selection of an intermediate or final layer of the first machine learning model, particularly if the first machine learning model is designed as an artificial neural network, such as a convolutional neural network (CNN) and / or a recurrent neural network (RNN).Advantageously, at least one intermediate value of the first machine learning model, which is used as part of the second input data, originates from the dynamically selected intermediate or final layer. In this way, the test of whether an improvement of the first machine learning model is possible is advantageously closely linked to the internal processes of the first machine learning model, in which an internal state of the first machine learning model, typically not intellectually accessible to the user ("hidden features"), is used by a second machine learning model that is different from the first.

[0034] According to some preferred embodiments, variants or refinements of embodiments, the sensor data of the at least one sensor comprise or are at least one of the following:

[0035] Image and / or video data from an image sensor;

[0036] - Temperature data from a temperature sensor; position data from a position sensor;

[0037] - Alignment data from an alignment sensor;

[0038] - Vital data of a patient (for example, pulse, blood pressure, oxygen saturation, ECG or EEG value, etc.);

[0039] - Current data from a current sensor;

[0040] - Voltage data from a voltage sensor;

[0041] Pressure data from a pressure sensor (e.g., an insufflator); flow data from a flow sensor;

[0042] - Engine data of a motor, such as speed and / or acceleration and / or torque data (where the motor may be, for example, a motor of a surgical robot or a robotic surgery assistance system);

[0043] - Device settings of a medical device.

[0044] According to some preferred embodiments, variants or refinements of embodiments, the method also includes:

[0045] Supplementing the transmitted portion of the sensor data with at least one of the following: information about a surgical procedure during which the transmitted portion of the sensor data was generated; a medical specialty to which the surgical procedure belongs; at least a portion of the output data of the second machine learning model; a location of the surgical procedure; a timestamp related to the surgical procedure; time information regarding a last preceding transmission of sensor data; a position of an input unit on a master, for example, an input unit that controls a master for the movement of one or more robotic arms with instruments; information from a patient record of a patient undergoing the surgical procedure, in particular as a label for training data for supervised learning of the first machine learning model.

[0046] The added data can, for example, be used automatically as a label or "ground truth", or form a basis for the automatic creation of labels or "ground truths".

[0047] According to some preferred embodiments, variants or refinements of embodiments, the method also includes capturing user input; and feeding the captured user input as a further part of the second input data into the second machine learning model.

[0048] In this way, it is possible, for example, to determine the extent to which a control signal from the initial output data, which is confirmed or countered by user input, has produced a desired result. If, for instance, the control signal from the initial output data automatically activates a function (e.g., smoke extraction), and this function is subsequently deactivated by user input (especially within a predefined period afterward), this indicates an erroneous prediction by the first machine learning model, which can then be detected by the second machine learning model.

[0049] According to some preferred embodiments, variants, or refinements of embodiments, the first machine learning model is designed to detect and / or classify smoke development during a surgical procedure. A control signal output by the first machine learning model as part of the initial output data can, for example, be a control signal for adjusting (on / off, or setting a specific level) a smoke extraction system.

[0050] According to a second aspect, the invention also provides a computing device which is configured to carry out the method according to an embodiment of the first aspect of the present invention.

[0051] Such a computing device can be implemented as any device capable of performing calculations, and in particular, of executing software, an application, or an algorithm. The computing device can, for example, include at least one processing unit, such as a central processing unit (CPU) and / or a graphics processing unit (GPU) and / or a field-programmable logic gate (FPGA) and / or an application-specific integrated circuit (ASIC) and / or a combination thereof. The computing device can also include or consist of a processor specifically designed for artificial intelligence (AI) applications, for example, for an optimized dot product operation on two vectors.

[0052] The computing device may also include main memory, which is operationally coupled to the at least one processor unit, and non-volatile memory, which is operationally coupled to the at least one processor unit and the main memory. The computing device may be implemented wholly or entirely in a local device and / or wholly or entirely in a remote system, such as a remotely located server and / or a cloud computing platform.

[0053] According to some preferred embodiments, variants, or refinements of embodiments, the computing unit is integrated into a camera controller. This offers synergistic advantages, since the camera controller, for example in an operating room, typically already performs tasks such as generating control signals to a camera (e.g., on an endoscope), evaluating camera images, and the like.

[0054] The computing device may have an input interface which is configured to receive sensor data from at least one sensor present in a clinical situation (i.e., in particular, sensor data of the clinical situation).

[0055] The computing device may also include a control module (or "controller") which is set up to implement a first and a second machine learning model.

[0056] The first machine learning model is configured to process initial input data to generate initial output data, which may represent a recommendation for action, a control signal for a medical device, or information for a user. The second machine learning model is configured to process second input data to generate second output data, which indicates whether the received sensor data should be used to improve the first machine learning model.

[0057] The control module is further configured to feed at least parts of the received sensor data as at least part of the first input data into the first machine learning model, and to feed at least parts of the first output data and / or intermediate values ​​of the first machine learning model as at least part of the second input data into the second machine learning model.

[0058] The control module is also configured to store at least some of the received sensor data as a basis for (additional future) training data for the first machine learning model and / or to transmit it to a data storage device if the second output data indicates that the received sensor data should be used to improve the first machine learning model. When "modules" or "interfaces" are mentioned herein, it is understood that this does not necessarily mean that such modules or interfaces are implemented as separate units isolated from one another.

[0059] In cases where modules or interfaces are implemented as software, they can be realized as sections or components of program code, which may be distinguishable from one another or intertwined. Similarly, in cases where one or more modules or interfaces are implemented as hardware, the functions of one or more modules or interfaces can be implemented by one and the same hardware component.

[0060] Alternatively or additionally, different functions of a single module or interface, or different functions of different modules or interfaces, can be implemented on one or more separate hardware components, which therefore do not necessarily have to be in a one-to-one relationship with the modules or interfaces.

[0061] In this sense, any device, system, process, etc., which possesses all the properties and functions attributed to a specific module or interface, can be understood as having, representing, or implementing such a module or interface. In particular, it is possible that all modules and / or interfaces are implemented as program code executed by a computing device, such as a server or a cloud computing platform.

[0062] According to a third aspect, the invention provides a system comprising a computing device according to an embodiment of the second aspect of the present invention. The system also includes at least one sensor, which is configured to acquire sensor data in the clinical situation and is coupled to the input interface of the computing device.

[0063] Depending on the wide variety of possible sensor data, the sensor, or sensors, can also be designed in a variety of ways, for example as:

[0064] Image sensor (single image sensor or video camera);

[0065] - Temperature sensor;

[0066] Position sensor;

[0067] - Alignment sensor;

[0068] - Vital data sensor (e.g. pulse monitor, blood pressure monitor, oxygen saturation monitor, ECG device or EEG device, etc.);

[0069] - Current sensor;

[0070] - Voltage sensor;

[0071] Pressure sensor (e.g., of an insufflator); flow sensor;

[0072] - Sensor of a motor, such as a speed and / or acceleration and / or torque sensor (where the motor may be, for example, a motor of a surgical robot or a robotic surgery assistance system).

[0073] According to a fourth aspect, the invention provides a computer program product comprising executable program code which, when executed by a computing unit, is configured to perform the method according to an embodiment of the first aspect of the present invention.

[0074] According to a fifth aspect, the invention provides a non-volatile, computer-readable data storage medium comprising executable program code which, when executed by a computing unit, is configured to perform or control the method according to an embodiment of the first aspect of the present invention.

[0075] The non-volatile, computer-readable data storage medium can include or consist of any type of computer memory, in particular semiconductor memory, such as solid-state memory. The data carrier can also include or consist of a CD, DVD, Blu-ray disc, USB flash drive, or the like.

[0076] According to a sixth aspect, the invention provides a data stream comprising executable program code or configured to generate executable program code which, when executed by a computing unit, is designed to perform or control the method according to an embodiment of the first aspect of the present invention.

[0077] Further advantageous variants, options, embodiments, and modifications will become apparent from the following figures, the detailed description, and the claims. It is understood, however, that while the detailed description and specific examples represent preferred embodiments of the invention, they are provided for illustrative purposes only, as various changes and modifications within the scope of the invention are obvious to the person skilled in the art.

[0078] According to a seventh aspect, a computer-implemented method for generating training data for a machine learning model for use in a clinical setting is provided. Sensor data is received from at least one sensor present in the clinical setting. Situational information is received, which depends on a change in the state of a device in the clinical setting and / or an action by a person in the clinical setting. At least some of the sensor data is automatically annotated using the situational information to generate the training data. A fundamental idea of ​​the present invention is that the sensor data can be at least partially annotated (i.e., labeled) automatically. This eliminates the need for time-consuming manual annotation.

[0079] Situational information is taken into account for this purpose. Within the scope of this invention, situational information can be understood as information that depends on or describes states, changes in state, actions, events, or the like in the clinical situation.

[0080] For example, a specific action by a person (such as a treating physician or assistant) can indicate the circumstances or events present in the clinical situation, and the sensor data can be labeled accordingly. Similarly, a change in the state of a device used in the clinical setting can indicate the occurrence of an event or the presence of a specific situation, allowing the sensor data to be labeled accordingly. This enables more efficient generation of training data for the machine learning model.

[0081] Within the scope of this invention, a clinical situation can be understood to mean an operation, a surgical procedure, the treatment of a patient, an external examination of a patient, or the like. In particular, medical or surgical instruments may be used.

[0082] A change in the state of a device can refer, in particular, to switching it on, off, or changing its operating parameters. The device could be, for example, a medical instrument or system. In an endoscopy, for instance, a smoke extraction system could be activated or deactivated.

[0083] According to one embodiment of the method for generating training data for the machine learning model, the training data can be generated autonomously, locally and anonymously, which is particularly advantageous with regard to data protection considerations.

[0084] According to one embodiment, the method is further provided for generating the machine learning model. For this purpose, the machine learning model is trained using the annotated sensor data. The trained machine learning model can then be output and used in the clinical setting.

[0085] According to one embodiment of the method for generating training data for the machine learning model, the system determines, based on situational information, whether sensor data should be automatically annotated or discarded. This allows for the automatic detection of whether sensor data is relevant to the training process. If so, it is automatically annotated; otherwise, it is discarded. For example, if a doctor generates and saves a still image during an examination or operation, it can be inferred that the still image is important or contains helpful information. The still image can then be used for training and is automatically annotated.

[0086] According to one embodiment of the method for generating training data for the machine learning model, situational information is determined based on sensor data and / or other sensor data. For example, gestures or actions of people in a clinical situation can be recognized based on sensor data, from which inferences about the situation can be made.

[0087] According to one embodiment of the method for generating training data for the machine learning model, the sensor data includes image data from at least one image sensor, audio data from at least one audio sensor, and / or video data from at least one video sensor. The sensor can, in particular, detect a person undergoing treatment.

[0088] According to one embodiment of the method for generating training data for the machine learning model, a person's action in the clinical situation includes activating or deactivating a device used in the clinical situation. For example, the person might activate or deactivate a medical or surgical device or an auxiliary device. The auxiliary device could be, for instance, a smoke extraction system during an endoscopy.

[0089] The action of the person can also be a specific movement of a piece of equipment used in the clinical situation. The progress of an operation or procedure can be inferred from this movement. The action can also be the person uttering a specific word or sentence, such as an instruction, from which the situation can be determined. The person in this context can be, in particular, a treating physician, assistant, or similar professional.

[0090] According to one embodiment of the method for generating training data for the machine learning model, the clinical situation includes video endoscopy. The sensor data can be image data and / or video data. The machine learning model can be trained for use in smoke detection, progress monitoring of the video endoscopy, and / or energy monitoring of a high-frequency device used in the video endoscopy.

[0091] According to an eighth aspect, the present invention provides a device for generating training data for a machine learning model for use in a clinical setting. The device comprises an interface that receives sensor data from at least one sensor present in the clinical setting. The interface receives situational information that depends on a change in the state of a device in the clinical setting and / or an action by a person in the clinical setting. A computing unit generates the training data by automatically annotating at least a portion of the sensor data using the situational information.

[0092] According to a ninth aspect, the invention provides a method for using a machine learning model in a clinical setting. Sensor data is received from at least one sensor present in the clinical setting. A machine learning model is used to generate a recommendation for action and / or to automatically perform an action, with the sensor data serving as input data for the machine learning model. Reaction information is determined based on a person's response to the recommendation for action and / or to the action performed.

[0093] Within the scope of the invention, a recommendation for action can be understood as the suggestion of one of several possibilities, or the suggestion to perform or not perform a specific action. For example, the user can be advised to activate a flue gas extraction system.

[0094] The automatic execution of an action could be, for example, the automatic activation of a device in a clinical situation, such as activating a smoke extraction system.

[0095] Reaction information can be understood as information that depends on how a person acts after receiving the action recommendation or after the action has been carried out automatically.

[0096] According to one embodiment of the method for using the machine learning model in a clinical setting, the machine learning model is adapted based on the response information. For example, if the person accepts the recommended course of action or does not interrupt the automatically performed action, it can be recognized that the machine learning model correctly predicted the action. The machine learning model can be retrained, for example, by generating new training data based on the sensor data and the results of the machine learning model.

[0097] According to one embodiment of the method for using the machine learning model in a clinical situation, the machine learning model is first trained using training data generated according to a method described in the seventh aspect. According to one embodiment of the method for using the machine learning model in a clinical situation, the response information includes information on whether the person acted in accordance with the recommended course of action. This allows actions by experts, such as a treating physician, to be taken into account to improve the machine learning model.

[0098] According to one embodiment of the method for using the machine learning model in a clinical situation, the response information includes whether the person canceled or reversed the automatic execution of the action. If so, the prediction was likely incorrect; otherwise, it was correct.

[0099] According to one embodiment of the method for using the machine learning model in a clinical setting, an algorithm is used to calculate a reliability metric that indicates how reliable the recommended action and / or the automatic execution of the action is. Thus, a metric or hit rate can be specified in advance, quantifying the expected reliability. The algorithm can be adapted depending on the response information. For example, if the user does not perform the recommended action or interrupts the automatic execution of the action, the reliability metric can be reduced by adjusting the algorithm. Conversely, if the user performs the recommended action or does not interrupt the automatic execution of the action, the reliability metric can be increased by adjusting the algorithm.

[0100] According to one embodiment of the method for using the machine learning model in a clinical situation, the machine learning model is adapted depending on a user profile of the responding person. For example, the initially pre-trained model can be specifically adapted for each user and thus better support the user taking into account their typical behavior.

[0101] According to one embodiment of the method for using the machine learning model in a clinical situation, the machine learning model is retrained during operation. In particular, new training data can be generated.

[0102] According to a tenth aspect, the invention provides a device for using a machine learning model in a clinical setting. An interface receives sensor data from at least one sensor present in the clinical setting. A computing unit uses a machine learning model to generate a recommendation for action and / or to automatically execute an action. The sensor data are used as input data for the machine learning model. A response information acquisition unit determines response information based on a person's reaction to the recommendation for action and / or to the action performed. According to an eleventh aspect, the invention provides a system for use in a clinical setting.The system includes a device for generating training data for a machine learning model according to the eighth aspect and a device for using the machine learning model according to the tenth aspect.

[0103] Although some functions are described here, in the foregoing and below, as being performed by "devices," "interfaces," or "modules," it should be understood that this does not necessarily mean that such devices, interfaces, or modules are provided as separate units. In cases where one or more devices, interfaces, or modules are provided wholly or partially as software, the devices, interfaces, or modules may be implemented by sections or snippets of program code that are distinct from one another but may also be intertwined.

[0104] Similarly, where one or more devices, interfaces, or modules are provided as hardware, the functions of one or more devices, interfaces, or modules may be provided by one and the same hardware component, or the functions of one device, interface, or module, or the functions of several devices, interfaces, or modules, may be distributed across several hardware components, which need not necessarily correspond one-to-one with the devices, interfaces, or modules. Therefore, any device, system, method, etc., that possesses all the features and functions attributed to a particular device and / or interface and / or module is to be understood as constituting, comprising, or implementing the device and / or interface and / or module.

[0105] In particular, it is possible that all facilities, interfaces, or modules are implemented by program code that is executed by a computing facility.

[0106] The computing device can be implemented as any device or means for performing calculations, in particular for executing software, an application, or an algorithm. For example, the computing device can include at least one processor, such as at least one central processing unit (CPU), and / or at least one graphics processing unit (GPU), and / or at least one field-programmable gate array (FPGA), and / or at least one application-specific integrated circuit (ASIC), and / or any combination thereof. The computing device can further include main memory operationally connected to the at least one processor, and / or non-volatile memory operationally connected to the at least one processor and / or the main memory. The computing device can be implemented partially and / or entirely in a local device and / or partially and / or entirely in a remote system, such as a remote system.The invention may be implemented through a cloud computing platform. According to a twelfth aspect, the invention provides a computer program product comprising executable program code which, when executed by a computing device, is configured to perform the method according to an embodiment of the seventh or ninth aspect of the present invention.

[0107] According to a thirteenth aspect, the invention provides a non-volatile, computer-readable data storage medium comprising executable program code which, when executed by a computing device, is configured to carry out the method according to an embodiment of the seventh aspect or the ninth aspect of the present invention.

[0108] The non-volatile, computer-readable data storage medium can include or consist of any type of computer memory, in particular semiconductor memory, such as solid-state memory. The data carrier can also include or consist of a CD, DVD, Blu-ray disc, USB flash drive, or the like.

[0109] According to a fourteenth aspect, the invention provides a data stream comprising executable program code or configured to generate executable program code which, when executed by a computing device, is set up to carry out the method according to an embodiment of the seventh aspect or the ninth aspect of the present invention.

[0110] Further advantageous variants, options, embodiments, and modifications will become apparent from the following figures and the accompanying detailed description, as well as from the claims. It is understood, however, that while the detailed description and specific examples indicate preferred embodiments of the invention, they are provided for illustrative purposes only, since various changes and modifications within the scope of the invention are obvious to the person skilled in the art.

[0111] Brief description of the characters

[0112] Individual embodiments of the present disclosure are explained in detail with reference to the following figures. The components in the drawings are not necessarily to scale, but serve to illustrate the principles of the present invention. Parts in the various figures that correspond to the same elements or process steps have been provided with the same reference numerals in the figures. The numbering of process steps initially serves only to distinguish them and does not necessarily imply a corresponding sequence; however, it is one option to carry out the steps in the order of their numbering. Several steps can also be carried out overlapping or simultaneously. The figures show:

[0113] Fig. 1 is a schematic flowchart to explain a method according to an embodiment of the present invention;

[0114] Fig. 2 is a schematic block diagram to explain a computing device according to a further embodiment of the present invention and a system according to yet another embodiment of the present invention, as well as to further explain the method according to Fig. 1;

[0115] Fig. 3 is a schematic block diagram to illustrate a computer program product according to an embodiment of the present invention;

[0116] Fig. 4 is a schematic block diagram to illustrate a non-volatile, computer-readable data storage medium according to an embodiment of the present invention.

[0117] Fig. 5 shows a schematic block diagram of a system for use in a clinical setting.

[0118] Situation according to a further embodiment of the invention;

[0119] Fig. 6 shows a flowchart of a procedure for generating training data for a machine learning model for use in a clinical situation;

[0120] Fig. 7 shows a flowchart of a process for using a machine learning

[0121] Models in a clinical situation;

[0122] Fig. 8 shows a schematic block diagram of a computer program product; and

[0123] Fig. 9 shows a schematic block diagram of a non-volatile, computer-readable

[0124] Data storage medium.

[0125] Detailed description of the figures

[0126] Fig. 1 shows a schematic flowchart to explain a method according to an embodiment of the present invention, namely a computer-implemented method for generating training data for a machine learning model.

[0127] The procedure includes at least the following steps:

[0128] In step S100, sensor data is received from at least one sensor present in a clinical situation. This sensor could be, for example, an image sensor, a temperature sensor, a position sensor, an orientation sensor, a vital signs sensor, a current sensor, a voltage sensor, a pressure sensor, a flow sensor, and / or a motor sensor. Preferably, sensor data is received from multiple sensors pertaining to the same clinical situation, such as the same ongoing surgical procedure, the same patient room, or the same operating room at the same time or during the same period, or similar.

[0129] Some or all of the sensor data are preferably available as time series, meaning that the acquired sensor data is a function of time and / or continues to be acquired and received over time. The acquisition and / or reception of the sensor data can occur continuously, regularly, or in response to a trigger event.

[0130] A step S10 of acquiring sensor data by one or more sensors can optionally also be part of the method according to the invention. Alternatively, the method can also begin with receiving S100 already generated sensor data. For example, sensor data can be received from a PACS (Picture Archiving and Communication System). A hybrid form is also possible, whereby some sensor data is acquired S10 and received by the sensors S100, while other sensor data is retrieved from a database and received from there S100.

[0131] Fig. 2 shows a schematic block diagram to further explain the method from Fig. 1. Therefore, references to reference numerals from Fig. 2 will be made repeatedly in the description of the method according to Fig. 1.

[0132] At the same time, Fig. 2 also shows a computing device 100, which is designed to carry out the method according to Fig. 1, as well as a system 1000, which comprises the computing device 100. Both the computing device 100 and the system 1000 are further embodiments of the present invention.

[0133] Figure 2 shows several sensors 10, 20, 30, which can also be individually (or jointly) integrated into one or more medical instruments. For example, a camera 10 can be integrated into an endoscope (i.e., the sensor can be an endoscopic camera 10), while a current sensor 20 is integrated into a high-frequency or ultrasonic generator, and a logic sensor 30 detects the activation / non-activation of a smoke extraction system and is integrated into a component of the technical smoke extraction system.

[0134] The following describes various properties and features of the present invention by way of example, using a situation in which work is carried out on organic tissue using a high-frequency or ultrasound generator, which can result in smoke generation. The smoke can be extracted by means of a technical smoke extraction system. This is particularly important when the smoke is generated during a minimally invasive procedure where the users (e.g., surgeons) rely on the field of view of the endoscopic camera 10, as this ensures that the camera's field of view remains unobstructed. It is understood, however, that all embodiments of the present invention can also be applied to any other fields of application and clinical situations.

[0135] In the computing device 100 shown as an example in Fig. 2, the sensor data 1 are received from an input interface 101 of the computing device, which can be configured to receive the sensor data 1 wirelessly or via a wired connection, using one or more arbitrary communication protocols (e.g. Bluetooth, Ethernet, ZigBee etc.).

[0136] In step S200, at least parts (or all) of the sensor data 1 received by sensors 10, 20, 30 are fed into a first machine learning model 110 as at least part of (or as) the first input data 11. The first machine learning model 110 can, for example, be implemented by the computing device 100, as shown schematically in Fig. 2.

[0137] As explained above, the first machine learning model 110 can be one (or more) of a variety of different types of machine learning models, for example, a support vector machine or an artificial neural network, such as a convolutional neural network (CNN) or a recursive neural network (RNN). The first machine learning model 110 is configured to process the initial input data 11 in order to generate initial output data 19, which represents a recommendation for action, a control signal for a medical device 200, or information for a user.

[0138] Accordingly, in step S300 the first input data 11 is processed by the first machine learning model 110 in order to generate first output data 19, which represents a recommendation for action, a control signal for a medical device 200, or information for a user.

[0139] For example, the first machine learning model 110 can be configured to process the sensor data 1 (or parts thereof) from sensors 10, 20, 30 in order to generate a control signal (as the first output data 19 or as a part of the first output data 19) that controls a medical device 200 (or several medical devices 200). The medical device 200 could be, for example, a monitor (such as in an operating room or a recovery room), a medical instrument, a robot (such as a surgical robot), or an assistance system (e.g., a smoke extraction system).

[0140] In some variants, one or more of the sensors 10, 20, 30 may be integrated into one or more of the medical devices 200. For example, one sensor may be an endoscopic camera 10 and the medical device 200 an endoscope, with the computing unit 100 advantageously being integrated into a camera control unit of the endoscope. The control signal may, in particular, be configured to control an electrical, preferably medical, function of the medical device 200, such as changing a current or voltage level, changing a pump or suction power, changing a radiation power, and / or the like.

[0141] A recommendation for action, or other information to the user, as part of the first output data 19, can be provided to the user visually via a monitor and / or audibly via a loudspeaker and / or haptically via a vibrator, for example.

[0142] For example, the first machine learning model 110 can be configured to detect (yes / no) and / or classify (e.g., by intensity) smoke (or smoke development) during a surgical procedure, such as in a pneumoperitoneum. To do this, the machine learning model 110, in this case, for example, an artificial neural network with at least one hidden layer, can perform a deep learning analysis.

[0143] It is understood that the initial input data 11 may include, in addition to the sensor data 1, other data, such as patient data of a patient in the clinical situation (e.g. from a patient database), user input, or the like.

[0144] In step S400, at least parts of the first output data 19 and / or intermediate values ​​18 of the first machine learning model 110 are fed into a second machine learning model 120 as at least part of the second input data 21.

[0145] As previously explained, if the first machine learning model 110 is implemented as an artificial neural network, values ​​from the output layer and / or an intermediate layer (e.g., the last or penultimate layer before the output layer) can be used as part of the second input data 21. If the first machine learning model 110 comprises a pipeline of several individual artificial intelligence entities, an intermediate value can also be taken from an artificial intelligence entity that is not the last in the pipeline, for example, from its output layer. Individual values ​​or all values ​​output by a layer can be used.

[0146] It can also be provided that the second machine learning model 120 is designed as a component of the first machine learning model 110, i.e., that the second output data 29 of the second machine learning model 120 in turn go into an intermediate layer of the first machine learning model 110 (dashed arrow in Fig. 2).

[0147] The second machine learning model 120 is configured to process the second input data 21 to generate second output data 29, which indicates whether the sensor data 1 received (as part of the first input data 11) can be used (or: usable) to improve the first machine learning model 110. The second machine learning model 120 is, or includes, for example, a support vector machine, an artificial neural network, and / or an extreme learning machine. The second machine learning model 120 can be optimized in various ways, such as by ROCKET (Random Convolutional Kernel Transform) model optimization or a kernel trick.

[0148] Accordingly, in step S500, the second input data 21 is processed by the second machine learning model 120 to generate the second output data 21. In particular, the second machine learning model 120 can be configured to determine whether the first output data 19 contains an error in any way, e.g., an incorrect interpretation of the sensor data 1 (such as a misinterpretation of an object visible in camera data).

[0149] For example, in the smoke detection example, a prediction probability (e.g., the probability that smoke is visible in the current camera image of the endoscopic camera 10) can be generated in an intermediate or final layer of a first machine learning model 110, which is implemented as an artificial neural network. Alternatively, an object in the camera image can be classified as smoke, with a corresponding probability that this classification is correct (e.g., using a SoftMax output layer).

[0150] The intermediate layer or final layer, from which the intermediate value(s) 18 and / or first output data 19 are taken and used as (or as part of) the second input data 21, can optionally be dynamically selected.

[0151] The second input data 21 can, in addition to at least one intermediate value 18 or output data 19 of the first machine learning model 110, also include further data, for example parts (or all) of the sensor data 1 (either of the sensor data 1 used in the first input data 11, or of all sensor data 1 received in step S100 or from the input interface 101), or any other data or data types mentioned above or below.

[0152] According to an advantageous option, in one step S450, a user input 31 is acquired (e.g., at a user input interface 300 of the system 1000) and the user input 31 is fed into the second machine learning model 120 as part of the second input data 21. The user input 31 can, for example, include user feedback in the broadest sense on the sensor data 1 and / or on the first output data 19.

[0153] User input 31 can thus indicate, for example, whether the first output data 19 was correct, i.e., whether any information or recommendation contained therein was correct, or whether a control signal contained therein triggered or correctly prevented a correct action.

[0154] The user input 31 can be queried directly via the user input interface 300 (e.g. a touchscreen) as a reaction to the first output data 19 (e.g.: "Has the recommendation X proven to be correct?", or "Was information Y correct?"), or determined indirectly by the user input 31 contradicting the content of the first output data 19 (e.g. if the user has reversed the action carried out by a control signal of the first output data 19 by means of the user input 31), or if a user rejects the offered recommendation for action (e.g. "Smoke detected, extraction is recommended - carry it out?").

[0155] In step S600, if the second output data 29 indicates that the received sensor data 1 is to be used to improve the first machine learning model 110, at least part of the received sensor data 1 (more precisely, the sensor data 1 used as the first input data 11 or as part of the first input data 11) is stored (e.g., in a data storage device 140 of the computing device 100) and / or transmitted to an external data storage device (e.g., via an output interface 109 of the computing device). The external data storage device can, for example, be implemented as a network or cloud data storage device.

[0156] The second output data 19 can, for example, include a numerical value which is compared with a threshold value, and based on the comparison it is determined whether the received sensor data should be used to improve the first machine learning model.

[0157] The numerical value could, for example, be a probability value representing how well the first machine learning model 110 made a prediction. If this value falls below a predetermined threshold (e.g., 80%), this variant stipulates that the sensor data on which the prediction by the first machine learning model 110 was based should be used to improve the first machine learning model 110.

[0158] Advantageously, the sensor data 1 transmitted in step S600 is supplemented by at least one of the following: information about a surgical procedure during which the transmitted part of the sensor data 1 was generated; a medical specialty to which the surgical procedure belongs;

[0159] - at least part of the output data 29 of the second machine learning model; a location of the surgical procedure; a timestamp related to the surgical procedure; time information regarding a last preceding transmission of sensor data 1; a position of an input unit on a master;

[0160] - information from a patient's medical record during the surgical procedure;

[0161] - a complication detected during or noted in connection with the surgical procedure, in particular as a label for training data for supervised learning of the first machine learning model 110. The corresponding information can be retrieved from a database (e.g., the data storage 140) or received or requested via the input interface 101.

[0162] The computing unit 100 may include a control module 130, which is configured to feed in the first and second input data 11 , 21 to S200, S400, to determine whether the received sensor data 1 should be used to improve the first machine learning model 110, and / or to perform other operations as necessary.

[0163] The system 1000 according to the invention can in turn comprise the computing device 100 according to the invention, as well as at least one sensor 10, 20, 30, which is configured to acquire sensor data 1, wherein the input interface 101 of the computing device 100 is configured to receive the sensor data 1 of the at least one sensor 10, 20, 30. The system 1000 can also comprise one or more medical devices 200, which receive the first output data 19 completely or partially and are, in particular, controllable or controlled by a control signal of the first output data 19. Furthermore, the system 1000 can comprise a user input interface 300, by means of which user inputs 31 can be acquired, for example, user feedback (or: user reactions) to the first output data 19.

[0164] Fig. 3 shows a schematic block diagram of a computer program product 400 according to an embodiment of the third aspect of the present invention. The computer program product 400 comprises executable program code 450, which, when executed (e.g., by a computing unit 100), is configured to perform or control the method according to an embodiment of the present invention, for example, according to Fig. 1 or Fig. 2.

[0165] Fig. 4 shows a schematic block diagram of a non-volatile, computer-readable data storage medium 500 according to an embodiment of the present invention. The data storage medium 500 comprises executable program code 550, which, when executed (e.g., by a processing unit 100), is configured to perform or control the method according to an embodiment of the present invention, for example, according to Fig. 1 or Fig. 2.

[0166] The non-volatile, computer-readable data storage medium 500 can, for example, be designed as or comprise a semiconductor memory, e.g., an SSD. The data storage medium 500 can also comprise or comprise a CD, DVD, Blu-ray disc, or a magnetic storage device.

[0167] Another aspect is illustrated in figures 5 to 9.

[0168] Figure 5 shows a schematic block diagram of a 300' system that can be used in a clinical setting. The clinical setting may refer to a situation in a building or vehicle dedicated to medical purposes, for example, a medical research institute, a laboratory, a hospital, a medical university, a doctor's private practice, or the interior of an ambulance.

[0169] The system 300' includes a device 100' for generating training data for a machine learning model and a device 200' for using the machine learning model.

[0170] The device 100' for generating training data for the machine learning model includes a first interface 101' which receives sensor data from one or more sensors. The first interface 10T can be a wired or wireless interface, such as a USB interface, optical interface, or the like.

[0171] The data is stored in a first storage device 103', such as a semiconductor memory, memory stick or the like.

[0172] The sensor data can include image data received from at least one image sensor. The sensor data can also additionally or alternatively include audio data from at least one audio sensor and / or video data from at least one video sensor. The sensor data can be recordings of the environment surrounding a patient being treated, or recordings of the patient's internal body, for example, using a probe, X-rays, ultrasound images, or similar methods.

[0173] The first interface 10T continues to receive situation information, which depends on a change in the state of a facility in the clinical situation.

[0174] The situational information may additionally or alternatively depend on an action taken by a person in the clinical situation. This action may consist of activating or deactivating a piece of equipment used in the clinical situation. It may also consist of performing a specific movement of a piece of equipment used in the clinical situation, such as a surgical instrument. Finally, it may consist of the person uttering a specific word or phrase.

[0175] Another example of an action is the user pressing a foot switch to generate an image. This suggests that corresponding sensor data is important and should therefore be considered when training the machine learning model.

[0176] More generally, situational information can encompass several pieces of information. In the case of smoke detection during an endoscopy, this can include whether the endoscope is inside the body, whether the procedure is in a phase where smoke can be generated, whether a high-frequency function is activated that could contribute to smoke development, and / or whether image processing can determine whether optical flow is occurring, whether the sensor image is generally blurry, or whether it has sharp segments in the focus area.

[0177] Situational information can be received externally or determined from received sensor data. Furthermore, it may be possible to consider additional sensor data to determine the situational information.

[0178] The device 100' for generating training data for the machine learning model further comprises a first computing unit 102', which includes, for example, a processor, an application-specific integrated circuit, or the like. The computing unit 102' automatically annotates at least a portion of the sensor data using the situation information. The training data is then generated using the annotated sensor data.

[0179] Depending on the situation information, it can be determined whether the sensor data is automatically annotated or discarded.

[0180] The training data can be used for supervised learning of a machine learning model.

[0181] The training data generally comprises a variety of data pairs, each consisting of an input and a corresponding target. The input can be generated from sensor data. For example, the input could be a camera image, a video sequence, or an audio sample. The target corresponds to the appropriate label, which is determined through automatic annotation. For example, the input can be classified, meaning the label corresponds to its assignment to a specific class.

[0182] For example, the target variable can correspond to a system state. In an endoscopy, for instance, a distinction can be made between two system states, one where flue gas is present and the other where it is not. The system state thus describes the presence of flue gas. The machine learning model can then be trained to detect flue gas.

[0183] More generally, automatic annotation can be used to generate a dataset that is then divided into training data and test data. After the machine learning model is trained with the training data, the test data is used to evaluate the performance of the machine learning model. The device 100' for generating training data for the machine learning model can also be configured to generate the machine learning model itself. In this case, the device 100' trains the machine learning model using the annotated sensor data.

[0184] The clinical situation could be, for example, a video endoscopy. The machine learning model can then be trained for use in smoke detection, progress monitoring of the video endoscopy, or energy monitoring of a high-frequency device used in the video endoscopy.

[0185] The device 200' for using the machine learning model includes a second interface 201 which receives sensor data from at least one sensor present in the clinical situation. The second interface 201' can be a wireless or wired interface.

[0186] The sensor data is stored in a second memory 203', such as a semiconductor memory, memory stick or the like.

[0187] A second computing unit 202' uses the machine learning model to generate a recommended course of action. Alternatively or additionally, an action can be carried out automatically. The sensor data is used as input data for the machine learning model.

[0188] For example, during an endoscopy, the second computing unit 202' can use the machine learning model to decide whether to activate smoke extraction. For this purpose, the machine learning model can be trained as a classifier that detects the presence of smoke.

[0189] The device 200' for using the machine learning model further comprises a reaction information determination unit 204', which determines reaction information depending on a person's reaction to the recommended action and / or to the action performed. The reaction information determination unit 204' can be identical to the second computing unit 202' or be a separate component.

[0190] The response information can include whether the person acted in accordance with the recommended course of action. It can also include whether the person canceled or reversed the automatic execution of the action. For example, if the user frequently performs a reversal (i.e., interrupts the action), this may indicate that the action should not have been performed. The machine learning model can then be adapted based on the response information, for example, by automatically adjusting the weights of layers in a neural network of the machine learning model. This enables continuous improvement of the machine learning model without the need for manual retraining.

[0191] The response information can be used to evaluate the machine learning model. Every single action can be traced. Because the process can be executed autonomously, self-learning, and locally, all values ​​and parameters are available during runtime. This allows all actions to be evaluated and documented retrospectively without gaps. A chain of evidence can thus be generated for quality assurance and potential regulatory approval.

[0192] The machine learning model can also be adapted depending on the user profile of the responding person.

[0193] Furthermore, it may be provided that an algorithm is used to calculate a reliability metric, indicating how reliable the recommended course of action and / or the automatic execution of the action is. The algorithm is adapted depending on the response information. For example, a specific action can only be executed automatically if the reliability metric exceeds a predefined threshold, which may depend on the type of action.

[0194] Furthermore, it can be provided that if the reliability value falls below a predefined threshold, the user is given a recommendation for action. The user can trigger this recommendation automatically, for example, by activating a control element (such as a button). Depending on whether the user performs the action or not, the calculation of the reliability value is adjusted accordingly and / or the machine learning model can be retrained. For instance, the sensor data can be automatically annotated based on the user's response. This allows the machine learning model to be improved.

[0195] Figure 6 shows a flowchart of a procedure for generating training data for a machine learning model for use in a clinical setting. The procedure can be performed using the device 100' described above for generating training data for a machine learning model.

[0196] In step S101, sensor data is received from at least one sensor present in the clinical setting. This sensor data may include image data received from at least one image sensor. The sensor data may also additionally or alternatively include audio data from at least one audio sensor and / or video data from at least one video sensor. The sensors may be located in the vicinity of a patient. However, the sensors may also be located on or in a device used during an examination or surgery.

[0197] In step S102, situation information is received, which depends on a change in the state of a device in the clinical situation and / or an action by a person in the clinical situation. The action of a person in the clinical situation could consist of activating or deactivating a device used in the clinical situation, such as a smoke extraction system.

[0198] A person's action may consist of performing a specific movement of a device used in the clinical situation, such as moving a medical instrument in a specific direction.

[0199] The action may also include the person uttering a specific word or sentence, such as giving an instruction to an assisting person or during an assessment.

[0200] In step S103', at least some of the sensor data is automatically annotated using the situation information, thereby generating training data.

[0201] For example, several scenarios can occur in a clinical situation. The sensor data can be used to determine which scenario is most likely, and the sensor data is annotated accordingly. Another example involves progress detection. Here, the sensor data is used to determine how far an examination or operation has progressed, and the sensor data is annotated accordingly.

[0202] Figure 7 shows a flowchart of a procedure for using a machine learning model in a clinical setting. The machine learning model can be based on the procedure described in Figure 6. In particular, training data for the machine learning model can be generated according to the procedure described there. The machine learning model can then be trained based on the generated training data. The machine learning model thus provided is then used in the procedure described below.

[0203] In step S20T, an interface 20T receives sensor data from at least one sensor present in the clinical situation, such as a sensor attached to a medical device or a sensor permanently installed in the environment.

[0204] In step S202', a computing unit 202' uses a machine learning model to generate a recommendation for action and / or to automatically execute an action. The sensor data is used as input data for the machine learning model. Based on the sensor data, the machine learning model determines which action should be recommended or executed.

[0205] In step S204', a reaction information determination unit 204' determines reaction information depending on a person's reaction to the action recommendation and / or to the action performed. For example, it can be determined whether the person accepts the action recommendation or accepts the automatically executed action, i.e., does not interrupt it.

[0206] Figure 8 shows a schematic block diagram of a computer program product 400'. The computer program product 400' comprises executable program code 401' which, when executed (e.g. by a computing device), is configured to perform the method according to an embodiment of the present invention, for example according to Figure 6 or 7.

[0207] Figure 9 shows a schematic block diagram of a non-volatile, computer-readable data storage medium 500' according to an embodiment of the present invention. The data storage medium 500' comprises executable program code 501' which, when executed (e.g., by a computing device), is configured to perform the method according to an embodiment of the present invention, for example, according to Figure 6 or 7.

[0208] The non-volatile, computer-readable data storage medium 500' can, for example, be designed as or comprise a semiconductor memory, e.g., an SSD. The data storage medium 500' can also comprise or include a CD, DVD, Blu-ray disc, or a magnetic storage device.

[0209] The foregoing description of the disclosed embodiments contains only examples of possible implementations, which are described to enable a person skilled in the art to manufacture or use the present invention. Various variations and modifications of these embodiments are readily apparent to a person skilled in the art – upon knowledge of the present invention – and the general principles defined herein can be applied to other embodiments without departing from the scope of this disclosure.

[0210] Therefore, the present invention is not to be limited to the specific embodiments shown herein, but is to be granted the broadest scope that is consistent with the principles and features disclosed herein. List of reference numerals

[0211] I Sensor data

[0212] 10 Endoscopic camera

[0213] II first input data

[0214] 18 Intermediate value

[0215] 19 first output data

[0216] 20 Current sensor

[0217] 21 second input data

[0218] 29 second output data

[0219] 30 Logical Sensor

[0220] 31 User input

[0221] 100 computer equipment

[0222] 101 Input interface

[0223] 109 Output interface

[0224] 110 first machine learning model

[0225] 120 second machine learning model

[0226] 130 Control module

[0227] 140 data storage devices

[0228] 200 medical devices

[0229] 300 User Input Interface

[0230] 400 computer program product

[0231] 450 program code

[0232] 500 data storage medium

[0233] 550 Program Code

[0234] 100' Device for generating training data for a machine learning model

[0235] 101 ' first interface

[0236] 102' first computing device

[0237] 103' first storage

[0238] 200' Device for using a machine learning model

[0239] 201 ' second interface

[0240] 202' second computing facility

[0241] 203' second storage

[0242] 204' Response Information Gathering Unit

[0243] 300' System

[0244] 400' Computer program product

[0245] 401 ' Program code

[0246] 500' data storage medium

[0247] 501 ' Program code S10..S600

[0248] Procedural steps

[0249] S10T to S103', S201' to S203'

[0250] Procedural steps

Claims

Patent claims 1. Computer-implemented method for generating training data for a machine learning model, comprising the following steps: Receiving (S100) sensor data (1) from at least one sensor (10; 20; 30) present in a clinical situation; Feeding (S200) at least parts of the received sensor data (1) as at least part of first input data (11) into a first machine learning model (110); Processing (S300) the first input data (11) by the first machine learning model (110) to generate first output data (19) which represents a recommendation for action, a control signal for a medical device (200), or information to a user; Feeding (S400) at least parts of the first output data (19) and / or intermediate values ​​(18) of the first machine learning model (110) as at least part of second input data (21) into a second machine learning model (120); Processing (S500) the second input data (21) by the second machine learning model (120) to generate second output data (29) which indicates whether the received sensor data (1) should be used to improve the first machine learning model (110), and, if so: Storing and / or transmitting (S600) to a data storage device (140) at least some of the received sensor data (1) as a basis for training data for the first machine learning model (110).

2. Method according to claim 1, wherein the second machine learning model (120) comprises a support vector machine, an artificial neural network, and / or an extreme learning machine.

3. Method according to claim 1 or 2, wherein the second machine learning model (120) is configured such that the second output data (29) includes a numerical value which is compared with a threshold value, and a result of the comparison indicates whether the received sensor data (1) is to be used to improve the first machine learning model (110).

4. Method according to any one of claims 1 to 3, further comprising: dynamically selecting an intermediate layer or a final layer of the first machine learning model (110); wherein at least one intermediate value (18) of the first machine learning model (110), which is used as part of the second input data (21), is derived from the dynamically selected intermediate layer or final layer.

5. Method according to any one of claims 1 to 4, wherein the sensor data (1) of the at least one sensor (10, 20, 30) are or comprise at least one of the following: Image and / or video data from an image sensor (10); - Temperature data from a temperature sensor; Position data from a position sensor; - Alignment data from an alignment sensor; - A patient's vital signs; - Current data from a current sensor; - Voltage data from a voltage sensor (20); Pressure data from a pressure sensor; Flow data from a flow sensor; - Engine data of an engine, such as speed and / or acceleration and / or torque data; - Device settings of a medical device (200).

6. A method according to any one of claims 1 to 5, further comprising: Supplementing the stored or transmitted part of the sensor data with at least one of the following: - information about a surgical procedure during which the transmitted part of the sensor data (1) was generated; - a medical specialty to which surgical intervention belongs; - at least part of the input data (29) of the second machine learning model (120); - a site of the surgical procedure; - a timestamp related to the surgical procedure; time information relating to a last preceding transmission of sensor data; a position of an input unit on a master; information from a patient's medical record in the surgical procedure; a complication identified during or noted in connection with the surgical procedure; in particular, each as a label for training data for supervised learning of the first machine learning model (110).

7. Method according to any one of claims 1 to 6, further comprising a capture (S450) of a user input (31); and Feeding the captured user input as a further part of the second input data (21) into the second machine learning model (120).

8. Method according to any one of claims 1 to 7, wherein the first machine learning model (110) is configured for the detection and / or classification of smoke development during a surgical procedure.

9. Computing device (100) configured to perform the method according to any one of claims 1 to 8, optionally integrated into a camera control system.

10. System (1000) comprising a computing device (100) according to claim 9 and at least one sensor (10, 20, 30) which is configured to acquire sensor data (1) from at least one clinical situation.

11. Computer-implemented method for generating training data for a machine learning model for use in a clinical setting, comprising the following steps: Receiving (S10T) sensor data from at least one sensor present in the clinical situation; Receiving (S102') situational information which depends on a change in the state of an institution in the clinical situation and / or an action of a person in the clinical situation; and Generating (S103') the training data by automatically annotating at least some of the sensor data, using the situation information.

12. Method according to claim 11, wherein the machine learning model is trained using the annotated sensor data.

13. Method according to claim 11 or 12, wherein, depending on the situation information, it is determined whether the sensor data is automatically annotated or discarded.

14. Method according to one of the preceding claims, continuing with the step: Determining situational information based on sensor data and / or other sensor data.

15. Method according to any of the preceding claims, wherein the action of a person in the clinical situation comprises at least one of activating or deactivating a device used in the clinical situation by the person, a certain movement of a device used in the clinical situation by the person, and a utterance of a certain word or phrase by the person.

16. A method according to any of the preceding claims, wherein the clinical situation comprises video endoscopy, wherein the sensor data comprises image data and / or video data, and the machine learning model is trained for use in smoke gas detection, progress detection of video endoscopy and / or energy monitoring of a high-frequency device used in video endoscopy.

17. Procedure for using a machine learning model in a clinical situation, comprising the following steps: Receiving (S201 ') sensor data from at least one sensor present in the clinical situation; Using (S202') a machine learning model to generate an action recommendation and / or to automatically perform an action, using the sensor data as input data for the machine learning model; and Determine (S203') a response information depending on a person's response to the recommendation for action and / or to the action carried out.

18. The method of claim 17, wherein the machine learning model is adapted depending on the reaction information.

19. Method according to claim 17 or 18, wherein the response information includes information on whether the person acted in accordance with the recommended course of action.

20. Method according to any one of claims 17 to 19, wherein the response information includes information on whether the person has cancelled or reversed the automatic execution of the action.

21. Method according to one of claims 17 to 20, wherein a reliability parameter is further calculated using an algorithm, which indicates how reliable the recommendation for action and / or the automatic execution of the action is, and wherein the algorithm is adapted depending on the reaction information.

22. Method according to one of claims 17 to 21, wherein the machine learning model is adapted depending on a user profile of the responding person.

23. Device (100') for generating training data for a machine learning model for use in a clinical situation, comprising: an interface (10T) designed to: to receive sensor data from at least one sensor present in the clinical situation, and to receive situational information that depends on a change in the state of a facility in the clinical situation and / or an action by a person in the clinical situation; and a computing device (102') which is trained to automatically annotate at least part of the sensor data using situation information in order to generate training data.

24. Device (200') for using a machine learning model in a clinical situation, comprising: an interface (201') configured to receive sensor data from at least one sensor present in the clinical situation; a computing unit (202') configured to use a machine learning model to generate a recommendation for action and / or to automatically perform an action, wherein the sensor data are used as input data for the machine learning model; and a response information determination unit (204') configured to determine response information depending on a person's response to the recommendation for action and / or to the action performed.

25. System (300') for use in a clinical situation, comprising: a device (100') for generating training data for a machine learning model according to claim 23; and a device (200) for using the machine learning model according to claim 24.

Images