Medical training system

Abstract

A medical training system comprises a manually operated simulator apparatus (1) configured to perform at least one medical procedure in a simulated medical procedure setup. The simulator apparatus comprises an input unit, a control unit and at least one manually operated tool and a sensor detecting movements of the manually operated tool. The control unit is configured to provide at least one medical procedure setup and to assess the detected activities. The medical training system further comprises a proctor unit (3) with a virtual proctor acting as an instructor for the user (1). The virtual proctor is configured to receive information of the control unit of the simulator apparatus, to bi-directionally communicate with the user during a training session, to analyse the user's performance, and to provide feedback based on the analysis performed. The inventive medical training system trains the user efficiently and in a targeted manner.

Description

[0001] TITLE

[0002] MEDICAL TRAINING SYSTEM

[0003] TECHNICAL FIELD

[0004] The present invention relates to a medical training system for medical procedures, especially for surgical operations.

[0005] PRIOR ART

[0006] Medical procedures, especially surgical operations, such as endoscopy, arthroscopy, laparoscopy, and other minimally invasive surgery applications, can be trained in medical training systems which simulate the medical procedure setup. The users, namely trainees, physicians, and surgeons, learn to master the indirect hand-eye coordination required by the manipulation of medical instrumentation, such as an endoscope or an ultrasound probe, in addition to the conventional medical instruments and procedures. Computerized medical training simulators enable the users to develop and improve their practice in a virtual reality environment before practicing in the real world-operation room.

[0007] In some embodiments, such medical training systems provide a mixed reality scenario where the user jointly interacts with real objects in a physical environment and in a related virtual environment. These simulators include a screen or virtual reality headset for displaying the virtual reality and a human anatomy model in real size, such as a joint model or an organ model, as well as a tangible medical procedure instrument. The model is adapted with sensors and mobile members for guiding, tracking, and controlling the medical instrument operation within the anatomy model. Such medical training systems are for example disclosed in WO 2014 / 041491 A1, US 2015 / 0325151 A1, and WO 2020 / 164829 A1.

[0008] These medical training systems provide instructions and visual and / or auditory feedback that guide the user during the procedure. These instructions and the feedback are hard-coded and provide the same guidance for all users, regardless of their specific present performance, abilities and skills.

[0009] WO 2024 / 006348 A1 discloses a system for clinical procedure training including a physical model of an anatomic region including a position sensor, a haptic sensor and a medical tool operable to interact with the physical model. The system comprises a controller having a display, input / output hardware and connections connecting components of the training system and allowing signal transmission between the components of the clinical procedure training system. The controller comprises a feedback module and a recommendation module. The feedback module gives immediate feedback to the user, the recommendation module may recommend additional, which may include machine learning algorithms or neural networks. The recommendation module may recommend additional training using a learning model, which incorporates games and other modes fortraining. The training may be one-to-one or one-to-many learners. A single user may interact with an instructor in realtime.

[0010] SUMMARY OF THE INVENTION

[0011] It is therefore an object of the present invention to provide an improved medical training system for medical procedures, especially for surgical operations, which trains the user efficiently and in a targeted manner.

[0012] This object is achieved by a medical training system with the features of claim 1.

[0013] The inventive medical training system comprises

[0014] a manually operated simulator apparatus configured to perform at least one medical procedure in a simulated medical procedure setup, the simulator apparatus comprising

[0015] - an input unit enabling a user to enter data and commands into the simulator apparatus,

[0016] - a control unit configured to provide the at least one medical procedure setup and to assess activities during the performance of the at least one medical procedure,

[0017] - at least one manually operated tool, the tool being at least one of

[0018] o a tangible medical instrument,o a tool for teleoperating a surgical instrument,

[0019] o a manually moveable virtual medical instrument displayed on a touchscreen of the simulator assembly, and

[0020] o a manually moveable tool for teleoperating a surgical instrument, the last-named tool being displayed on a touchscreen of the simulator assembly,

[0021] and

[0022] - at least one sensor detecting movements of the tool.

[0023] The medical training system further comprises a proctor unit with a virtual proctor acting as an instructor for the user, the virtual proctor being configured

[0024] to receive information of the control unit of the simulator apparatus,

[0025] to bi-directionally communicate with the user during a medical procedure simulated by the simulator apparatus,

[0026] to analyse the user's performance, and

[0027] to provide feedback based on the analysis performed.

[0028] Preferably, the simulator apparatus also comprises a display. Depending on the embodiments, the display is just a display or it is a touchscreen, preferably comprising at least one of the at least one sensor.

[0029] This medical training system allows customized training for each individual user by providing a simulator apparatus capable of performing standardised medical procedures combined with a virtual proctor capable of considering special needs of the individual user. The level of guidance and the topics to be trained can therefore be adapted and even be created individually. The inventive medical training system can replace a human proctor, or it can even provide an enhanced alternative to a human proctor. The time needed by human proctors decreases, thereby also decreasing the costs of training.

[0030] The further proctor unit is a separate execution module outside of the simulator apparatus control with an own logic. The proctor unit is an additional unit, i.e. additional to the control unit or intelligence of the simulator apparatus. The proctor unit is universally applicable. It may communicate not only with the user, but also with the simulator apparatus, especially with the control unit. Communication with the user is preferably by using spoken language and / or by displaying texts, symbols or signs on a screen, preferably of the simulator apparatus. Preferably, the communication of the proctor unit with the user is multimodal. Preferably, there is bidirectional, structured communication between the proctor unit andthe simulator apparatus, preferably including state synchronization of the exercise, events and assessment data. Feedback is preferably not just displayed on a screen but it is preferably a closed-looped control cycle with inputs from the proctor that influence guidance, workflow and assessment.

[0031] Since the proctor is an additional unit, it may be used with more than one simulator apparatus, even with different types of simulator apparatus. The principle of the training system comprising a separate artificial intelligence proctor unit is therefore scalable.

[0032] Existing simulator technology can be used in the inventive training system, i.e. existing simulator apparatus can be used. The simulator apparatus may be a mixed reality scenario medical training system enabling the user to interact with real objects using real medical instruments in a physical environment and a virtual environment. Such simulator devices are produced and sold by the applicant. In other embodiments, the simulator apparatus is simulating a teleoperated medical treating system, especially a robotic surgery system. The simulator apparatus may simulate the well-known da Vinci® system and / or the well-known MM I Symani® Surgical system.

[0033] The simulator apparatus may also be a Box Trainer. Box Trainers typically consist of a box with at least one insert designed to mimic specific anatomical features or surgical scenarios. The inserts are tangible and represents, for example, cavities, incisions, or pathways for tangible instruments to navigate. The Box Trainers are used for training specific skills, such as performing specific movements.

[0034] Instead or in addition to any of the different types of simulator apparatus mentioned above, a touchscreen may be present allowing to display a tool, such as a medical instrument or a tool of a teleoperated system. By touching the touchscreen with at least one finger, the user can manually move and operate this virtual tool in order to perform the at least one medical procedure. This virtual tool and therefore the simulator apparatus is therefore manually operated as well.

[0035] The existing simulator apparatuses are operated as before, i.e. the interaction between the simulator apparatus and the user is unchanged. However, the virtual proctor of the additional proctor unit communicates as a third party with the user and the simulator apparatus, thereby guiding the user through the training session based on feedback of the simulator apparatus and of the user.Training time can therefore be utilised in a more efficient and more result-oriented to improve the skill of the user.

[0036] The virtual proctor preferably comprises artificial intelligence (Al). Preferably, the virtual proctor is not static over time but is configured to be trained and deployed, thereby improving the future training. The virtual proctor preferably looks at the current performance of the user, in the form of data, performance metrics, and user interactions. The virtual proctor guides the user with proctor feedback, in the form of at least one of video, text, audio, pictures and haptic feedback or combinations thereof.

[0037] Having a proctor unit being in addition to the control unit of the simulator apparatus enables the use of such a proctor unit for different types of simulator apparatus, either with identical set up of the virtual unit proctor or with only slight amendments to it. This makes it easier, less time consuming and therefore also less expensive to create proctor units with virtual proctors for additional simulator apparatus. Also updating medical training systems with improved proctor units is feasible in an easy way.

[0038] Preferably, the virtual proctor communicates orally with the user, i.e. with spoken language. Like this, the user can focus his / her view onto the task to be solved, preferably to the physical human anatomy model or to the screen displaying a human anatomy image.

[0039] The virtual proctor communicates with the user during the performance of the medical procedure, giving feedback and guidance in order to enable the user to realize the goals set for the specific medical procedure. Each user is therefore provided with guidance adapted to his / her strengths and weaknesses. The guidance and feedback are preferably in real-time. Preferably, the virtual proctor is configured to change the training in real time after receipt of feedback from the simulator apparatus and / or the user. Feedback from the simulator apparatus is preferably mainly based on the at least one sensor detecting the behaviour of the user, such as the movements of the manually operated tangible and / or virtual tools. Preferably, a multiple of sensors are present in the simulator apparatus. At least some of the sensors are, depending on the embodiments, located at or in the manually operated tools and / or in the touchscreen and / or at or in the human anatomy model.

[0040] The virtual proctor preferably bases the guidance on how experts completed the tasks in the past. The virtual proctor preferably uses information provided by the simulationapparatus, for example when the at least one sensor of the simulation apparatus detects that the user moves the medical tool along a correct or a wrong path or when the user would have injured tissue in a real surgery.

[0041] The medical procedures are procedures known in the prior art, such as finding optimum trajectories, limiting numbers of contact with surfaces, such as patient's tissue, optimize visualization, optimize tool handling. In addition or alternatively, the medical procedures are training of medical tool basic skills, especially in Box Trainers.

[0042] Preferably, a training comprises a goal defined in advance. Depending on the goal defined, a specific standard medical procedure provided by the simulator apparatus is performed and the virtual proctor is guiding through this medical procedure. Preferably, the medical training system is also configured such that the virtual proctor can combine standard medical procedures or amend standard medical procedures to provide an improved customized new medical procedure. This is preferably achieved by the configuring the control unit of the simulator apparatus to receive and execute instructions from the virtual proctor of the proctor unit.

[0043] The proctor of the medical training system is preferably an independent execution unit, local or cloud-based, which is distinct from the control unit of the simulator apparatus, which communicates bidirectionally with the user, preferably via multimodal interaction channels including preferably voice and text and preferably also symbols and / or signs, which autonomously analyses user actions and sensor data to derive context-dependent procedural feedback to the user, preferably also to the control unit, wherein the proctor preferably coordinates with the control unit through a structured data-exchange protocol for real-time performance assessment and adaptive guidance.

[0044] The medical training system comprises a distributed architecture enabling the proctor unit to operate autonomously and independently from the simulator control unit.

[0045] In preferred embodiments, the proctor unit is a non-transitory computer readable medium comprising instruction, which, when executed on one or more processors, implement a software program configured

[0046] to receive information of the control unit of the simulator apparatus,

[0047] to bi-directionally communicate with the user during a medical procedure simulated by the simulator apparatus,to analyse the user's performance, and

[0048] to provide feedback based on the analysis performed

[0049] The instructions to be executed can be hardcoded. Preferably, they are soft-coded. This enables more flexibility of the virtual proctor in training the user as well as in training itself, i.e. in self-learning. Preferably, the proctor is an Al-proctor logic (Al = Artificial Intelligence) which is implemented as a modular, soft-coded framework allowing modification or replacement of behavioral rules without physical changes to the simulator hardware.

[0050] "Hardcoded" means that the guidance is directly defined in the software code. The parameters can react dynamically based on inputs but if an action is repeated, the same guidance is to be expected until the source code is modified and released as a new version or edition of a product; "soft-coded" means that the guidance is based on changeable inputs, preferably an artificial intelligence model, database, or configuration. Preferably, these inputs can be updated on remote training systems based on models and databases available on a remote server and that can be dynamically applied to the training scenario without intervention by the user of the system.

[0051] The proctor logic is preferably modular allowing behavioral changes without hardware modification. Up-dates of the system are made easier.

[0052] The non-transitory computer readable medium can be implemented in the simulator apparatus, or it can be implemented in a separate device. Preferably, the proctor unit is a non-transitory computer readable medium being separate from the control unit of the simulator apparatus and being configured to be executed on one or more processors being separate from the control unit of the simulator apparatus. These one or more separate processors can be integrated in the simulator apparatus, or they can be implemented in a separate device. Having them located in separated in device enables the system to comprise more than one simulator apparatus combined with one single proctor unit, wherein the proctor unit is configured to work together with each of the simulator apparatus. Preferably, the Al-Proctor unit operates as a separate computing process or distributed service communicating with the simulator via a network interface.

[0053] Preferably, the proctor unit is configured to bi-directionally communicate with the user. Preferably, the bi-directional communication is orally, i.e. with spoken language in both directions. By using natural-language dialogue based on speech recognition and synthesisand by providing verbal feedback, guidance, and evaluation , the user can not only focus his / her view onto the task to be solved, but he / her can also keep the hands on the manually operated tools used to perform the medical procedure.

[0054] In simple embodiments, the communication between the control unit of the simulator apparatus and the virtual proctor of the proctor unit is only one-directional, i.e. from the simulator apparatus to the proctor unit. In preferred embodiments however, the communication is bi-directional. In such preferred embodiments, two principal modalities of communication exist. One modality comprises communication on a human level, in which the virtual proctor transmits guidance, instructions, or questions to the user (e.g., verbally or via text or symbols or signs) and receives responses from the user. The other modality comprises digital communication between the proctor unit and the control unit of the simulator apparatus, by which the proctor unit analyses real-time data received from the simulator apparatus, and conversely, provides digital instructions to modify or reconfigure the ongoing training session. Because the virtual proctor can issue commands directly to the control unit, the current training session may be more effectively customized. For example, the virtual proctor may cause the control unit of the simulator apparatus to skip some tasks or to repeat some tasks of the current training procedure in this training session. In some embodiments, the virtual proctor can cause the control unit of the simulator apparatus to even enter single tasks from another training procedure of the simulator apparatus into the current training session.

[0055] Preferably, the proctor unit is configured to enter instructions into the simulator apparatus based on the analysis of the user's performance. The analysis may be based on the present performance and / or on past performance. This improves the customisation of the training session. Such an approach not only enhances the training session’s adaptability but also ensures that the user receives personalized guidance on both a human-communication level and a system-control level. By leveraging these two complementary modalities of communication, the invention affords a more interactive and efficient training environment compared to systems that rely solely on unidirectional communication.

[0056] Preferably, the simulator apparatus, preferably the control unit, transmits structured procedural data, including step identifiers, sensor-derived metrics, and timestamps, to the proctor unit for synchronized analysis. Preferably, the proctor unit returns structured feedback packets comprising evaluation results, corrective instructions, and progression markers to the simulator apparatus, preferably to the control unit. Preferably, acommunication protocol implements state synchronization to maintain consistency between the simulator apparatus and the proctor unit during real-time operation.

[0057] The proctor unit performs real-time cognitive evaluation of the user’s performance, communicates bidirectionally via speech or text or symbols or signs, and adapts its instructional behavior based on procedural context, historical performance, and patientspecific parameters. This separation allows the proctor unit to evolve through software updates and learning processes without requiring modifications to the simulator apparatus, thereby enhancing scalability and maintainability.

[0058] The communication between the simulator apparatus and the proctor unit is preferably achieved via a defined protocol that supports structured data exchange, enabling consistent synchronization between both systems during training sessions. The result is an interactive training environment combining real-time procedural simulation with intelligent conversational guidance.

[0059] In simple embodiments, data to be used to perform a medical procedure and therefore to perform a training session are stored in the simulation apparatus and / or in the proctor unit. However, the proctor unit preferably does not comprise a general data storage. In preferred embodiments, the medical training system further comprises a training data storage unit, which is preferably a unit external to the simulation apparatus and the proctor unit. It is preferably a distinctive execution entity. Preferably it is a cloud. This enables high adaptability and scalability.

[0060] Preferably, the simulator apparatus and the proctor unit are configured to independently retract data from and upload data to the training data storage unit. The training data storage unit may comprise for example at least some of the following data: trajectory data points from the instrumentation and / or anatomical elements, orientation data points from the instrumentation and / or anatomical elements, interactions and collisions between modelled elements of the virtual environment, physical parameters and dimensions of the instrumentation and / or anatomical elements, duration of training of sub-tasks, measured deviations from defined parameters of the training scenarios for areas such as safety and / or efficiency.

[0061] Preferably, the training data storage unit is configured to receive data entered directly into the training data storage unit by the user. Such data can be for example personal data ofthe user, achievements of the user, objectives of the user, preferences of the user, handicaps of the user or other data which the user considers to be important. Additionally or alternatively, the user may enter such data into the simulator apparatus or into the proctor unit.

[0062] In some embodiments, the medical training system does not use any further data collections. In preferred embodiments, the medical training system comprises in addition a human expert data collection storage unit, wherein at least the proctor unit is configured to retract data from the training data storage unit. In some embodiments, only the proctor unit is configured to retract data from this storage unit. The human expert data collection storage unit may comprise for example at least some of the following data: experience reports of human experts, scientific papers, medical handbooks, designated expert runs from the training system.

[0063] The simulator apparatus is preferably a device known in the state of the art. In some embodiments, it comprises at least one physical human anatomy model, such as an insufflated abdominal cavity, joint of a limb, or pelvic anatomical models. In other embodiments, the simulator apparatus is a training unit of a teleoperated surgery system, such as the da Vinci® Surgical System.

[0064] In some embodiments, the medical training apparatus comprises one single simulator apparatus configured to simulate one or more medical procedures and one single proctor unit. In other embodiments, the medical training system comprises two or more simulator apparatus, each being configured to perform at least one medical procedure, and wherein the one proctor unit is configured to perform the medical procedures with the two or more simulator apparatus.

[0065] The communication between the control unit of the simulator apparatus and the virtual proctor of the proctor unit may be realized with classical means, such as interactive graphical control elements in a user interface. Preferably, the simulator apparatus, especially the control unit, is configured to use text-based prompts to inform or instruct the proctor unit and the proctor unit is configured to act based on text-based prompts received from the simulator apparatus. Likewise, the proctor unit is preferably configured to use textbased prompts to inform or instruct the simulator apparatus and the simulator apparatus is configured to act based on text-based prompts received from the proctor unit. Preferably, both units, i.e. the simulator apparatus and the proctor unit use artificial intelligence tounderstand the prompts and to act according to the prompts, preferably, they use LLM, i.e. large language model. 15. The proctor unit comprises at least an artificial intelligence module. Preferably, the module is a natural language understanding and generation module, such as GPT-4, Dialogflow NLU, or LUIS. In some embodiments, speech recognition and synthesis modules, such as Polly, may be implemented. In some embodiments, conversational Al (Artificial Intelligence) and dialogue management modules, such as ChatGPT, Watson Assistant, or Rasa, may be implemented. In some embodiments, the proctor has a virtual avatar to simulate emotions and non-verbal cues.

[0066] The inventive medical training system preferably combines advanced machine learning models — including deep learning neural networks, gesture recognition algorithms, anomaly detection systems, reinforcement learning agents, and NLP techniques — adaptive learning mechanisms, personalized user modeling, and interactive dialogue capabilities in order to achieve a high efficiency. The proctor unit provides the user with a comprehensive and immersive training experience that closely replicates the insights and feedback of human expert instructors.

[0067] Some embodiments provide training sessions for the user based on a standard patient model. Only the training sessions are customized, and the standard patient is kept fixed. In enhanced embodiments, the medical training system comprises in addition a twin patient unit for generating a virtual patient. The virtual patient is a twin of a real human patient, comprising the specific features of this real patient. For example, the virtual patient has the same unusual shape of the gallbladder of the real human patient and / or it has the same curve in some artery or vein or the same length or width of a ductus.

[0068] The twin patient unit is configured to transform images and / or data of the real human patient entered into the twin patient unit to generate the virtual patient. The proctor unit is configured to train the user based on the medical procedure performed by the simulator apparatus, the medical procedure being a standard procedure, whereby the training is adapted to the virtual patient being the twin of the real human patient. The proctor unit is configured to use data of the virtual patient and of data which apply to the virtual patient being stored in the training data storage unit and expert data collection storage unit, whichever applies. This system is highly advantageous in preparing real surgery since obstacles or problems which may occur during the real surgery because of a special variation of the anatomy of the real patient can be considered and the best procedure can be found training with the twin virtual patient prior to the surgery performed on the real patient.The medical training system enables a bidirectional, adaptive training look that decouples real-time interaction via multimodal communication and implements a feed-back driven adaptation of simulated procedures.

[0069] The Al proctor unit preferably undergoes a comprehensive training process. Preferably, the training involves human and / or virtual experts watching videos of in reality performed surgery procedures and commenting the procedures watched. Preferably, the experts stop the videos at different times and comment the scenes they see on the frozen screen. Preferably, they describe what they see and the comment about positive and negative aspects of the scenario seen. This method of obtaining qualified information for training a virtual proctor and an Al proctor unit is herein claimed as a separate invention as well. It can also be implemented within a control unit or proctor being part of a simulator apparatus.

[0070] Further embodiments of the invention are laid down in the dependent claims.

[0071] BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Preferred embodiments of the invention are described in the following with reference to the drawings, which are for the purpose of illustrating the present preferred embodiments of the invention and not for the purpose of limiting the same. In the drawings,

[0073] Figure 1 shows a simulator apparatus of an inventive medical training system;

[0074] Figure 2 shows a schematic presentation of the interactions in between the inventive medical training system and the interaction of the medical training system with a user;

[0075] Figure 3 shows a schematic presentation of the communication in between the inventive medical training system and the communication of the medical training system with a user;

[0076] Figure 4 shows a schematic presentation of parts of the medical training system being relevant for an implementation of a virtual patient being a twin of a real human patient;Figure 5 shows a medical training system simulating a teleoperated medical treating system, also called robotic surgery simulator, in a schematic representation;

[0077] Figure 6 shows a box trainer comprising a portable smart device in a schematic representation with a tangible instrument in use; and

[0078] Figure 7 shows the box trainer of figure 6 with a finger of the user being used as tangible instrument.

[0079] DESCRIPTION OF PREFERRED EMBODIMENTS

[0080] The simulator apparatus 1 shown in figure 1 is a mixed reality scenario medical training apparatus. The basic elements are known in the state of the art. The simulator apparatus comprises a main body with a control unit 13, a display 14, a human anatomy model 15 in real size and tangible instruments, for example at least one hand tool 16 and at least one pedal 17. The hand tool 16 is a medical instrument, for example a surgical instrument. Alternatively, the simulator apparatus 1 is a training apparatus of a teleoperated medical system.

[0081] The human anatomy model 15 is usually representing only a specific part of a human body. In this picture an insufflated abdomen is shown for laparoscopy training.

[0082] The hand tools 16 correspond to real instruments used in laparoscopy. The pedals 17 are used for activation of electro-cauterization elements and other device mechanism controls.

[0083] The tangible instruments 16, 17 have input sensors 160, 170 implemented within, for the acquisition of position, orientation, other degrees of freedom, and relative position to the other instruments and the human anatomy model 15. The human anatomy 15 also contains input sensors 150 within for acquisition of parameters. The input sensors 150, 160, 170 connect to the control unit 13 for data acquisition. Additional sensors may be cameras or other optical sensors arranged on and in the simulator apparatus.

[0084] The hand tools and the pedals as well as the associated sensors and the human anatomy model 15 form a tangible and manually operated simulator assembly.The control unit 13 controls the virtual views displayed on the display 14 based on the activities of the user, i.e. , his use of the tangible instruments 16, 17. The virtual views are generated by the values of the input sensors 150, 160, 170 in tangible instruments 16, 17 and in the human anatomy body 15 that are controlled and manipulated by the user.

[0085] The control unit 13 has also input means for user input, such as his information about his skills and experiences. The input means are not shown in the picture.

[0086] The control unit 13 has means of acquiring input data. The input unit can be a touchscreen or a keyboard of the simulator apparatus. The input unit can also be a receiving unit, receiving wireless data, transmitted for example by a smart device, such as a tablet or a smartphone. The control unit 13 has means of storing data sets for the optimal path and for the data sets. In the example shown, the screen 14 can be a touchscreen or a wireless keyboard or a mouse.

[0087] The control unit 13 of the simulator apparatus 1 provides different exercises to train the skills of the user. The user can use the hand tools 16 to perform the exercise at the human anatomy model 15, whereby he sees on the display 14 a picture of the corresponding virtual human anatomical elements and the hand tool 16 he is presently using in the anatomy model 15. Like this he sees his movements and actions in real time but represented virtually.

[0088] During the exercise the simulator apparatus 1 detects with its sensors 150, 160, 170 the performance of the user and the control unit 13 obtains the performance metrics, such as trajectories and orientations of the devices and virtual anatomical elements, time of the procedure, interactions between the instrumentation and the virtual anatomical models, safety deviations, and procedural deviations.

[0089] Preferably, the data obtained and the performance of the user are stored.

[0090] Figure 2 shows additional parts of the inventive medical training system S cooperating with a user 2. It comprises at least one of the mentioned simulator apparatus 1 , a proctor unit 3, a training data storage unit 4 and preferably a human expert data collection storage unit 5.

[0091] The training data storage unit 4 may be part of the simulator apparatus 1 or it is an external unit, preferably a cloud. In case the system comprises more than one simulator apparatus1, there may be more than one training data storage units 4 assigned to one or more of the simulator apparatus 1 or just a single one assigned to all simulator apparatus 1.

[0092] The training data storage unit 4 comprises at least some of the following data, functions and procedures, all of them herein called data: image data; annotated images; procedure steps; force, position, and motion telemetry of the instrumentation; patient demographics; user demographics.

[0093] The proctor unit 3 is preferably a separate unit or a separate module within the simulator apparatus 1. The proctor unit 3 comprises a virtual proctor which communicates with the simulator apparatus 1 and the user 2. The virtual proctor is a modular Al system (Artificial Intelligence system) combining speech technologies with medical training procedural information. In preferred embodiments, the proctor unit 3 facilitates bi-directional communication across two principal modalities. On one hand, the virtual proctor interacts with the user on a human level, providing verbal or textual guidance and receiving user responses in natural language. On the other hand, the virtual proctor engages in a digital communication channel with the simulator apparatus 1’s control unit, transmitting instructions that dynamically modify or enhance the current training procedure.

[0094] As an optional unit, the medical training system S comprises the at least one human expert data collection storage unit 5. This unit 5 comprises data from a multiple of experts in the at least one medical field to be trained. The data are at least some of the following: experience reports of human experts, scientific papers, medical handbooks, previous runs of the training simulator assigned to expert users.

[0095] As can be seen in figure 3, the data of the training data storage unit 4 can be retrieved from the simulator apparatus 1 and the simulator apparatus 1 can preferably add additional data obtained from performed training sessions. The communication channel uses at least one first API 41 (API = application programming interface), which preferably is a modular interface configured and designed for this application to facilitate interaction between the training data storage unit 4 and the simulator apparatus 1. This first API 41 includes a set of predefined operations for data exchange, processing, and control, enabling specific functionality, e.g., secure data transfer or dynamic configuration. The first API 41 and the others API mentioned in this text are structured using architecture styles common in the practice, e.g., RESTful principle, and supports specific features, e.g., version control, authentication mechanisms, and multi-format data compatibility.The proctor unit 3 is also configured to retrieve and upload data to the training data storage unit 4 by using a second API 43, which is a second modular interface configured and designed for this application to facilitate interaction between the proctor unit 3 and the training data storage unit 4. In some embodiments, this second API 43 includes a different set of predefined operations for data exchange, processing, and control, enabling specific functionality, e.g., secure data transfer or dynamic configuration.

[0096] The proctor unit 3 interacts with the simulator apparatus 1 using a further communication channel, i.e. a third API 31.

[0097] The proctor unit retrieves and adds data to the human expert data collection storage unit 5 by using a further communication channel, i.e. a fourth API 53.

[0098] The user 2 can preferably also upload and retrieve, at least within some limitations, data from the data training data storage unit 4. The communication channel is marked with the reference number 42. It is preferably a web-based architecture, wherein the functionalities are hosted on one or more servers and accessed by users through networked client devices via a web interface and secured with authentication mechanisms.

[0099] The interaction, marked in figure 3 with reference number 21, between the user 2 and the simulator apparatus 1 is as described above. The simulator apparatus 1 provides visual, auditory, and / or haptic interactions, the user 2 performs the task and the apparatus 1 detects and measures and gives feedback, at least on the display, usually also with other means such as with reports and sound, for example when a surface is touched in an uncontrolled way. This interaction is well known in the state of the art and can comprise additional features and elements not described here in detail but nevertheless incorporated in this description as well.

[0100] During use of the system, the control unit of the simulator apparatus 1 will preferably assess the performance of the user 2 during the training session and / or when the training session is completed, and it will adjust the curriculum based on the assessment made. This data may be kept in the storage of the simulator apparatus 1 or it will be uploaded into the training data storage unit 4.

[0101] In addition, the simulator apparatus 1, in more detail its control unit, will update the proctorunit 3 during the training session about the present performance of the user 2, so that the virtual proctor of the proctor unit 3 can communicate with the user 2. The communication channel used for this communication is marked with reference number 32. Preferably, the communication is made orally by using the human language. The proctor unit 3 comprises a microphone and a speaker unit or it uses the according means of the simulator apparatus 1. The user 2 preferably wears a headset having a microphone and speaker unit. The virtual proctor asks questions and assesses the answers given by the user 2. The user 2 can ask for advice or help in a special situation. The virtual proctor may suggest a specific action, may give feedback to present actions and can warn of obstacles or problems to be considered in the next few steps of the procedure. The virtual proctor acts like a supervisor and trainer, wherein the virtual proctor obtains or retrieves the inputs needed for his guidance and assessments from the present performance of the user detected by simulator apparatus 1 , the accumulated know-how present in the training data storage unit 4 and the knowledge stored in the expert data collection storage unit 5.

[0102] At least the communication between the simulator apparatus 1 and the proctor unit 3 is text based, i.e. text prompts are exchanged. At least the proctor unit 3 preferably uses artificial intelligence (Al) to interpret and use the information or execute the instructions or suggestions received from the simulation apparatus 1.

[0103] In the following, two examples are given of training sessions and the communication between the simulator apparatus 1 , T, the user 2, the proctor unit 3, the training data storage 4 and the expert data collection storage unit 5.

[0104] Example 1:

[0105] The simulator 1 in the first example is a mixed reality scenario medical training system comprising a human anatomical model and real medical instruments as tangible tools

[0106] In the first example of a medical training scenario, a surgical trainee uses the simulator apparatus 1 to perform a training scenario, specifically a laparoscopic cholecystectomy in a mixed reality environment.

[0107] The information that the user 2 receives from the simulator apparatus 1 is relevant patient data on the display 14, such as age and sex, symptoms, relevant medical history, and preoperative data and imaging. The preoperative data can include ultrasound, CT, or MRI images. The anatomy model 15 represents an insufflated abdomen with tactile responsethrough the tangible instruments, here the hand tool 16 and the pedal 17.

[0108] The bi-directional communication occurs across the human modalities, like text exchanges, and digital communication between the simulator apparatus 1 and the proctor unit 3 occur over the third API 31. In this example, the simulator sends real-time data from the anatomy model sensors 150, the hand tool sensors 160, and the pedal sensors 170 to the proctor unit. Real-time performance measurements and metrics are also communicated, such as instrument trajectories, forces applied, and procedural steps initiated and completed. The proctor unit 3 sends updates on progress to the simulator apparatus 1, along with instructions to adjust simulation parameters when needed.

[0109] Text exchanges and communication between the user 2 and the proctor unit 3 occur over several media. The virtual proctor of the proctor unit 3 can communicate initial instructions and guidance via the headset with microphone and speaker unit to the user 2, for example, explaining the initial steps of the medical procedure. For example, with a cholecystectomy, this may be text such as “Begin by inserting the first trocar at the umbilical point.” Communication continues during the procedure. Errors are detected using sensors 150, 160, 170 with accompanying text, such as “You placed the trocar in a position that is too lateral.” Additionally, the virtual proctor can add questions and prompts to the user 2, such as, “Why is a too lateral position not recommended?” The procedure can continue after receiving a correct answer from the user 2. Correct techniques are also detected with the sensors 150, 160, 170. Procedure complete is detected with sensors 150, 160, 170 and communicated via the headset to the user 2. Following completion of the medical procedure, final assessments and feedback are provided to the user 2. Performance metrics and goal values are shown on the display 14. Further explanation can be provided via the headset. This feedback can include values like procedure time, the quality of instrument handling, identification of anatomical landmarks, and deviations from safe handling procedures. This text-based feedback can provide qualitative assessments. The text-based feedback can also include recommendations for future training of the scenario, such as “Your adjustment of the trocar was corrected after guidance. Focus on anatomical landmarks to improve structure identification in the future.”

[0110] The data exchanged with the training data storage unit 4 from the simulator apparatus 1 via the second API 41 is the performance data and sensor logs. The data storage unit 4 stores demographics and session metrics for future reference and model training. The human expert data collection storage unit 5 provides best practice data to the proctor unit 3 via thefourth API 53, updating the knowledge base to provide improved feedback and guidance to user 2.

[0111] The order of the activities can be described as follows: session initiation, data transmission, procedure initiation, interactive guidance, procedure completion, assessment, and data storage. Session initiation can include steps such as the user 2 inputting their experience level into the control unit 13, the simulator unit 1 loading case and patient data and showing it on the display 14. Data transmission starts with the control unit 13 initializes sensor communication with the sensors 150, 160, 170; the simulator apparatus 1 sends initial state data to the proctor unit 3 via the third API 31. The user 2 commences training using the instruments, here the hand tools 16 and the pedal 17, while the proctor unit 3 monitors user actions through the real-time data stream 31. The proctor unit 3 communicates with the user 2 via the headset. During the interactive guidance, the user 2 can respond in word and / or action to the guidance from the virtual proctor of the proctor unit 3. The data is sent back to the proctor unit 3. Procedure completion occurs when the user 2 completes or terminates the training scenario; the virtual proctor of the proctor unit 3 provides immediate feedback. The performance data is uploaded to the training data storage unit 4; the proctor unit 3 updates its knowledge database when new data is provided from the fourth API 53.

[0112] Example 2:

[0113] The second example of a medical training scenario uses a simulator T which is simulating a teleoperated medical treating system, especially a robotic surgery system. This simulator apparatus T may simulate the well-known da Vinci® system and / or the well-known MMI Symani® Surgical system. It does not use any anatomical medical instruments. The tangible tools, such as hand held tools 16' and pedals 17, are usually formed differently from medical instruments, but they activate and move virtual medical instruments displayed on the display 14. The simulator apparatus T comprises a control unit 13. Sensors are integrated in the hand held tools 16' and pedals 17 or arranged nearby. Additional sensors may be cameras or other optical sensors arranged on and in the simulator apparatus T.

[0114] In this second example of a medical training scenario, a cardiothoracic surgery fellow uses the simulator apparatus T to perform a training scenario, specifically a robotic-assisted mitral valve repair. This scenario focuses solely on the use of instrumentation and a robotics console.The user 2 receives relevant patient data on the display 14, such as age and sex, symptoms, relevant medical history, and preoperative data and imaging. The preoperative data includes echocardiograms, CT scans, and MRI images detailing the patient's cardiac anatomy and the specifics of the mitral valve pathology. The simulator apparatus provides initial instructions and setup parameters for the robotic surgical system through the control unit 13 and the display 14.

[0115] Figures 2 and 3 apply for this example as well. In the following, reference is made to these figures.

[0116] The bi-directional communication occurs across the human modalities, like text exchanges, and digital communication between the simulator apparatus 1 and the proctor unit 3 occur over the third API 31. In this example, the simulator sends real-time data from the hand tool sensors 160 and the pedal sensors 170 to the proctor unit 3. These sensors 160, 170 capture the user's interactions with a robotic console controls and foot pedals 17, including instrument movements, force feedback, and activation of surgical instruments, such as hand tools 16. Real-time performance measurements and metrics are communicated, such as robotic arm positions, instrument trajectories, forces applied, and procedural steps initiated and completed. The proctor unit 3 sends updates on progress to the simulator apparatus 1, along with instructions to adjust simulation parameters when needed, such as introducing virtual complications or varying the difficulty level based on the user's performance.

[0117] The proctor unit 3 communicates initial instructions and guidance via the headset with a microphone and speaker unit to the user 2, explaining the initial steps of the medical procedure. For example, fora mitral valve repair, the virtual proctor of the proctor unit 3 may say: “Begin by setting up the robotic system and establishing cardiopulmonary bypass.” Communication continues during the procedure. Errors are detected using the sensors 160, 170, with accompanying feedback such as, “Your manipulation of the robotic instruments is too forceful.” The virtual proctor of the proctor unit 3 can add questions and prompts to the user 2, such as, “What adjustments can you make to reduce instrument collision risk?” The procedure continues after the user 2 provides a correct response or adjusts their technique accordingly. Correct techniques are also detected with the sensors 160, 170. Procedure completion is detected through these sensors 160, 170 and communicated via the headset to the user 2.Performance metrics and goal values are shown on the display 14, with further explanations being provided via the headset. This feedback includes values like procedure time, instrument handling quality, adherence to surgical procedures guidelines, and deviations from safe procedures guidelines. The text-based feedback provides qualitative assessments and recommendations for future training, such as, “Your handling of the instruments was precise, but take care to avoid unnecessary movements that could prolong the procedure. Focus on improving your efficiency during the setup phase.”

[0118] The data exchanged with the training data storage unit 4 from the simulator apparatus 1 via the second API 41 includes the performance data and sensor logs. This data encompasses detailed records of instrument movements, user inputs on the robotic console, timing of procedural steps, and any errors or deviations from the expected procedure. The data storage unit 4 stores user demographics and session metrics for future reference, performance tracking, and model training.

[0119] The human expert data collection storage unit 5 provides advanced surgical techniques, best practice guidelines, and expert recommendations to the proctor unit 3 via the fourth API 53. This information updates the proctor's knowledge base to provide improved feedback and guidance to the user 2, ensuring that the training reflects the latest standards in cardiothoracic surgery.

[0120] The order of the activities can be described as follows: session initiation, data transmission, procedure initiation, interactive guidance, procedure completion, assessment, and data storage. T o initiate the session, the user 2 inputs their experience level and previous training details into the control unit 13 using input means. The input means can be part of the control unit 13 or it can be a separate keyboard or a touchscreen. It can even be a receiving unit, which receives wireless inputs from a smartphone or another smart device, such as a tablet. The simulator apparatus 1 loads patient data and displays it on the display 14, including relevant medical history and imaging studies such as echocardiograms, CT scans, and MRI images. The control unit 13 configures the simulator for robotic-assisted surgery, initializing the hand tools 16 as robotic console controls and the pedals 17 for instrument activation. Data transmission starts with the control unit 13 initializing sensor communication with the sensors 160 of the hand tools 17and the sensors 170 of the pedals 17. The simulator apparatus 1 sends initial state data, including the setup configuration and baseline metrics, to the proctor unit 3 via the third API 31. The user 2 commences training by interacting with the robotic console controls, which are usually formed by the hand tools 16, and foot pedals17. The proctor unit 3 monitors user actions in real-time through the data stream over the third API 31. For interactive guidance, the proctor unit 3 communicates with the user 2 via the headset, providing guidance and feedback. Errors detected by the sensors 160, 170 are communicated immediately. For example, the proctor might say, “Warning: Excessive force applied to the instrument could damage delicate cardiac tissues.” The proctor asks questions to assess the user's understanding, such as, “What adjustments can you make to reduce instrument collision risk?” The user 2 responds verbally or by adjusting his actions, which are then detected by the sensors and communicated back to the proctor unit 3. Procedure completion occurs when the user 2 successfully completes the mitral valve repair or decides to terminate the training scenario. The proctor unit 3 provides immediate feedback via the headset and displays comprehensive performance metrics on the display 14. The performance data, including detailed logs of instrument usage and procedural steps, are uploaded to the training data storage unit 4 via the second API 41. The proctor unit 3 updates its knowledge database with new insights and user performance data when new information is retrieved from the human expert data collection storage unit 5 via the fourth API 53.

[0121] Figures 6 and 7 show a simulator apparatus 1" being a Box Trainer. Through-openings leading into an interior of the box are marked with reference number 18. Sensors arranged within or near the through-openings are marked with reference number 180. Tangible task places located in the interior of the box, such are recesses, are marked with reference number 19, corresponding sensors with reference numbers 190. Virtual tasks 19', corresponding to the tangible tasks 19, are displayed on a touchscreen 14'. The touchscreen 14' is present in addition or alternatively to the display described above.

[0122] The user 2 can move the tangible hand tool 16, preferably corresponding in shape, weight and size to a real medical instrument, wherein he also sees a corresponding virtual instrument or tool 16" displayed on the screen 14 and / or on the touchscreen 14'. The tangible tool 16 is connected wirelessly or with a wire 161 with the touchscreen of the tablet shown in figure 6.

[0123] The user 2 can also move with his hand or finger the virtual instrument 16' displayed on the touch screen without having to move the tangible instrument 16. This is shown in figure 7.

[0124] Box Trainers can be used in the same way as described above and below with reference to the other simulator apparatus 1, T. In addition, the other simulator apparatus 1, T mayalso be equipped in addition or alternatively to the display 14 with a touchscreen 14' for moving virtual tools 16' instead of tangible tools 16, 17. The touchscreen 14' can be the screen of a smart device, such as a tablet as shown in figures 6 and 7. The tablet may comprise the control unit 13 as well.

[0125] The Box Trainer or the other systems 1, T mentioned above may be present only virtually, thereby comprising a touchscreen to move the virtual tools displayed, without having other tangible tools or models.

[0126] The control unit 13 of the system may include a web-based application, especially when the control unit 13 is part of a smart device, such as a tablet.

[0127] Training of the proctor unit

[0128] The proctor unit 3 is initially trained by using data of already performed training sessions, by assessing the results of the proctor unit 3 performance and by categorizing the user's performance into expert and non-expert data.

[0129] The Al proctor unit 3 undergoes a comprehensive training process that leverages high-fidelity data collected from prior training sessions on the simulator apparatus 1. This data includes detailed sensor information from the tangible instruments 16 and 17, capturing instrument trajectories, force applications, and procedural sequences. The data is categorized into expert and non-expert datasets, with expert sessions serving as benchmarks for optimal performance.

[0130] Advanced machine learning algorithms are used to process this rich data. These algorithms include, but are not limited to, deep learning neural networks, gesture recognition models, anomaly detection systems, reinforcement learning agents, and natural language processing techniques. In one embodiment, by utilizing, for example, convolutional neural networks (CNNs) and recurrent neural networks (RNNs), the proctor can analyze spatial and temporal patterns in instrument movements and user interactions. For instance, CNNs process visual data from the simulation to identify whether the user's instrument positioning and movements align with expert techniques, such as correctly aligning a suture needle with tissue during a robotic-assisted mitral valve repair. In some embodiments, RNNs and long short-term memory (LSTM) networks analyze sequences of instrument movements to detect patterns indicative of proficiency or potential errors. For example, an LSTM networkcan predict whether the user's next action aligns with expert protocols based on previous movements.

[0131] In an embodiment, gesture recognition algorithms, such as Hidden Markov Models (HMMs), enable the proctor to identify specific surgical gestures performed by the user, comparing them to expert models to assess accuracy and technique. For example, HMMs classify sequences of observable events like surgical gestures, allowing the proctor to determine if the user is performing a knot-tying gesture correctly during a laparoscopic procedure.

[0132] In an embodiment, anomaly detection models, including autoencoders and one-class Support Vector Machines (SVMs), are employed to detect deviations from expert performance, alerting the user to suboptimal techniques or potential safety risks. Autoencoders, by learning normal patterns of expert actions, detect when a user's force application deviates significantly, indicating potential tissue damage. One-class SVMs identify when instrument angles or movements fall outside the learned boundaries of expert performance, prompting the proctor to advise adjustments.

[0133] Reinforcement learning agents are preferably utilized to adapt the training scenarios dynamically, adjusting complexity and introducing challenges based on the user's performance. The proctor unit employs these agents to introduce more complex scenarios if the user performs well or to simplify tasks if the user struggles, ensuring a personalized training trajectory.

[0134] In preferred embodiments, the proctor unit integrates procedural knowledge and contextual understanding by incorporating detailed surgical protocols and anatomical information into its knowledge base. This is achieved, for example, through the development of surgical ontologies and knowledge graphs, which map relationships between anatomical structures, surgical instruments, procedural steps, and potential complications. These structures allow the proctor to understand the context of each action; for example, during a mitral valve repair, the proctor verifies that the user has correctly identified the valve leaflets before proceeding. By referencing these knowledge structures, the virtual proctor provides context-aware guidance, ensuring that feedback and instructions are relevant to the specific stage of the procedure and the user's current actions.

[0135] In some embodiments, the proctor's training includes expert commentary. Expert users provide insights into their decision-making processes during training sessions. Thiscommentary is processed using natural language processing (NLP) techniques, enhancing the virtual proctor's ability to provide explanations and rationales, fostering a deeper understanding in users. The proctor unit employs NLP algorithms to interpret and generate human language, facilitating interactive communication. If a user asks, "Should I proceed to close the incision now?" the proctor unit interprets the question and provides an informed response based on surgical protocols.

[0136] In a preferred embodiment, the proctor unit 3 continues to improve through adaptive learning mechanisms. It analyzes data from ongoing training sessions to refine its machine learning models and personalize guidance for individual users. The proctor unit develops user profiles that capture each user's strengths, weaknesses, and learning preferences, allowing for customized feedback and adaptive scenario adjustments. For instance, if the proctor unit detects that a user consistently applies excessive force with instruments, anomaly detection models identify this pattern, and the virtual proctor provides targeted feedback to adjust the technique.

[0137] In a preferred embodiment, the proctor unit regularly updates its knowledge base by accessing the human expert data collection storage unit 5 via the fourth API 53. This ensures that the guidance provided reflects the most current medical practices and procedural innovations. Updates may include new surgical protocols, advancements in medical techniques, or updated best-practice guidelines, which are integrated into the proctor's knowledge structures.

[0138] Additional features of the proctor unit

[0139] In a preferred embodiment, the proctor unit is configured to provide context-aware guidance within the simulated environment of the simulator apparatus 1. By understanding the procedural flow and integrating comprehensive medical knowledge, the virtual proctor offers accurate and relevant guidance tailored to the user's current actions. The high-fidelity simulator data utilized in training the proctor unit enables it to recognize subtle nuances in user performance, setting it apart from standard Al systems. In some embodiments, the techniques such as multimodal data fusion combine information from visual inputs, haptic feedback, and instrument telemetry, allowing the proctor to assess the user's performance holistically.

[0140] Furthermore, some embodiments of the proctor unit include features such as multilingual support, emotion recognition, and augmented reality integration. Multilingual support allowsthe user from different linguistic backgrounds to interact with the system in his / her preferred language. Emotion recognition enables the proctor to detect emotional cues from the user's voice or interaction patterns, adjusting its guidance to provide encouragement or clarification as needed. Augmented reality integration supplements the proctor's guidance with visual cues displayed through AR devices (AR = augmented reality), such as highlighting anatomical structures or instrument paths directly in the user's field of view.

[0141] Some embodiments include real-time skill assessment in laparoscopic training, where the proctor uses an LSTM network to monitor the user's instrument movements during a laparoscopic cholecystectomy. It recognizes patterns indicating hesitation or uncertainty, such as irregular instrument trajectories or prolonged pauses, and provides encouraging feedback or suggests reviewing specific procedural steps. In adaptive learning during robotic surgery simulation, the proctor employs reinforcement learning to adjust the scenario's complexity based on the user's performance. If the user demonstrates proficiency, the proctor introduces challenges like simulated arrhythmias, enhancing the training's robustness.

[0142] In some embodiments, error prevention through knowledge integration is present, wherein the proctor unit integrates knowledge from surgical ontologies to prevent procedural errors. For instance, during a hysterectomy simulation, the proctor unit ensures the user identifies and safeguards the ureters before proceeding with uterine artery ligation, aligning with best surgical practices.

[0143] Twin patient and twin virtual proctor

[0144] Figure 4 shows an enhanced medical teaching system. This system is especially used before a real surgery is performed on a real patient for getting familiar with the anatomy of the real patient trying to avoid risks and to find the best procedure.

[0145] This additional module or unit of the medical training system enables to generate a virtual patient which is a twin of a real patient. This unit comprises an imaging tool 70 with is fed with pictures and images of the real patient. The images, i.e. the volume objects, are converted by a volume to mesh modification 71 to a geometric mesh 72. The geometric mesh 72 may be optimized in a refinement process. After converting the volumetric images of the real patient into the geometric mesh 72 using the volume to mesh modification unit 71, the next step involves refining this mesh into a simulation mesh 73. This simulation mesh 73 serves as a computational model that accurately represents the patient'sanatomical structures in a format suitable for simulation purposes. A refinement process may include optimizing the simulation mesh 73 for computational efficiency, enhancing mesh quality by adjusting mesh density in areas requiring higher detail, and ensuring that the simulation mesh 73 accurately captures the intricate geometries of the patient's anatomy.

[0146] Following the creation of the simulation mesh 73, the process advances to developing a simulation model 74. This simulation model 74 incorporates not only the geometric data from the simulation mesh 73 but also integrates physical properties such as tissue elasticity, density, and other biomechanical characteristics relevant to the patient's anatomy. The simulation model 74 enables realistic interaction within the simulation environment, allowing the virtual patient to respond to simulated surgical procedures in a manner that closely mimics real-life physiological responses.

[0147] A development phase 75 involves integrating the simulation model 74 into the medical training system and configuring it for interactive use. This includes embedding the simulation model 74 within the simulator apparatus's software architecture, linking it with the control unit 13, and ensuring compatibility with the tangible instruments, like the hand tools 16 and the pedals 17. During the development phase 75, the virtual patient's behaviors are tested and calibrated to ensure that responses to user interactions — such as cutting, suturing, or applying force — are accurate and provide meaningful feedback to the user 2. This phase may also involve validating the simulation model 74 against clinical data to confirm its accuracy and reliability as a training tool.

[0148] A case module 6 is generated by the abovementioned steps and improved with expert data of a patient specific expert data collection storage unit 5'. This storage unit 5' is similar to the already mentioned expert data collection storage unit 5. In addition, it comprises also specific data of the real patient and inputs of the surgeon and / or the medical team being responsible for the patient. The proctor unit 3 uses data of the storage unit 5' and has the information of the case module 6, i.e. of the virtual patient being the twin of a real patient. The virtual proctor of the proctor unit 3 is therefore generated and can act as a twin proctor of a real proctor being familiar with the case of the real patient. The surgeon training prior to the real surgery obtains an optimized support by this virtual proctor, having been generated for this specific case.

[0149] After the development phase 75, the software components of the medical training system —including the virtual patient module and the proctor unit 3 — proceed to a testing stage 10. During this testing stage 10, comprehensive testing ensures that the software functions correctly and meets all specified requirements. For executable applications, this involves unit testing individual components, integration testing to ensure seamless interaction between modules, and system testing to evaluate the software as a whole. Performance testing assesses the application's responsiveness and stability under various conditions. For web-based applications, additional tests such as cross-browser compatibility, security assessments to identify vulnerabilities, and load testing under high user demand are preferably conducted.

[0150] Once the software passes all testing criteria and any identified issues are resolved, it advances to the release stage 11. In this phase, the software is prepared for distribution to end-users or deployment within the medical training system. For executable applications, this includes creating installation packages that bundle all necessary files, libraries, and resources. Version control systems are updated, and release notes are generated to document new features, enhancements, and bug fixes. For web-based applications, the release process involves finalizing the codebase, optimizing assets for performance, and preparing deployment configurations.

[0151] A further step, preferably a final step, is the deploy stage 12, where the software is delivered to its operational environment. For executable applications, deployment may involve installing the software on individual machines or servers within training facilities, using automated deployment tools or manual installation procedures. Deployment scripts can streamline this process by handling configurations and environment-specific settings. For web-based applications, deployment entails uploading the application to web servers or cloud platforms, configuring databases, setting up necessary services, and ensuring that network settings and security certificates are correctly applied. Continuous integration and continuous deployment (CI / CD) pipelines may be utilized to automate and streamline the deployment process, enabling rapid and reliable updates.

[0152] The testing stage 10, release stage 11, and deploy stage 12 ensure the reliability and effectiveness of the medical training system and its components, including the proctor unit 3. By combining rigorous testing with a structured release and deployment process, and by continuously updating the proctor with new medical knowledge and technological advancements, the system maintains a high-quality experience that facilitates effective learning and simulation of medical procedures. This systematic approach ensures that themedical training system operates reliably, providing the user with an adaptive tool that evolves alongside advancements in medical education and practice.

[0153] Preferably, the proctor unit is updated in future releases in order to maintain the medical training system's effectiveness and relevance. Ongoing updates ensure that the proctor remains current with medical advancements and continues to meet users' training needs. This involves several key considerations, including incorporating new medical knowledge, enhancing machine learning models, and implementing user feedback.

[0154] Firstly, the proctor's knowledge base requires regular updates to reflect the latest medical research, surgical techniques, and procedural guidelines. This involves accessing updated information from the human expert data collection storage unit 5 via the fourth API 53. Medical professionals and experts contribute new data, including revised protocols, emerging best practices, and advancements in medical technology. Integrating this new information ensures that the proctor provides guidance that aligns with the most current standards of medical care.

[0155] Secondly, the machine learning models underlying the proctor's functionality are periodically retrained with new data collected from users' training sessions. As more users interact with the system, a wealth of new performance data becomes available. This data refines the proctor's algorithms, improving its ability to recognize patterns, detect errors, and provide personalized feedback. Retraining the models involves processing this new data while ensuring that the models generalize well and do not overfit to specific cases.

[0156] Enhancements to the proctor may also include improvements to its interactive capabilities. This could involve advancing natural language processing algorithms to better understand users' questions and provide more accurate and contextually relevant responses. Enhancements might also include expanding the proctor's multilingual support to accommodate a broader range of users or integrating more sophisticated emotion recognition features to respond appropriately to users' emotional states during training.

[0157] User feedback plays a crucial role in guiding updates to the proctor. Feedback collected from users about their experiences, challenges, and suggestions informs developers about areas needing improvement. This may involve refining the proctor's guidance strategies,adjusting the level of difficulty in training scenarios, or adding new features that enhance the learning experience.

[0158] The process of updating the proctor in future releases follows a similar cycle of development, testing, release, and deployment as the initial software. During the development phase, new features and updates are designed and implemented, ensuring compatibility with existing system components. Rigorous testing is conducted to verify that updates function correctly and do not introduce new issues. This includes unit testing of new modules, integration testing with the simulator apparatus 1 and other system components, and user acceptance testing to gather feedback from a sample of end-users.

[0159] Once testing is complete and the updates meet all quality standards, the new version of the proctor is prepared for release. For executable applications, updated installation packages are created, and versioning is managed to keep track of changes. For web-based applications, codebases are finalized, and deployment scripts are updated to reflect the new configurations. Deployment involves distributing the updated software to users or updating the web-based platform. For systems installed in training facilities, deployment may require coordination with IT staff to schedule updates and minimize disruptions. In cloud-based or web-based systems, updates can be deployed centrally, ensuring that all users have access to the latest version simultaneously.

[0160] To facilitate ongoing updates, the system may employ a modular architecture that allows individual components of the proctor to be updated independently. This approach minimizes the impact on the overall system and allows for more frequent and agile updates. Implementing continuous integration and continuous deployment (CI / CD) pipelines streamlines the process of integrating, testing, and deploying updates, enabling the development team to respond quickly to new requirements and feedback.

[0161] Preferably, security considerations are made throughout this process. Ensuring that updates do not introduce vulnerabilities is critical, especially when the system handles sensitive data such as user performance records or patient information used in virtual patient twins. Secure coding practices, regular security assessments, and adherence to data protection regulations are integral parts of the update process.

[0162] The inventive medical training system trains, due to the virtual proctor, the user efficiently and in a targeted manner.LIST OF REFERENCE SIGNS

[0163] 1, T, 1" simulator apparatus

[0164] 10 testing stage

[0165] 11 release stage

[0166] 12 deploy stage

[0167] 13 control unit

[0168] 14 display

[0169] 14' touchscreen

[0170] 15 human anatomy model

[0171] 150 sensor

[0172] 16, 16' hand tool

[0173] 16" virtual tool

[0174] 160 sensor

[0175] 161 wire

[0176] 17 pedal

[0177] 170 sensor

[0178] 18 through-opening

[0179] 180 sensor

[0180] 19 tangible task

[0181] 19' virtual task

[0182] 190 sensor

[0183] 2 user

[0184] 21 interaction

[0185] 3 proctor unit

[0186] 31 thitrd API

[0187] 32 communication channel

[0188] 4 training data storage unit

[0189] 41 second API

[0190] 42 communication channel

[0191] 43 second API

[0192] 5 human expert data collections storage unit

[0193] 5' patient specific expert data collection storage unit 53 fourth API

[0194] 6 case module

[0195] 70 imaging module71 volume to mesh

[0196] 72 mesh

[0197] 73 simulation mesh

[0198] 74 simulation model 75 development model S medical training system

Claims

33CLAIMS1. A medical training system comprisinga manually operated simulator apparatus configured to perform at least one medical procedure in a simulated medical procedure setup,the simulator apparatus comprisingan input unit enabling a user to enter data and commands into the simulator apparatus,a control unit configured to provide the at least one medical procedure setup and to assess activities during the performance of the at least one medical procedure,at least one manually operated tool, the manually operated tool being at least one ofo a tangible medical instrument,o a tangible tool for teleoperating a surgical instrument, o a manually moveable virtual medical instrument displayed on a touchscreen of the simulator assembly, ando a manually moveable tool for teleoperating a surgical instrument, the last-named tool being displayed on a touchscreen of the simulator assembly,andat least one sensor detecting movements of the tool, characterized in thatthe medical training system further comprises a proctor unit with a virtual proctor acting as an instructor for the user, the virtual proctor being configured- to receive information of the control unit of the simulator apparatus, - to bi-directionally communicate with the user during a medical procedure simulated by the simulator apparatus,- to analyse the user's performance, and- to provide feedback based on the analysis performed.

2. The medical training system of claim 1 wherein the proctor unit is a non-transitory computer readable medium comprising instructions, which, when executed on one or more processors, implement a software program configured- to receive information of the control unit of the simulator apparatus,34- to bi-directionally communicate with the user during a medical procedure simulated by the simulator apparatus,- to analyse the user's performance, and- to provide feedback based on the analysis performed3. The medical training system of claim 2 wherein the instructions are soft-coded.

4. The medical training system of any one of claims 1 to 3 wherein the proctor unit is configuredto bi-directionally communicate with the control unit with the simulator apparatus,wherein the communication between the proctor unit and the controller of the simulator apparatus is digital, the proctor unit receiving real-time data from the simulator apparatus and providing digital instructions to modify or reconfigure an ongoing training session.

5. The medical training system of any one of claims 1 to 4 wherein the proctor unit is a non-transitory computer readable medium being separate from the control unit of the simulator apparatus and being configured to be executed on one or more processors being separate from the control unit of the simulator apparatus.

6. The medical training system of any one of claims 1 to 5 wherein the proctor unit is configured to bi-directionally communicate orally with the user.

7. The medical training system of any one of claims 1 to 6 wherein the proctor unit is configured to enter instructions into the simulator apparatus based on the analysis of the user's performance.

8. The medical training system of any one of claims 1 to 7 wherein the medical training system further comprises a training data storage unit, preferably a cloud, and wherein the simulator apparatus and the proctor unit are configured to independently retract data from and upload data to the training data storage unit.

9. The medical training system of claim 8 wherein the training data storage unit is configured to receive data entered directly into the training data storage unit by the user.

10. The medical training system of any one of claims 1 to 9 wherein the medical training system further comprises an expert data collection storage unit and wherein the proctor unit is configured to retract data from the training data storage unit.

11. The medical training system of any one of claims 1 to 10 wherein the simulator apparatus comprises at least one physical human anatomy model.

12. The medical training system of any one of claims 1 to 11 wherein the medical training system comprises two or more simulator apparatus, each being configured to perform at least one medical procedure, and wherein the proctor unit is configured to perform the medical procedures with the two or more simulator apparatus.

13. The medical training system of any one of claims 1 to 12 wherein the simulator apparatus is configured to use text-based prompts to inform or instruct the proctor unit and the proctor unit is configured to act based on text-based prompts received from the simulator apparatus.

14. The medical training system of any one of claims 1 to 13 wherein the proctor unit is configured to use text-based prompts to inform or instruct the simulator apparatus and the simulator apparatus is configured to act based on text-based prompts received from the proctor unit.

15. The medical training system of any one of claims 1 to 14 in combination with one of claim 8 to 10 wherein the medical training system comprises a twin patient unit for generating a virtual patient being a twin of a real human patient, the twin patient unit being configured to transform images and / or data of the real human patient entered into the twin patient unit to generate the virtual patient, and wherein the proctor unit is configured to train the user based on the medical procedure performed by the simulator apparatus, the medical procedure being a standard procedure, whereby the training is adapted to the virtual patient being the twin of the real human patient, wherein the proctor unit is configured to use data of the virtual patient and of data which apply to the virtual patient being stored in the training data storage unit and expert data collection storage unit, whichever applies.

16. The medical training system of any one of claims 1 to 15 wherein the proctor unit comprises an artificial intelligence module, such as ChatGPT 4o.

Images