Remote expert surgical support platform and application method therefor

Abstract

The present disclosure provides a remote expert surgical support platform and an application method therefor. The remote expert surgical support platform comprises an operating room terminal, an interaction server, and an expert terminal. The operating room terminal comprises: a support request button, an audio / video acquisition device, and an operating room camera. The interaction server is configured to perform audio / video processing and command forwarding required for remote expert support. The expert terminal is configured to perform multi-screen display and remote control required for remote expert support. The operating room terminal interacts with the expert terminal by means of the interaction server. The remote expert surgical support platform and the method of the present disclosure significantly improve the support efficiency of remote experts, increase the support response speed of remote experts, and improve the accuracy of remote guidance.

Description

Remote expert surgical support platform and its application methods

[0001] Cross-reference to related applications

[0002] This disclosure claims the benefit and priority of Chinese patent application filed on December 10, 2024, with application number 202411812601.2 and entitled “Remote Expert Surgery Support Platform and Application Method Thereof”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to medical imaging, and in particular to a remote expert surgical support platform and its application methods. Background Technology

[0004] In modern, highly developed healthcare systems, remote operating room consultations and expert support are receiving increasing attention as key measures to improve the efficiency of medical resource allocation and the level of medical care. However, despite their enormous potential, existing remote consultation systems still face a series of pressing technical challenges in practical applications.

[0005] While some remote operating room consultation and surgical guidance systems exist in the existing technology, these technologies still have the following problems:

[0006] First, traditional telemedicine systems are mainly limited to one-to-one or many-to-many video conferencing modes in their design. This single mode is difficult to adapt to complex and ever-changing medical scenarios, especially the urgent need for an expert to provide simultaneous support to multiple operating rooms at the same time, thus limiting the optimal allocation and efficient utilization of medical resources.

[0007] Secondly, the existing system is not convenient enough in triggering expert support, often relying on a series of cumbersome procedures. This not only increases the workload of medical staff, but may also delay precious treatment time due to the complexity of the procedures, making it difficult for expert support to arrive in a timely manner in emergencies.

[0008] Furthermore, remote experts often face the problem of not being able to effectively control the field of vision in the operating room when providing guidance through the system. Due to the limited field of vision, experts find it difficult to fully and clearly observe the key details of the surgical procedure, which undoubtedly weakens the accuracy and effectiveness of remote guidance and affects the success rate of the surgery.

[0009] Finally, regarding the management of multiple audio and video signals, existing systems are susceptible to crosstalk, which can lead to interruptions, delays, or distortions in the audio and video signals. These issues not only reduce the real-time performance and clarity of remote consultations but may also interfere with the judgment and decision-making of medical staff, thus posing a potential threat to the quality of consultations and patient safety. Summary of the Invention

[0010] To address the shortcomings of existing technologies, this disclosure provides a remote expert surgical support platform and method.

[0011] The first aspect of this disclosure is a remote expert surgical support platform.

[0012] Includes operating room terminals, interactive servers, and expert terminals.

[0013] The operating room terminal includes: a support request button, audio and video capture equipment, and an operating room camera.

[0014] The interactive server is configured to handle the audio and video processing and command forwarding required for remote expert support.

[0015] The expert terminal is configured to provide multi-screen display and remote control capabilities necessary for remote expert support.

[0016] The operating room terminal and the expert terminal interact through an interactive server.

[0017] Preferably, the audio and video acquisition device isolates and transmits multiple audio and video signals.

[0018] Preferably, isolated transmission employs channel isolation and resource isolation.

[0019] Preferably, the interactive server includes a command processing module.

[0020] The command processing module responds to the support request button and performs intelligent scheduling.

[0021] Preferably, the interactive server includes an audio / video processing module.

[0022] The audio and video processing module processes multiple audio and video signals from the audio and video acquisition device.

[0023] Preferably, the interactive server includes an interactive service module.

[0024] The interactive service module is used for audio and video interaction between the operating room terminal and the expert terminal.

[0025] Preferably, the expert terminal includes a remote control module.

[0026] The remote control module is used to remotely control the operating room camera.

[0027] Preferably, the expert terminal includes a multi-screen display module.

[0028] The multi-screen display module is used to switch audio and video in real time through dynamic scheduling.

[0029] The second embodiment of this disclosure is an application method of a remote expert surgical support platform, which is a method for audio and video interaction using a remote expert surgical support platform according to the first aspect of this disclosure.

[0030] As described above, this disclosure provides a remote expert surgical support platform and method, which significantly improves the support efficiency of remote experts, increases the support response speed of remote experts, and enhances the accuracy of remote guidance compared with the prior art. Attached Figure Description

[0031] Figure 1 is a block diagram of the remote expert surgical support platform of the first aspect of this disclosure.

[0032] Figure 2 is a flowchart of remote support in the first aspect of this disclosure.

[0033] Figure 3 is a block diagram of the interface when the multi-screen display module of the first aspect of this disclosure is used for display. Detailed Implementation

[0034] The following specific embodiments illustrate the implementation of this disclosure. Those skilled in the art can understand the advantages and effects of this disclosure from the content disclosed in this specification. This disclosure can be implemented or applied through other different specific embodiments, and the details in this specification can also be modified and changed based on different viewpoints and applications without departing from the concept of this disclosure. Furthermore, it should be stated in advance that the accompanying drawings of this disclosure are for simple illustration only and are not depictions based on actual dimensions. The following embodiments will further describe the relevant technical content of this disclosure in detail, but the disclosed content is not intended to limit the scope of protection of this disclosure. Additionally, the term "or" as used herein may, depending on the actual situation, include any or more combinations of the associated listed items.

[0035] Referring to Figure 1, which is a block diagram of a remote expert surgical support platform according to the first aspect of this disclosure.

[0036] As shown in Figure 1, the remote expert surgical support platform of the first aspect of this disclosure includes an operating room terminal 100, an interactive server 200, and an expert terminal 300. The operating room terminal 100 includes audio and video acquisition equipment, a controllable camera (operating room camera), and a support request button. The interactive server 200 performs core processing such as audio and video processing and command forwarding. The expert terminal 300 performs processing such as multi-screen display and remote control.

[0037] Preferably, the audio and video acquisition device isolates and transmits multiple audio and video signals. Specifically, it employs a multi-channel audio and video signal processing mechanism, namely audio and video signal isolation technology, to ensure efficient and stable signal processing, which includes the following:

[0038] Each operating room was carefully allocated an independent audio and video channel to ensure that the signals between the operating rooms do not interfere with each other, thereby improving the clarity and accuracy of signal processing.

[0039] During signal transmission, a strict signal encapsulation and decapsulation mechanism is implemented to protect the signal and effectively prevent signal leakage and interference.

[0040] To ensure signal quality, a real-time monitoring and signal quality assurance mechanism is also adopted, which can promptly detect and correct problems in signal transmission, providing a strong guarantee for the smooth progress of the operation.

[0041] As a specific isolation method, the first step was to adopt an independent channel allocation mechanism, which allocated an independent signal acquisition and processing channel to each operating room, achieving complete signal isolation and avoiding signal crosstalk between operating rooms. The allocation method is as follows.

[0042] Assign a unique Channel ID to each operating room terminal.

[0043] The Channel ID is composed of: {Hospital Code (8 digits)} - {Operating Room Number (4 digits)} - {Equipment Type (2 digits)} - {Timestamp (6 digits)}.

[0044] Use the Channel ID as the namespace prefix for all audio and video streams in this operating room.

[0045] For example, the Channel ID can be named as follows:

[0046] 1. Hospital code (8 digits): For example, "BJGH30001" represents a hospital in Beijing;

[0047] 2. Operating room number (4 digits): "0101" represents operating room number 1;

[0048] 3. Device type (2 digits): "10" represents surgical field camera;

[0049] 4. Timestamp (6 digits): "240130" represents January 30, 2024.

[0050] The following are some examples of names and their specific meanings.

[0051] BJGH30001-0101-10-240130 / / Surgical field camera in operating room No. 1 of a hospital in Beijing;

[0052] BJGH30001-0101-11-240130 / / Panoramic camera in operating room No. 1 of a hospital in Beijing;

[0053] BJGH30001-0101-13-240130 / / Endoscopic camera in operating room No. 1 of a hospital in Beijing;

[0054] BJGH30001-0101-20-240130 / / Vital signs monitoring device in operating room No. 1 of a hospital in Beijing.

[0055] In addition to independent channel allocation, resource isolation control is implemented. Specifically, sufficient computing resources are independently allocated to each operating room terminal 100 to ensure smooth and efficient signal processing. Independent encoder instances are also set up to perform high-quality encoding and decoding of audio and video signals, improving the stability and clarity of signal transmission. Furthermore, CPU and memory usage limits are implemented to effectively prevent resource abuse and ensure stable system operation.

[0056] In addition, the system manages session status. It groups sessions according to the operating room dimension and records in detail the status of the main video stream, auxiliary video stream, ambient audio stream, control signaling, and expert connection status of each session group, providing comprehensive support for the smooth operation of the surgery.

[0057] In addition, data flow isolation is implemented. Data flow isolation includes traffic partitioning and access control. Specifically, traffic partitioning creates an independent logical partition for each operating room, which contains complete audio stream channels, video stream channels, control signaling channels, and monitoring data channels, achieving complete isolation and efficient transmission of data flows.

[0058] The access control mechanism specifically implements a role-based access control mechanism to ensure clear and unambiguous access permissions for each terminal. Operating room terminal 100 can only access the data stream of its own partition, ensuring the security and privacy of information within the operating room. Expert terminal 300 can access data streams of multiple partitions according to permissions, facilitating remote guidance and assistance from experts.

[0059] To ensure that the operating room can keep track of the progress of support requests in real time, the current status of the support requests is also displayed on the operating room terminal 100.

[0060] Request sent: This indicates that the support request has been successfully sent to the server.

[0061] "Expert Response in Progress" indicates that the system has pushed the request to the expert and is waiting for their response.

[0062] Experts have been connected: This means that experts have successfully connected to the operating room and have begun providing support.

[0063] Support Completed: This indicates that the expert has completed the support task and the system has recorded it.

[0064] To provide operating room staff with a more intuitive understanding of the status, a touchscreen status display and voice status announcement functions are also provided, enabling operating room staff to quickly grasp the progress of support requests even when busy.

[0065] Preferably, the interactive server 200 includes a command processing module, which intelligently schedules responses to support request buttons. Specifically, as shown in Figure 2, the one-click support triggering process is as follows: when the operating room faces an emergency and urgently needs remote professional support, a trigger request is first made. Professionals in the operating room can quickly trigger the support request using a prominent physical button. The button design is intuitive and easy to use, ensuring it can be quickly found and operated even in an emergency. Then, the request is received and verified. Once the button is pressed, the server immediately receives the request and initiates an authentication process. This process ensures that only authorized operating rooms can send valid support requests, avoiding false alarms or malicious operations. Next, a notification is pushed. After verification, the server quickly pushes the support request to the designated expert terminal. Although experts are usually located in different locations, they can all be notified promptly through the system. Then, on the expert terminal, a clear visual cue (such as a flashing icon or red notification) is displayed, and a clear audio prompt is played, ensuring that the expert immediately notices the new support request. To ensure timely expert response, a multi-level notification strategy, such as L1, L2, and L3 levels, is adopted as described below. From in-app text and image prompts to notification sounds, and then to pop-up prompts, the system has been upgraded step by step.

[0066] Level 1: Support requests are displayed in the app with images and text, making them easy for experts to understand at a glance.

[0067] Level 2: If the expert does not respond within T1 time (30 seconds), a prompt tone will be played to remind the expert that there is a new task.

[0068] Level L3: If the expert still does not respond within T2 time (60 seconds), the notification will be further escalated and the expert will be forcibly prompted in the form of a pop-up.

[0069] In addition, intelligent scheduling is implemented, which includes expert status management and priority evaluation.

[0070] Expert status management involves real-time monitoring of each expert's current status, categorized into four states: idle, occupied, busy, and offline. This facilitates more efficient allocation of support tasks.

[0071] Idle state: The expert is currently not on any other tasks and can immediately respond to new support requests.

[0072] Status: Experts are currently assisting other operating rooms and are temporarily unable to accept new tasks.

[0073] Busy status: Experts are currently working on other important tasks and are not accepting new support requests at this time.

[0074] Offline status: The expert cannot be contacted temporarily, possibly due to network issues or other reasons.

[0075] Priority assessment involves automatically evaluating and assigning priority to support requests based on the urgency of the operating room and the availability of specialists. The specific assessment indicator system is as follows.

[0076] Set the surgical complexity to E1.

[0077] E1 = Ws × S + We × E,

[0078] Where S represents the surgical grade score, determined according to the national four-level surgical grading standard.

[0079] Level IV surgery: 5 points;

[0080] Level 3 surgery: 4 points;

[0081] Level 2 surgery: 3 points;

[0082] Level 1 surgery: 2 points

[0083] E represents the urgency score of the surgery; specifically...

[0084] Immediate surgery (≤30 minutes): 5 points;

[0085] Emergency surgery (≤2 hours): 4 points;

[0086] Surgery within a limited time (≤24 hours): 3 points;

[0087] Elective surgery (by appointment): 2 points

[0088] Ws represents the surgical grade weight, Ws = 0.6.

[0089] We represents the urgency weight, We = 0.4.

[0090] E2 represents the waiting time; specifically...

[0091] E2 = Actual waiting time / Standard response time;

[0092] The standard response time is:

[0093] Immediate surgery: 1 minute;

[0094] Emergency surgery: 3 minutes;

[0095] Limited-time surgery: 5 minutes;

[0096] Elective surgery: 10 minutes.

[0097] Let the priority be P, and perform the following calculation:

[0098] P = W1 × E1 + W2 × E2;

[0099] in,

[0100] W1 is the surgical complexity weight, W1 = 0.7.

[0101] W2 is the waiting time weight, W2 = 0.3.

[0102] Finally, the score of P is matched with the corresponding priority. When P≥4.5, it corresponds to priority P0; when 3.5≤P<4.5, it corresponds to priority P1; when 2.5≤P<3.5, it corresponds to priority P2; and when P<2.5, it corresponds to priority P3. The meaning of each priority is as follows.

[0103] P0: Immediately allocate expert resources (highest priority);

[0104] P1: Prioritize the allocation of expert resources;

[0105] P2: Normal allocation of expert resources;

[0106] P3: Allocate expert resources according to plan (lowest priority).

[0107] The following is a specific example. Assuming the conditions are level 4 surgery (S=5) and immediate surgery (E=5) with a 1-minute wait (standard response time of 1 minute), the priority assessment calculation is performed as described above.

[0108] E1 = 0.6 × 5 + 0.4 × 5 = 5.0;

[0109] E2 = 1 / 1 = 1.0;

[0110] P=0.7×5.0+0.3×1.0=3.8;

[0111] The calculation result corresponds to the P1 priority.

[0112] Preferably, the interactive server 200 includes an interactive service module. This module facilitates audio-visual interaction between the operating room terminal and the expert terminal, enabling intelligent audio interaction control. Specifically, it can sensitively detect any screen switching actions on the expert terminal 300. Once a screen switching event is detected, the system responds quickly and automatically constructs a corresponding directional audio channel based on the currently displayed content or scenario requirements. After the directional audio channel is established, it can switch the audio signal in real time to ensure that the sound heard by the participants is completely synchronized with the currently displayed screen content. In addition, it has sophisticated mixing capabilities, enabling intelligent mixing of multiple audio sources according to different scenario requirements to achieve the best auditory effect.

[0113] Preferably, the expert terminal 300 includes a remote control module for remotely controlling the operating room camera. This establishes a complete remote camera control link between the expert terminal 300 and the operating room equipment, ensuring precise guidance and efficient collaboration during remote surgery. Experts on the expert terminal 300 can issue various precise control commands, including camera angle adjustment, focus shift, and zoom operations, through a professional interface to meet the visual requirements at different surgical stages. Upon receiving control commands from the expert terminal 300, the server immediately parses and efficiently forwards the commands to the relevant equipment in the operating room, ensuring accuracy and transmission speed and avoiding potential delays or errors. Upon receiving the control commands forwarded by the server, the camera equipment in the operating room immediately performs corresponding adjustments, such as changing the camera's shooting angle, adjusting the focus, or zooming, to ensure a clear and accurate surgical view. After the camera adjustments are completed, the video feed from the operating room is fed back to the expert terminal 300 in real time for further observation and judgment by the expert, ensuring that the expert can always grasp the latest situation in the operating room and make more accurate decisions.

[0114] Preferably, the expert terminal 300 includes a multi-screen display module, which is used for real-time audio and video switching through dynamic scheduling. Real-time audio and video switching under combined operating room screens ensures that the local switching time is strictly controlled within 50ms. Specifically, signal preloading technology is adopted, and a streaming media buffer pool is designed to improve switching speed. This buffer pool allocates independent buffer space for each video stream, ensuring that the video streams do not interfere with each other. Simultaneously, a preloading strategy is used for potentially switching screens, loading relevant video data into the buffer in advance, thereby significantly reducing the waiting time during switching.

[0115] As shown in Figure 3, under the premise of ensuring the uniqueness of the Channel ID and achieving strict signal isolation, experts at the remote end can stably and clearly view surgical audio and video feeds from multiple operating rooms in different hospitals simultaneously (no more than 24), ensuring the real-time nature and accuracy of medical guidance. Furthermore, to meet the diverse viewing needs of experts, the expert-end system is designed to be extremely flexible, capable of simultaneously outputting surgical footage to multiple different monitors, enhancing the breadth and depth of information display.

[0116] Taking the output of 24 audio and video feeds to two different monitors as an example, the system can support displaying up to 12 sets of real-time feeds from different operating rooms on each monitor. This means that experts can quickly switch between and focus on the status of each operating room within a wide visual range, facilitating timely and accurate judgments. Furthermore, a layout screen function is provided on each monitor, allowing experts to freely select and display detailed supporting footage from two key operating rooms on each monitor, such as high-definition close-ups or replays of crucial surgical steps. Moreover, the focus of the viewing is entirely determined by the expert based on actual needs, greatly enhancing the system's flexibility and practicality.

[0117] In addition, dynamic priority management is implemented, allocating more system resources, such as CPU, memory, and network bandwidth, to active scenes based on their activity level to ensure smooth playback. Inactive scenes, on the other hand, have their processing priority appropriately reduced to conserve system resources. This not only ensures the rational use of system resources but also further improves switching efficiency.

[0118] The operating room screen activity determination mechanism is as follows:

[0119] 1. Interaction Time Rating (It)

[0120] Interactions within 30 minutes are counted as active; interactions exceeding 30 minutes are no longer counted as active.

[0121] It = 1 (Δt ≤ 30 minutes);

[0122] It = 0 (Δt > 30 minutes);

[0123] Where Δt is the time (in minutes) since the last interaction.

[0124] 2. Interaction Type Weight (Iw)

[0125] When the interaction type is support guidance, the weight is 1.0, indicating that an expert is providing remote annotation guidance;

[0126] When the interaction type is voice dialogue, the weight is 0.8, indicating that the expert is communicating with the operating room via voice.

[0127] When the interaction type is "screen adjustment," the weight is 0.6, indicating that the expert is adjusting the camera or screen.

[0128] 3. Activity Calculation

[0129] A = It × Iw;

[0130] Where A is the final activity score (0, 0.6, 0.8, or 1); It is the interaction time score (0 or 1); Iw is the interaction type weight (0.6, 0.8, or 1.0);

[0131] 4. Screen display strategy

[0132] Based on the above calculations, when the activity level A ≥ 0.8, the screen type is a high activity level screen; when 0.6 ≤ A < 0.8, the screen type is a standard activity level screen; and when A = 0, the screen type is a low activity level screen.

[0133] Therefore, the above-mentioned operating room screen activity determination comprehensively considers multiple factors such as screen update frequency, user attention, and surgical progress, and can accurately determine the activity level of each operating room screen, thereby enabling reasonable resource allocation.

[0134] The multi-screen display module also ensures synchronized playback of audio and video signals, avoiding audio-video misalignment issues. Simultaneously, it optimizes the ability to handle concurrent switching of multiple signals, ensuring the continuity of the signal stream remains unaffected during switching, making audio-video switching in multi-operating-room combined screens more stable and reliable.

[0135] The second embodiment of this disclosure is an application method of a remote expert surgical support platform, which is a method for audio and video interaction using the remote expert surgical support platform of the first aspect of this disclosure. The audio and video interaction method corresponds to the remote expert surgical support platform of the first aspect, therefore, various modifications in the first aspect are also applicable to the second embodiment, and will not be described again here.

[0136] The following specific embodiment illustrates the audio and video interaction of this disclosure.

[0137] Neurosurgeons at a top-tier hospital often need to supervise multiple surgeries simultaneously to ensure both quality and efficiency in their busy medical work. Through the audio-visual interaction disclosed herein, the expert can monitor the real-time feeds of three operating rooms simultaneously and clearly using a dedicated expert terminal. This allows the expert to have a comprehensive understanding of the progress in each operating room and be ready to provide necessary guidance at any time. During surgery, if operating room A encounters a complex or emergency situation, the surgeon can quickly send a distress signal to the expert by simply tapping the support button on the screen. Upon receiving the support request from operating room A, the expert terminal will immediately display a prompt. The expert can then quickly switch the current view to the full-screen view of operating room A with a single click for more detailed observation and judgment. After switching to the operating room A view, the expert can use the remote control function to adjust the surgical field of view to ensure clarity and accuracy. Subsequently, the expert will combine their professional knowledge and experience to provide detailed remote guidance to the surgeon, ensuring the smooth progress of the surgery.

[0138] As described above, the audio-visual interaction technology for operating rooms provided in this disclosure significantly improves the support efficiency of remote experts, enables one expert to provide real-time support to multiple operating rooms, simplifies the support triggering mechanism, improves response speed, ensures stable transmission of multiple audio and video signals through signal isolation technology, and enhances the accuracy of remote guidance through remote camera control.

[0139] In summary, this disclosure provides a remote expert surgical support platform and method. Compared with existing technologies, it significantly improves the efficiency of remote expert support, increases the response speed of remote experts, and enhances the accuracy of remote guidance.

[0140] The content disclosed above is only a preferred and feasible embodiment of this disclosure and is not intended to limit the scope of the patent application of this disclosure. Therefore, all equivalent technical changes made using the content of this disclosure and its drawings are included in the scope of the patent application of this disclosure.

Claims

1. A remote expert surgical support platform, characterized in that, Includes operating room terminals, interactive servers, and expert terminals. The operating room terminal includes: a support request button, audio and video capture equipment, and an operating room camera. The interactive server is configured to handle the audio and video processing and command forwarding required for remote expert support. The expert terminal is configured to provide multi-screen display and remote control capabilities necessary for remote expert support. The operating room terminal and the expert terminal interact through an interactive server.

2. The remote expert surgical support platform as described in claim 1, characterized in that, Audio and video acquisition equipment isolates and transmits multiple audio and video signals.

3. The remote expert surgical support platform as described in claim 2, characterized in that, Isolated transmission employs channel isolation and resource isolation.

4. The remote expert surgical support platform as described in claim 1, characterized in that, The interactive server includes a command processing module. The command processing module responds to the support request button and performs intelligent scheduling.

5. The remote expert surgical support platform as described in claim 1, characterized in that, The interactive server includes audio and video processing modules. The audio and video processing module processes multiple audio and video signals from the audio and video acquisition device.

6. The remote expert surgical support platform as described in claim 1, characterized in that, The interactive server includes an interactive service module. The interactive service module is used for audio and video interaction between the operating room terminal and the expert terminal.

7. The remote expert surgical support platform as described in claim 1, characterized in that, The expert terminal includes a remote control module. The remote control module is used to remotely control the operating room camera.

8. The remote expert surgical support platform as described in claim 1, characterized in that, The expert terminal includes a multi-screen display module. The multi-screen display module is used to switch audio and video in real time through dynamic scheduling.

9. A method for applying a remote expert surgical support platform, characterized in that, It is a method for audio and video interaction using the remote expert surgical support platform as described in any one of claims 1 to 8.

Images