Surgical robot system, operating control method for console, product, and device

Abstract

Embodiments of the present application provide a surgical robot system, an operating control method for a console, a product, and a device. The surgical robot system comprises: at least one surgical robot, wherein each surgical robot comprises a first console and a patient-side robot, and the first console is used for controlling the patient-side robot in the surgical robot to perform a surgical procedure; and at least one second console, wherein each second console is connected to at least some surgical robots among the at least one surgical robot, and each second console is used for acquiring surgical procedure data from the connected surgical robots, and on the basis of the surgical procedure data, controlling its own operation and control component to move, so as to provide haptic feedback. By providing haptic feedback in this way, during training, trainee surgeons can experience the realistic resistance and haptic sensations in surgical procedures by means of the haptic feedback provided by the second console, thereby improving the accuracy and effectiveness of training, and achieving a better training effect.

Description

Surgical robot systems, control methods for control consoles, products and equipment Cross-references to related applications

[0001] This application is based on and claims priority to Chinese Patent Application No. 202411864746.7, filed on December 16, 2024, the entire contents of which are hereby incorporated herein by reference. Technical Field

[0002] This application relates to the field of surgical robot system technology, and more specifically, to a surgical robot system, a control method for a control console, a product, and equipment. Background Technology

[0003] Before performing actual surgeries using surgical robot systems, doctors typically undergo training to improve their operational skills. Current training methods include observing surgeries in person, watching recorded surgical videos, and using surgical robot simulators to provide doctors with hands-on experience operating the control console. While these methods can improve surgical skills to some extent, the first two methods only provide visual feedback, and the simulator-based training is limited to individual operation and cannot facilitate mentor training. In other words, using current training methods, doctors struggle to perceive the strength and precision of a mentor's control over the control console during surgery, hindering the development of a feel for operating the surgical robot. This results in low training efficiency and less than ideal training outcomes. Summary of the Invention

[0004] In view of this, this application provides a surgical robot system, a control method for a control console, a product, and an equipment.

[0005] According to a first aspect of this application, a surgical robot system is provided, comprising: at least one surgical robot, each of the at least one surgical robot including a first console and a patient-side robot, the first console being configured to control the patient-side robot in which it is located to perform surgery; at least one second console, each of the at least one second console being connected to at least a portion of the at least one surgical robot; each of the at least one second console being configured to acquire surgical operation data from the connected surgical robot and control the movement of its own control components according to the surgical operation data to provide tactile feedback.

[0006] According to a second aspect of this application, a surgical robot system is provided, comprising: a server; at least one second console, each of the at least one second console being connected to the server, each of the at least one second console being configured to acquire surgical operation data from the server and control the movement of its own control components according to the surgical operation data to provide tactile feedback.

[0007] According to a third aspect of this application, a method for controlling the operation of a console is provided. The method includes: acquiring surgical scene visual feedback data and surgical operation data, wherein the surgical operation data includes motion information of a control component of a first console, the motion information of the control component of the first console being used to control a surgical robot to perform surgical operations indicated by the surgical scene visual feedback data; and during the output of the surgical scene visual feedback data, controlling the movement of a control component of a second console based on the surgical operation data to provide tactile feedback through the second console.

[0008] According to a fourth aspect of this application, an electronic device is provided, the electronic device including a processor, a memory, and a computer program stored in the memory that is executable by the processor, wherein the processor executes the computer program to implement the method mentioned in the first aspect above.

[0009] According to a fifth aspect of this application, a computer program product is provided, the computer program product comprising a computer program that, when executed, implements the method mentioned in the first aspect above.

[0010] By applying the solution provided in this application, the movement of the control components of the second console used by the trainee doctor during training can be controlled based on the motion information of the control components of the first console included in the surgical operation data. This allows the second console to provide the trainee doctor with tactile feedback on the surgical operation, which is indicated by the visual feedback data of the surgical robot performing the surgical scene. By providing tactile feedback in this way, the trainee doctor can experience the real resistance and tactile sensation in the surgical operation through the tactile feedback provided by the second console during the training process. This makes the trainee doctor feel as if they are in a real surgical environment, thereby improving the accuracy and effectiveness of the training and achieving better training results.

[0011] In some embodiments, during the output of surgical scene visual feedback data, the movement of the control components of the second console is controlled based on surgical operation data to provide tactile feedback. This allows the trainee physician, while viewing the output surgical scene visual feedback data, to control the movement of the control components of the second console used during training, based on the movement information of the control components in the surgical scene reflected in the visual feedback data. This combination of visual and tactile feedback allows the trainee physician to not only observe each step of the surgery but also experience the realistic resistance and tactile sensations during the surgical operation through the tactile feedback provided by the second console. This makes the trainee physician feel as if they are in a real surgical environment, thereby improving the accuracy and effectiveness of the training and achieving better training results.

[0012] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 is a schematic diagram of a surgical robot system according to an embodiment of this application.

[0015] Figure 2A is a schematic diagram of an application scenario shown in an embodiment of this application.

[0016] Figure 2B is a schematic diagram illustrating another application scenario in an embodiment of this application.

[0017] Figure 2C is a schematic diagram illustrating another application scenario in an embodiment of this application.

[0018] Figure 3 is a schematic diagram of the structure of dynamic access to the console according to an embodiment of this application.

[0019] Figure 4 is a schematic diagram of various working modes of a console according to an embodiment of this application.

[0020] Figure 5 is a schematic diagram of the structure of dynamic access to the console according to another embodiment of this application.

[0021] Figure 6 is a flowchart illustrating an operation control method for a console according to an embodiment of this application.

[0022] Figure 7 is a schematic diagram illustrating motion compensation of the control components of a second console according to an embodiment of this application.

[0023] Figure 8 is a schematic diagram illustrating one-to-one online training according to an embodiment of this application.

[0024] Figure 9 is a schematic diagram of the logical structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] A surgical robot system is an advanced medical system designed to help doctors perform more precise and safer surgical procedures. Referring to Figure 1, the surgical robot system 100' may include one or more surgeon consoles (SGCs) 101 and one or more patient-side robots 102. In some cases, the surgical robot system 100' may also include one or more vision carts 103. The surgeon console 101 serves as a user input device for the doctor, including a display unit for showing the surgical instrument environment, an operating control mechanism, and armrests. The display unit has an observation window (also called a stereoscopic display) for the doctor to observe; the movements of the operating control mechanism correspond to the movements of the surgical instruments, and the doctor can adjust the position of the surgical instruments by controlling the operating control mechanism; the armrests are for supporting the doctor's arms. In addition, the surgeon console 101 also includes other control switches that are easily touched or pressed by hand or foot for various functional operations and human-machine interaction.

[0027] The patient-side robot 102 may include several robotic arms 1021, which are connected to a frame via several connecting arms. Adjacent connecting arms move relative to each other with specific degrees of freedom, allowing the end effector of the robotic arm 1021 to achieve multiple degrees of freedom (e.g., 7 degrees of freedom, depending on the instrument). An instrument actuator is mounted on the robotic arm 1021, and surgical instruments or image acquisition devices are detachably mounted on the instrument actuators. The image acquisition device may be an endoscope (e.g., a 3D endoscope). For ease of understanding, the following description uses an endoscope as the image acquisition device; however, it is understood that the image acquisition device disclosed herein may also be of other types.

[0028] The visual cart 103 includes a display screen that can display images acquired by the endoscope. The endoscope can acquire images (such as surgical images) and send them to the visual cart 103. The endoscope can send acquired images directly to the visual cart 103 or via a console. The endoscope can perform image processing (e.g., decoding) on ​​the acquired images using its own processor or through a processor. The processed images can also be sent to other image processing devices for further image processing (e.g., noise reduction, contrast enhancement, sharpness improvement, etc.).

[0029] In some application scenarios, such as during remote surgery, the patient-side robot 102 or vision cart 103 in the surgical robot system 100' can establish a communication connection with the doctor's console 101. The doctor's console 101 can interact with the patient-side robot 102 or vision cart 103 through this communication connection. For example, the doctor's console 101 can send control commands to the patient-side robot 102 through the above communication connection to control the pose of the robotic arm 1021 of the patient-side robot 102. For another example, the patient-side robot 102 or vision cart 103 can send its own status information or feedback information during the surgical process to the doctor's console 101 through the above communication connection.

[0030] During surgery, the surgeon can manipulate the control components on the surgeon's console 101 to move the surgical instruments on the patient-side robot 102, thereby performing surgery on the patient. Because precise control of the surgeon's console 101 requires a certain level of skill and expertise, surgeons are usually trained before using the surgical robot system for actual surgery to improve their skills in operating the surgeon's console 101.

[0031] Current surgical training methods mainly include observing surgeries in person, watching recorded surgical videos, and providing hands-on opportunities for doctors using surgical robot simulators. The first two methods allow trainees to understand each surgical step by watching live surgeries and videos. However, because they only provide visual feedback, trainees find it difficult to perceive the strength and precision with which the instructor manipulates the control console during surgery. Training using surgical robot simulators is limited to single-person operation, requiring trainees to complete a series of simulated surgical tasks to practice operating the surgical robot. This method cannot provide instructor-led training.

[0032] In conclusion, current training methods fail to provide trainees with tactile feedback, making it impossible for them to truly experience the intensity and precision of the surgical procedure. The effectiveness of the training needs further improvement.

[0033] Based on this, this application provides a console operation control method. While watching a surgical video stream, the trainee physician can control the movement of the control components of the console used for training (hereinafter referred to as the second console) based on the motion information of the control components of the console executing the surgery in the video stream (hereinafter referred to as the first console). This allows the second console to provide tactile feedback to the trainee physician. By combining visual and tactile feedback, the trainee physician can not only observe each step of the surgery during training but also experience the realistic resistance and tactile sensations during the surgical procedure through the tactile feedback provided by the second console, making the trainee physician feel as if they are in a real surgical environment. For example, during suturing, cutting, and other operations, the control components of the second console will generate precise force feedback in a timely manner, allowing the trainee physician to experience the realistic resistance and tactile sensations during the surgical procedure, thereby improving the accuracy and effectiveness of the training and achieving better training results.

[0034] The console operation control method of this application embodiment can be executed by a second console, a first console communicatively connected to the second console, or a server communicatively connected to the first console. The server can pre-store surgical video streams and surgical operation data, or it can simultaneously connect to the first console to acquire surgical video streams and surgical operation data from the first console in real time. The server can be other consoles, a vision cart, or a system consisting of a vision cart and a patient-side robot. In some scenarios, the second console can pre-store surgical video streams and corresponding surgical operation data. During training, the second console can play the surgical video stream and control the movement of its control components based on the surgical operation data, providing tactile feedback.

[0035] In some scenarios, as shown in Figure 2A, the second console can be directly connected to the first console. After the first console acquires the surgical video stream and surgical operation data, it can control the second console or other display devices to play the surgical video stream and control the movement of the control components of the second console based on the surgical operation data to provide tactile feedback.

[0036] In some scenarios, as shown in Figure 2B, the second console can communicate with a server. The server has pre-stored surgical video streams and surgical operation data. The server can control the second console or other display devices to play the surgical video streams and control the movement of the control components of the second console based on the surgical operation data, providing tactile feedback.

[0037] In some scenarios, as shown in Figure 2C, the first console and the second console can establish a communication connection through a server. The server can obtain surgical operation data and surgical video stream from the first console in real time, and control the second console or other devices with display functions to play the surgical video stream, and control the movement of the control components of the second console based on the surgical operation data to provide tactile feedback.

[0038] The connection between consoles or between a console and a server can be a fixed connection that is bound at the factory or a dynamic connection as shown below. The specific connection can be set according to actual needs, and this application embodiment does not impose any restrictions.

[0039] Please refer to Figure 3, which shows a schematic diagram of the structure of the surgical robot system dynamically integrated with the console. This surgical robot system includes:

[0040] At least one user input device, and a surgical robot device;

[0041] The surgical robot device is capable of establishing a communication connection with any number of target user input devices in response to an access request from at least one user input device, and disconnecting the communication connection with any number of target user input devices that have established a communication connection in response to an exit request from any number of target user input devices that have established a communication connection, so that the number of target user input devices that have established a communication connection with the surgical robot is dynamically variable.

[0042] In other words, the surgical robot device is used to establish a communication connection with the target user input device in response to an access request from the target user input device in at least one user input device, and to disconnect the communication connection with the target user input device that sent the exit request in response to an exit request from the target user input device with which the communication connection has been established. The target user input device may be some or all of the user input devices in at least one user input device.

[0043] At least one user input device is configured to send operation information to the surgical robot device through the communication connection when the user input device is the target user input device and establishes a communication connection with the surgical robot device, so as to cause the surgical robot device to perform the operation indicated by the operation information.

[0044] It should be noted that the above content only uses the example of a user input device sending information to a surgical robot device, and this information is only operational information indicating surgical procedures. In some embodiments, the user input device may also send other information to the surgical robot device, or the surgical robot device may send information to the user input device, etc.

[0045] In some embodiments, the information transmitted between the surgical robot device and the target user input device via a communication connection may include at least one of the following: control instructions for instructing surgical operations; information acquisition instructions for acquiring training materials; training materials; and information instructing surgical operations; wherein the training materials include surgical videos, audio information, and / or haptic feedback information.

[0046] In other words, each of at least one user input device is used to send information to or receive information sent by the surgical robot device through the communication connection when the surgical robot device is established as a target user input device, so that the surgical robot device and / or the target user input device respond to the received information and perform corresponding operations.

[0047] In the above embodiments, the user input device may include a doctor's console 101, a personal computer (PC), or other types of devices. The user input device may be deployed in the doctor's office or in the operating room. The surgical robot device may include at least one of the doctor's console 101, the patient-side robot 102, and the vision cart 103 of the surgical robot system 100' shown in FIG1. ​​The surgical robot device may be deployed in the operating room. For example, in a training scenario, the surgical robot system may include two doctor's consoles 101 of the surgical robot system 100' shown in FIG1. ​​After dynamic access, one doctor's console 101 controls the surgical robot system to perform surgery. After dynamic access, the other doctor's console 101 can obtain the surgical video stream acquired by the image acquisition device set in the surgical robot system, and then conduct surgical skills training based on the surgical video stream.

[0048] In this embodiment, the user input device can dynamically establish or disconnect a communication connection with the surgical robot device as needed; that is, the number of user input devices establishing a communication connection with the surgical robot device is variable. Each user input device can independently send an access request to the surgical robot device. After receiving an access request from a user input device, the surgical robot device can establish a communication connection with that user input device in response to the access request. Each user input device can also independently send a withdrawal request to the surgical robot device. After receiving a withdrawal request from a user input device, the surgical robot device can disconnect the communication connection with that user input device in response to the withdrawal request.

[0049] In this embodiment, the number of user input devices that establish communication connections with the surgical robot can be changed according to actual needs, which improves the flexibility of the surgical robot system and makes the hardware configuration of the surgical robot system more in line with clinical needs.

[0050] In some embodiments, after a user input device disconnects from one surgical robot device, it can establish a communication connection with another surgical robot device. This allows the same user input device to dynamically establish communication connections with different surgical robot devices, rather than each user input device establishing a communication connection with a fixed surgical robot device. In this way, a single user input device can access multiple surgical robot devices; that is, multiple surgical robot devices can share a single user input device, eliminating the need for a separate user input device for each surgical robot device. This improves the reusability of user input devices and thus reduces hardware costs.

[0051] In some embodiments, referring to FIG4, the target user input device may include several operating modes. These operating modes include, but are not limited to, at least one of the following:

[0052] In the surgical mode, the target user input device can control the surgical robot to perform surgical operations or assist in performing surgical operations through a communication connection with the surgical robot device. The choice between performing surgical operations and assisting in performing surgical operations is determined by the type of equipment included in the surgical robot device. In some embodiments, assuming the surgical robot device includes a patient-side robot, the user can operate input components (such as a master hand or foot pedal) on the target user input device. The user's control over the input components can be mapped to the movement of the robotic arm on the patient-side robot within the surgical robot device. After moving the robotic arm to the surgical area, the user controls the movement of the robotic arm and drives the surgical instruments held on the robotic arm to perform the surgical operation.

[0053] In the teaching mode, the target user input device can acquire demonstration videos from the surgical robot via a communication connection. These videos can then be played on the target user input device's display interface. The demonstration videos can be videos of the surgical robot performing surgical procedures; they can be real-time captured videos or pre-stored videos. The demonstration videos may include actual surgical instruments held by the surgical robot. Furthermore, the target user input device can acquire the user's control inputs on the input components. Based on these control inputs, a simulator is rendered on the display interface to simulate the surgical instruments held by the surgical robot. Users can adjust the control inputs based on the pose differences between the simulator and the actual surgical instruments in the demonstration video, thereby providing guidance and instruction for surgical procedures.

[0054] In the training mode, the target user input device can obtain training materials from the surgical robot through a communication connection with the surgical robot and play the training materials on the display interface of the target user input device for the user to view.

[0055] Test drive mode: In this mode, the target user input device can establish a communication connection with the new model of surgical robot and control the new model of surgical robot through the communication connection to remotely experience the new model of surgical robot.

[0056] The target operating mode can be selected from multiple candidate operating modes by the operator of the target user input device. For example, multiple candidate operating modes can be displayed on the display interface of the target user input device, and the user can select the target operating mode from the multiple candidate operating modes by clicking, voice input, gesture input, or operating the input components of the target user input device.

[0057] Different operating modes can be matched with different lists of available resources. For example, the list of available resources matched with the surgical mode includes a list of connectable robotic arms; the list of available resources matched with the teaching mode includes a list of viewable demonstration videos; the list of available resources matched with the training mode includes a list of training materials; and the list of available resources matched with the test drive mode includes a list of new surgical robot models available for hands-on experience. The surgical robot can obtain the target operating mode from the target user input device and return a list of available resources matching the target operating mode to the target user input device.

[0058] Referring to Figure 5, a schematic diagram of the structure of the surgical robot system dynamically integrated with the console is shown. The surgical robot system includes:

[0059] At least one user input device, a server, and at least one surgical robot device;

[0060] At least one user input device can send a connection pairing request to the server.

[0061] The server is configured to respond to receiving a target connection pairing request sent by a target user input device, determine the target surgical robot device to which the target connection pairing request is directed from at least one surgical robot device, and establish a communication connection between the target user input device and the target surgical robot device, wherein the target user input device is any one of the at least one user input devices.

[0062] Unlike the previous embodiments, where the communication connection between the user input device and the surgical robot was managed by the surgical robot itself, in this embodiment, since the number of user input devices and surgical robots in the surgical robot system 100 are dynamically variable, each user input device has the potential to establish a communication connection with each surgical robot in the surgical robot system 100. Therefore, the communication connection between each user input device and each surgical robot is uniformly managed by the server. Each user input device can connect to the server via wired or wireless means, and similarly, each surgical robot can also connect to the server via wired or wireless means. The functions of the server and the surgical robot in the previous embodiments overlap to some extent. The following mainly describes the differences between the two; similar or identical functions can be referred to interchangeably and will not be repeated here.

[0063] In some embodiments, any target user input device in the surgical robot system 100 can send an access request to the server. The access request may include identification information of the target user input device and identification information of the target surgical robot device to which the target user input device wishes to connect. The server may pre-establish a first correspondence between the identification information of each user input device and the corresponding connection information of the user input device, and pre-establish a second correspondence between the identification information of each surgical robot device and the corresponding connection information of the surgical robot device. Based on the identification information of the target user input device included in the access request and the first correspondence, the server determines the connection information of the target user input device, based on the identification information of the target surgical robot device included in the access request and the second correspondence, and establishes a communication connection between the target user input device and the target surgical robot device based on the connection information of the target user input device and the connection information of the target surgical robot device.

[0064] In some embodiments, the server may also return a list of available resources to the target user input device in response to a resource request sent by the target user input device. The target user input device can select a target resource from the list of available resources, and the server can determine the target surgical robot device based on the target resource.

[0065] As one implementation, after powering on, the surgical robot can report its resources to the server. The server stores the resources reported by each surgical robot in an available resource list and establishes a third-party correspondence between the resources and the surgical robot providing them. Upon receiving a resource request from the target user input device, the server can return the available resource list to the target user input device. After the target user input device selects a target resource, the server can determine the surgical robot providing the target resource based on the target resource and the aforementioned third-party correspondence, designate that surgical robot as the target surgical robot, and establish a communication connection between the target user input device and the target surgical robot.

[0066] In some embodiments, the server may also grant operation permissions to the target user input device, thereby enabling the target user input device to operate on the target resource, such as watching software resources like video files or controlling hardware resources like robotic arms. For details on the target resource and its operation methods, please refer to the foregoing embodiments.

[0067] In some embodiments, the server may obtain the target operating mode of the target user input device and return a list of available resources matching the target operating mode to the target user input device. The target operating mode may be selected by the operator of the target user input device from multiple candidate operating modes.

[0068] In some embodiments, the server may determine the list of available resources based on the target user's input device operation permissions and the connection status of each device in the surgical robot system 100.

[0069] The operation permissions of the target user input device can be determined based on at least one of the following: the operating status information of the target user input device; the user login information on the target user input device; and the role type of the target user input device, which includes master device type and slave device type.

[0070] In some embodiments, the server may also update the available resource list in response to the following conditions: at least one surgical robot device goes offline; or at least one surgical robot device other than the one mentioned goes online. For example, after a surgical robot device goes online, it can send a heartbeat signal to the server at a certain frequency. If the server receives a heartbeat signal from a surgical robot device, it can determine that the surgical robot device is online. If the server does not receive a heartbeat signal from a surgical robot device, it can determine that the surgical robot device is offline. Alternatively, a surgical robot device can send an online message to the server after going online to inform the server that it is online, and a surgical robot device can send an offline message to the server before going offline to inform the server that it is about to go offline. After determining that a surgical robot device is offline, the server can remove the resources provided by that surgical robot device from the available resource list; after determining that a surgical robot device is online, it can add the resources provided by that surgical robot device to the available resource list.

[0071] In some embodiments, the server may also verify the connection pairing request. If the verification passes, a communication connection is established between the target user input device and the target surgical robot device. The verification may include, but is not limited to, at least one of the following: verifying the access permissions of the target user input device; verifying the availability of the target resource requested by the target user input device; and verifying the connection status of the target user input device and the target surgical robot device. The verification method for the connection pairing request is similar to the verification method for the access request described above, and will not be repeated here.

[0072] In some embodiments, the server may also receive an exit request sent by the target user input device and disconnect the communication connection between the target user input device and the target surgical robot device in response to the exit request.

[0073] In some embodiments, the server may, in response to an exit request, determine the role type of the target user input device, and if the role type of the target user input device is a slave device, disconnect the communication connection between the target user input device and the target surgical robot device.

[0074] If the target user input device's role type is primary device type, the server may refuse to disconnect the communication connection between the target user input device and the target surgical robot device, or send a prompt message to the target user input device to confirm whether to disconnect the communication connection with the target surgical robot device. If the server receives a confirmation instruction from the target user input device, it will disconnect the communication connection between the target user input device and the target surgical robot device.

[0075] In some embodiments, the communication connection between the target user input device and the target surgical robot device is used to send control instructions sent by the target user input device to the target surgical robot device, the control instructions being used to control the target surgical robot device to perform surgical operations; and / or to send surgical videos and / or haptic feedback information from the target surgical robot device to the target user input device.

[0076] In some embodiments, the target user input device's role type includes master device type and slave device type. When the target user input device's role type is master device type, it can send control commands to the target surgical robot device via a communication connection to control the target surgical robot device to perform surgical operations or to provide assistance in performing surgical operations. When the target user input device's role type is slave device type, it can acquire surgical video and / or haptic feedback information transmitted by the target surgical robot device via a communication connection.

[0077] In some embodiments, a target user input device with a role type of primary device type can establish a communication connection with a target surgical robot device in an idle state. The server is further configured to: if the target user input device's role type is primary device type, update the state of the target surgical robot device to occupied state after establishing a communication connection between the target user input device and the target surgical robot device. Since the primary device type target user input device has surgical permissions, updating the state of the target surgical robot device to occupied state after establishing a communication connection with the primary device type target user input device prevents user input devices with the same role type as primary device type from establishing a communication connection with the target surgical robot device, thus avoiding conflicts during surgery.

[0078] In some embodiments, the server is further configured to: if the communication connection between the target user input device and the target surgical robot device is disconnected when the state of the target surgical robot device is updated to an occupied state, update the state of the target surgical robot device to an idle state. After the communication connection between the target user input device and the target surgical robot device is disconnected, it indicates that the target user input device will no longer perform surgical operations on the target surgical robot device. Therefore, disconnecting the communication connection between the target surgical robot device and the target user input device and updating the state of the target surgical robot device to an idle state allows the target surgical robot device to be connected to and controlled by other user input devices of the same role type (also the main device type), and to perform surgical operations under the control of these other user input devices.

[0079] In some embodiments, the server is further configured to: in response to receiving a connection switching request sent by the target user input device, disconnect the communication connection between the target user input device and the target surgical robot device, re-determine the target surgical robot device, and establish a communication connection between the target user input device and the re-determined target surgical robot device. The connection switching request may include identification information of another surgical robot device besides the target surgical robot device, and the server can re-determine the target surgical robot device based on the identification information of the surgical robot device included in the connection switching request. In some embodiments, the server may determine the stage of the surgery based on video of the target surgical robot device performing surgical operations or providing assistance for the surgery. If the stage of the surgery at the time of receiving the switching request is the end stage of the surgery, the server may disconnect the communication connection between the target user input device and the target surgical robot device, re-determine the target surgical robot device, and establish a communication connection between the target user input device and the re-determined target surgical robot device. If the stage of the surgery at the time of receiving the switching request is the ongoing stage of the surgery, the server may refuse to disconnect the communication connection between the target user input device and the target surgical robot device.

[0080] In some embodiments, the server is further configured to: after establishing a communication connection between the target user input device and the target surgical robot device, in response to receiving a connection pairing request for the target surgical robot device sent by at least one other user input device, determine the role type of the other user input device, and if the role type of the other user input device is a slave device, establish a communication connection between the other user input device and the target surgical robot device.

[0081] In some embodiments, the server is further configured to: refuse to establish a communication connection between the other user input device and the target surgical robot device if the role type of the other user input device is the primary device type and the role type of the target user input device is the primary device type, or send a connection disconnection request to the target user input device, the connection disconnection request being used to request the disconnection of the communication connection between the target user input device and the target surgical robot device; if a confirmation indication is received from the target user input device in response to the connection disconnection request, disconnect the communication connection between the target user input device and the target surgical robot device, and establish a communication connection between the other user input device and the target surgical robot device.

[0082] In some embodiments, the server is further configured to: refuse to establish a communication connection between other user input devices and the target surgical robot device if no confirmation indication is received from the target user input device in response to the connection disconnection request.

[0083] In some embodiments, the server is further configured to: establish a communication connection between the other user input device and the target surgical robot device if the role type of the other user input device is master device and the role type of the target user input device is slave device.

[0084] In some embodiments, the target user input device includes at least one user input device with a role type of master device and / or at least one user input device with a role type of slave device.

[0085] In some embodiments, the target user input device includes multiple user input devices with a main device type and ...

[0086] In some embodiments, the target user input device includes multiple user input devices with a role type as the main device type; each of these multiple user input devices with a role type as the main device type establishes a communication connection with the target surgical robot device; wherein, different user input devices among the multiple user input devices with a role type as the main device type are used to control different parts of the target surgical robot device through their own communication connections with the target surgical robot device. For example, when the target surgical robot device includes a device with multiple robotic arms, different user input devices among the multiple user input devices with a role type as the main device type can control different robotic arms. As another example, when the target surgical robot device includes multiple devices, different user input devices among the multiple user input devices with a role type as the main device type can control different devices within the target surgical robot device.

[0087] In some embodiments, the target surgical robot device includes at least one patient-side robot, the number of robotic arms included in each of the at least one patient-side robot being related to the configuration of the patient-side robot; the target user input device includes at least one input component, and each input component included in the target user input device is capable of being paired with each robotic arm included in the corresponding patient-side robot based on the configuration of the patient-side robot in the target surgical robot device.

[0088] In some embodiments, the target user input device includes a doctor's console, and the target surgical robot device includes a doctor's console, a patient-side robot, and / or a vision cart.

[0089] In some scenarios, the console operation control method provided in this application embodiment can be implemented based on the scheme mentioned in the above embodiments that allows for the dynamic addition or removal of devices in the surgical robot system 100. For example, two user input devices can be used in the surgical robot system, both of which are consoles. One console is operated by the trainee physician, and the other console is operated by the instructor. The console operated by the instructor is called the first console, and the console operated by the trainee physician is called the second console. In this case, the surgical robot system can be a system composed of a vision cart and a patient-side robot. The server mentioned above can be a surgical robot system composed of a vision cart, a patient-side robot, or a combination of a vision cart and a patient-side robot. As shown in Figure 6, the console operation control method may include the following steps:

[0090] S61. Acquire surgical video stream and surgical operation data, wherein the surgical operation data includes motion information of the control components of the first console, and the first console is used to control the surgical robot system to perform the surgery in the surgical video stream;

[0091] During surgery, the surgical robot system controlled by the first console can capture real-time images of the surgical scene, resulting in a surgical video stream. Simultaneously, to allow trainees to view the surgical video stream and understand the instructor's control of the first console during surgery, motion information of the control components under the instructor's control can be collected, yielding surgical operation data. This surgical video stream and surgical operation data can be stored together and used as training data for training physicians.

[0092] The control components of the first console can be various input components in the first console for users to perform interactive operations to control the surgical robot system. For example, they can be the main hand, foot pedal, or other types of switches or buttons. This application embodiment does not impose any restrictions.

[0093] The motion information of the control components of the first console can be various types of information reflecting the movement of the control components during surgery. For example, the motion information can be the motion information of some or all joints of the control components. Of course, considering that the movement of the control components of the first console corresponds to the movement of the surgical instruments in the surgical robot system, in some scenarios, the motion information can also be the movement information of the surgical instruments.

[0094] In some embodiments, the motion information may be one or more types of information such as the position, spatial orientation, motion speed, motion acceleration, and rotation angle of the control component.

[0095] In step S61, during the training of the physician, the aforementioned surgical video stream and surgical operation data can be acquired. The surgical video stream can be a two-dimensional or a three-dimensional video stream, depending on the specific circumstances.

[0096] S62. While playing the surgical video stream, control the movement of the control components of the second console based on the surgical operation data to provide tactile feedback through the second console.

[0097] In step S62, after acquiring the surgical video stream and surgical operation data, the surgical video stream can be played so that the trainee doctor can view the surgical scene during the operation. Simultaneously, to allow the trainee doctor to intuitively experience the surgeon's manipulation of the first control console during the operation, the movement of the control components of the second control console used by the trainee doctor can be controlled based on the surgical operation data during the playback of the surgical video stream, providing tactile feedback to the trainee doctor through the second control console.

[0098] The surgical video stream can be played through the second console or through other devices with display functions besides the second console; this application embodiment does not impose any restrictions.

[0099] For example, in some scenarios, the second console includes a display device, through which the surgical video stream can be displayed. This allows the trainee to view the surgical video stream through the second console and place their hands on the control components of the second console to feel the movement of these components, thus gaining a direct understanding of how the instructor controls the control components of the first console during the surgery.

[0100] In some scenarios, the surgical video stream can also be played through other display devices besides the second console. For example, in some scenarios, during training, instructors can use a large tablet to display teaching materials, and therefore, the surgical video stream can also be played through the tablet so that trainees can watch the surgical video stream on the tablet.

[0101] The first console and the second console can be the same console or different consoles; this application embodiment does not impose any restrictions.

[0102] Through this training method, trainees can watch surgical video streams during training, and simultaneously receive tactile feedback from the control components of the second console, allowing them to experience the interaction between surgical instruments and tissues in real time. For example, during suturing, the control components of the second console accurately reproduce the operational postures and force changes shown in the video, providing trainees with nuanced tactile feedback. This real-time tactile feedback accurately reproduces subtle force changes during surgical procedures, enabling trainees to gradually adapt to the intensity and precision requirements of actual surgery during training.

[0103] In some embodiments, the surgical video stream may be a video stream of a real surgery, and the surgical robot system may include a surgical robot device that moves with the first console, such as a patient-side robot, and the surgical video stream may be acquired by an endoscope in the surgical robot.

[0104] In some embodiments, the surgical video stream may also be a video stream of a virtual surgery, and the surgical robot system may include a surgical simulator, wherein the surgical video stream may be a virtual surgical scene displayed in the surgical simulator.

[0105] In some embodiments, the surgical video stream can be a live video stream. For example, during the operation of a surgical robot system controlled by a first console, the surgical video stream and surgical operation data of the first console can be collected in real time, and the real-time collected surgical video stream can be forwarded to a second console or other device with display function to play the surgical video stream, and the movement of the control components of the second console can be controlled based on the real-time collected surgical operation data to provide tactile feedback.

[0106] In some embodiments, the surgical video can also be a pre-recorded video stream. For example, the instructor can pre-record a surgical video stream for surgical training, along with corresponding surgical operation data, which the trainee can then download. In some scenarios, the surgical video stream and surgical operation data are available for paid download, meaning the trainee can pay to access the pre-recorded training data.

[0107] In some embodiments, to allow trained physicians to simultaneously understand the manipulation status of the control components of the first console corresponding to each surgical scene in the surgical video stream while viewing each surgical scene, the surgical video stream and surgical operation data can be synchronized in time. For example, for each video frame in the surgical video stream, motion information of the control components of the first console corresponding to the acquisition time of that video frame can be recorded; that is, each video frame can correspond to a set of motion information. Each video frame and its corresponding set of motion information can then be associated and stored, thereby obtaining the motion status of the control components of the first console at different video acquisition times. During the playback of the surgical video stream through the display device of the second console, a set of target motion information corresponding to the currently played video frame can be determined based on the time synchronization between the surgical video stream and the surgical operation data, and the movement of the control components of the second console can be controlled according to this target motion information.

[0108] By synchronizing the surgical video stream and surgical operation data in time, and controlling the movement of the control components of the second console based on the time synchronization of the two, trainees can not only watch each step of the surgery through the surgical screen during the training process, but also simultaneously feel the corresponding tactile information, as if they were in a real surgical environment, providing trainees with an immersive surgical training experience.

[0109] Considering that the control components of the second console can move autonomously based on the motion information of the control components of the first console, providing tactile feedback to the trainees, in some scenarios, to allow trainees to perceive the instructor's control force and precision on the control components of the first console throughout the training process, and to prevent trainees' misoperation of the control components of the second console from affecting the tactile feedback effect, trainees should be prohibited from operating the control components of the second console themselves during training. That is, trainees only need to place their hands or feet on the control components of the second console to feel their movement. In other scenarios, to allow trainees to personally operate the second console during training and feel the difference in force and precision between their own operation and the instructor's operation, trainees should be allowed to operate the control components of the second console.

[0110] To meet the needs of the different scenarios described above, in some embodiments, the second console can be configured with two operating modes: a first mode and a second mode. In the first mode, the control components of the second console do not respond to user control operations, i.e., user operation of the second console is prohibited. In the second mode, the control components of the second console respond to user control operations, i.e., user operation of the second console is allowed during training.

[0111] In some embodiments, during the playback of the surgical video stream, the control components of the second console can be directly controlled to move according to the motion information of the control components of the first console included in the surgical operation data. For example, assuming that during the acquisition of video frames 1-N, the main hand of the first console moves from position A to position B, then during the playback of video frames 1-N, the main hand of the second console can also be controlled to move from position A to position B. By controlling the movement of the control components of the second console based on the motion information of the first console, the control components of the second console can reproduce the movement of the control components of the first console during the surgery. The control components of the second console will reproduce the operation details and tactile sensations corresponding to the surgical video stream one-to-one, allowing the trained physician to truly experience the interaction between surgical instruments and tissues during the surgery.

[0112] In some embodiments, when the control component of the second console moves according to the motion information of the control component of the first console, the control component of the second console and the control component of the first console can be the same type of control component or different control components. For example, if the motion information is the motion information of the master hand of the first console, the motion information of the master hand of the first console can be directly used as the motion information of the master hand of the second console, realizing a 1:1 motion mapping from the motion of the master hand of the first console to the motion of the master hand of the second console. When the motion information is the motion information of the slave hand of the first console, the motion information of the master hand of the second console can be calculated using the inverse kinematics principle, which can also achieve a 1:1 motion mapping from the motion of the master hand of the first console to the motion of the master hand of the second console.

[0113] In some embodiments, if the second console is detected to be in the first mode or the second mode, and no user control operation on the second console is detected, the control component of the second console can be directly controlled to move according to the motion information of the control component of the first console, so that the motion states of the two are consistent.

[0114] In some embodiments, to provide trainees with practical experience and allow them to perceive the differences between their own manipulation actions and those of their instructors, trainees may also manipulate the control components of a second console during training. In this scenario, when controlling the movement of the control components of the second console based on surgical operation data, motion information of the control components under the trainee's control can be acquired first. This motion information is the motion information of the control components of the second console under the trainee's control. Then, the movement of the control components of the second console can be compensated based on the difference between the motion information of the control components of the second console and that of the control components of the first console. By comparing the differences between the motion information of the control components of the second console and those of the control components of the first console, motion compensation can be performed on the control components of the second console in real time, ensuring that the motion state of the control components of the second console remains consistent with that of the control components of the first console. This tactile feedback mechanism based on motion compensation enables trainees to quickly identify and correct errors between their own operations and standard operations while manipulating the second console, thereby improving the trainee's operational accuracy.

[0115] In some embodiments, if the second console is detected to be in the second mode described above, and a user's control operation on the second console is also detected, the motion information of the second console under the user's control operation can be obtained first, and then the difference between the motion information and the motion information of the control component of the first console can be compared, and then motion compensation can be performed on the control component of the second console based on the difference.

[0116] In some embodiments, when trainees are allowed to watch a surgical video stream while personally operating the control components of a second console, a virtual surgical tool can be overlaid on the surgical video stream while the video stream is played through the display device of the second console. This virtual surgical tool moves based on the movement of the control components of the second console. By overlaying a virtual surgical tool that moves in sync with the movement of the control components of the second console onto the surgical video stream, a realistic surgical scenario can be simulated, providing trainees with hands-on experience.

[0117] In some embodiments, where trainees are allowed to operate a second console during training, operational skills can be assessed based on the differences between the motion information of the control components on the second console and those on the first console. This assessment evaluates the trainee's console operation skills. By evaluating the trainee's operational skills based on these differences, the trainee can understand their own operational level and any shortcomings in their operation, which helps them improve their skills.

[0118] In some embodiments, when compensating for the movement of the control component of the second console based on the difference between the movement information of the control component of the second console and the movement information of the control component of the first console, the current position of the control component of the second console can be determined based on the movement information of the control component of the second console. Then, the target position of the control component of the second console can be determined based on the movement information of the control component of the first console, and the control component of the second console can be controlled to move from the current position to the target position. For example, as shown in Figure 7, suppose that during the acquisition of the first to Nth frames of video, the main hand of the first console moves from position A to position B. During the playback of the first to Nth frames of video, the main hand of the second console moves from position A to B' under the control of the trainee doctor. At this time, the main hand of the second console can be automatically controlled to move from position B' to position B. At this time, the trainee doctor can feel the autonomous movement of the main hand, and thus can truly perceive the deviation between his own operation and the standard operation.

[0119] During motion compensation based on differences in the second console, to prevent erroneous manipulation of the control components of the second console from interfering with the motion compensation and causing the control components to fail to provide accurate tactile feedback, in some embodiments, when controlling the control components of the second console to move from the current position to the target position, the movement of the control components of the second console in directions other than a specified direction can be restricted. This specified direction is the direction from the current position to the target position. For example, taking the above example, when controlling the main hand of the second console to move from position B to position B' for motion compensation, the movement of the main hand of the second console in directions other than the direction from position B to position B' can be restricted.

[0120] When using a console to control a surgical robot system, it's important to consider that surgeons may have different preferred control modes for different types of surgeries, or even for the same type of surgery, different surgeons may have different preferences. For example, some surgeons prefer highly sensitive and low-resistance control components on the console, meaning they can control large movements with minimal effort, while others prefer lower sensitivity and higher resistance, requiring greater force to achieve similar movements. To meet the needs of the same surgeon in different surgical scenarios or to accommodate different surgeons' preferences, the console typically includes multiple control modes, with varying sensitivity and resistance of the control components in each mode. In some embodiments, to allow trainees to experience the different control modes during training, if the surgical video stream is a pre-recorded stream, the trainee can adjust the movement speed of the control components on the second console by adjusting the playback speed of the pre-recorded video stream. For example, if the trainee physician prefers lower resistance and higher sensitivity in the control components of the second console, they can set the playback speed of the surgical video stream to be higher, thus increasing the movement speed of the control components. Conversely, they can set the playback speed of the surgical video stream to be lower, resulting in slower movement of the control components. In this way, the second console can provide a suitable operating experience based on the trainee physician's preferences when offering haptic feedback.

[0121] When training trainees, instructors can provide training through real surgeries or simulated surgeries, and can also implement a one-on-one remote online training mode. For example, in some embodiments, as shown in Figure 8, instructors can train trainees through real surgeries. The surgical video stream is a live video stream, and the surgical robot system includes a surgical robot device that moves with a first console. During the real surgery, the instructor can use the endoscope of the surgical robot device to collect the surgical video stream in real time, as well as the surgical operation data from the first console, and forward it to a second console in real time via the network. The second console can play the surgical video stream and control the movement of its control components based on the surgical operation data.

[0122] In some embodiments, an instructor can train trainees through simulated surgery, where the surgical video stream is a live video stream and the surgical robot system is a surgical simulator. The instructor can simulate surgery in a first console using the surgical robot simulator and forward the video data from the surgical simulator and the information from the control components of the first console to a second console in real time. The second console can play the surgical video stream and simultaneously control the movement of the control components of the second console based on the surgical operation data.

[0123] This one-on-one remote online training model allows trainees to perceive the strength and precision of the instructor's control over the control console during real surgery, creating a highly interactive training environment. During the training, the instructor can guide the trainees to complete various surgical procedures through real-time visual and tactile feedback, enabling them to gradually master surgical skills in a near-realistic operating environment. This not only improves learning efficiency but also enhances the immersiveness and practicality of the teaching.

[0124] In some embodiments, considering that the instructor may communicate and exchange surgical details with other doctors or nurses during the actual surgery, or that the instructor may explain surgical steps and precautions during the simulated surgery, and considering that this part of the voice data can also be used to guide the trainee doctor to learn surgical skills, the voice data during the surgical process can be collected at the same time when the surgical video stream is collected, so that the trainee doctor can hear the details or explanations about the surgery in the voice data when playing the surgical video stream.

[0125] It should be noted that the above embodiments mainly use surgical video streams as an example to provide visual feedback to trainees, but this does not mean that visual feedback can only be surgical video streams.

[0126] Based on this, in some embodiments, a surgical robot system is provided, the surgical robot system comprising: at least one surgical robot, each of the at least one surgical robot including a first console and a patient-side robot, the first console being used to control the patient-side robot in which it is located to perform surgery; at least one second console, each of the at least one second console being connected to at least a portion of the at least one surgical robot; each of the at least one second console being used to acquire surgical operation data from the connected surgical robot and control the movement of its own control components according to the surgical operation data to provide tactile feedback.

[0127] In some embodiments, the surgical robot system further includes: a server, wherein at least a portion of the second consoles is connected to at least a portion of the surgical robots via the server; and / or, at least a portion of the second consoles is directly connected to at least a portion of the surgical robots.

[0128] In some embodiments, each of the at least one second console is connected to the first console or patient-side robot of at least a portion of the surgical robot in at least one surgical robot.

[0129] In some embodiments, at least a portion of a surgical robot further includes a vision cart; each of at least one second console is connected to a first console, a patient-side robot, or a vision cart of at least a portion of the surgical robot in the at least one surgical robot.

[0130] In some embodiments, at least a portion of the second consoles in at least one second console are further configured to send a download instruction to a surgical robot in at least one surgical robot, so that the surgical robot receiving the download instruction sends surgical operation data to the corresponding second console in response to the download instruction; and / or, the first console is further configured to actively send surgical operation data to at least a portion of the second consoles in at least one second console.

[0131] In some embodiments, surgical operation data includes information about the first console controlling the patient-side robot.

[0132] In some embodiments, the information controlled by the first console to control the patient-side robot includes motion information of the control components of the first console; at least a portion of the second console is also used to control its own control components to move according to the motion information of the control components of the first console.

[0133] In some embodiments, at least a portion of the second console is further configured to acquire motion information of its own control component when controlling the movement of its own control component, and to compensate for the movement of its own control component based on the difference between the motion information of its own control component and the motion information of the control component of the connected first console.

[0134] In some embodiments, at least a portion of the second console is further configured to, when compensating for the movement of its own control component based on the difference between the movement information of its own control component and the movement information of the control component of the connected first console, determine the current position of its own control component based on the movement information of its own control component, determine the target position of its own control component based on the movement information of the control component of the connected first console, and control its own control component to move from the current position to the target position.

[0135] In some embodiments, at least a portion of the second console is further configured to restrict the movement of its own control component in directions other than a specified direction during the process of controlling its own control component to move from the current position to the target position, wherein the specified direction is the direction from the current position to the target position.

[0136] In some embodiments, the difference between motion information of at least a portion of the control components of at least one second console and motion information of the control components of the connected first console is also used for operational skill assessment.

[0137] In some embodiments, the motion information of the first console and / or at least a portion of the motion information of at least one second console includes one or more of the following: position, spatial orientation, motion speed, motion acceleration, and rotation angle.

[0138] In some embodiments, when the control component controlling itself moves according to the motion information of the control component of the first console, at least a portion of the control components of the second console and the control components of the first console are the same type of control component or different control components.

[0139] This application also provides a surgical robot system. In some embodiments, the surgical robot system includes: a server; at least one second console, each of the at least one second console being connected to the server, and each of the at least one second console being used to acquire surgical operation data from the server and control the movement of its own control components according to the surgical operation data to provide tactile feedback.

[0140] In some embodiments, at least a portion of the second console in at least one of the second consoles is also used to upload locally stored surgical operation data to the server.

[0141] In some embodiments, the system further includes a display device, or the second console further includes a display interface; the system is also configured to output surgical scene visual feedback data corresponding to the surgical operation data on the display device or the display interface of the second console.

[0142] In some embodiments, the surgical scene visual feedback data includes real-time acquired images of the surgical scene during the actual surgical procedure, or pre-recorded and stored images of the surgical scene during the actual surgical procedure.

[0143] In some embodiments, the visual feedback data of the surgical scene includes the actual surgical instruments held on the surgical robot during the surgery.

[0144] In some embodiments, the surgical scene visual feedback data includes a video stream of a virtual surgery, which includes: images of a virtual surgical scene displayed through a surgical simulator.

[0145] In some embodiments, the virtual surgical scene includes images obtained by the user simulating surgery using a surgical simulator, and / or the surgical operation data includes information about the user controlling the surgical simulator.

[0146] In some embodiments, the surgical scene visual feedback data and surgical operation data are synchronized in time.

[0147] In some embodiments, at least a portion of the second console is further configured to determine a set of target motion information corresponding to the content of the currently output surgical scene visual feedback data based on the time synchronization between the surgical scene visual feedback data and the surgical operation data, and control the movement of its own control components according to the target motion information.

[0148] In some embodiments, at least a portion of the second console is further configured to overlay a virtual surgical tool onto the output screen of the surgical scene visual feedback data, the virtual surgical tool moving in the output screen of the surgical scene visual feedback data based on the movement of the control components of the second console.

[0149] In some embodiments, the surgical scene visual feedback data also includes voice data, which includes details or explanations about the surgery.

[0150] In some embodiments, the surgical scene visual feedback data includes a recorded video stream, and at least a portion of the second console in at least one of the second consoles is further configured to adjust the movement speed of its own control components according to the user's adjustment of the playback speed of the recorded video stream.

[0151] In some embodiments, the second console includes an input component and a display interface; the second console is also used to acquire the user's control input component and render a simulator on the display interface of the second console according to the control input. The simulator is used to simulate the surgical instruments held by the surgical robot so that the user can adjust the control input based on the pose difference between the simulator and the surgical instruments in the training materials.

[0152] In some embodiments, at least a portion of the second console in at least one second console is further configured to prevent the user from operating its own control components; or, at least a portion of the second console in at least one second console is further configured to allow the user to operate its own control components; or, at least a portion of the second console in at least one second console includes a first mode and a second mode, wherein in the first mode, the control components do not respond to the user's control operations, and in the second mode, the control components respond to the user's control operations.

[0153] In some embodiments, at least a portion of the second console in at least one second console includes multiple control modes, with different sensitivity and / or resistance of the control components in different control modes.

[0154] This application also provides a console operation control method. In some embodiments, the console operation control method includes: acquiring surgical scene visual feedback data and surgical operation data, wherein the surgical operation data includes motion information of the control components of a first console, and the motion information of the control components of the first console is used to control the surgical robot to perform the surgical operation indicated by the surgical scene visual feedback data; during the output of surgical scene visual feedback data, controlling the movement of the control components of a second console based on the surgical operation data to provide tactile feedback through the second console.

[0155] In some embodiments, the surgical scene visual feedback data and surgical operation data are synchronized in time. During the output of the surgical scene visual feedback data, the movement of the control components of the second console is controlled based on the surgical operation data. This can be achieved by: playing the surgical scene visual feedback data through the display device of the second console, and determining a set of target motion information corresponding to the currently output video frame based on the time synchronization between the surgical scene visual feedback data and the surgical operation data; and controlling the movement of the control components of the second console based on the target motion information.

[0156] In some embodiments, controlling the movement of the control components of the second console based on surgical operation data can be achieved by controlling the control components of the second console to move according to the movement information of the control components of the first console included in the surgical operation data.

[0157] In some embodiments, controlling the movement of the control components of the second console based on surgical operation data can be achieved by: acquiring the movement information of the control components of the second console; and compensating for the movement of the control components of the second console based on the difference between the movement information of the control components of the second console and the movement information of the control components of the first console.

[0158] In some embodiments, the surgical scene visual feedback data includes a surgical video stream, and the operation control method of the console further includes: playing the surgical video stream through the display device of the second console, and superimposing a virtual surgical tool on the surgical video stream, wherein the virtual surgical tool moves based on the movement of the control components of the second console.

[0159] In some embodiments, the console operation control method further includes: performing an operation skill assessment based on the difference between the motion information of the control component of the second console and the motion information of the control component of the first console.

[0160] In some embodiments, the motion of the control component of the second console is compensated based on the difference between the motion information of the control component of the second console and the motion information of the control component of the first console. This can be achieved by: determining the current position of the control component of the second console based on the motion information of the control component of the second console; determining the target position of the control component of the second console based on the motion information of the control component of the first console; and controlling the control component of the second console to move from the current position to the target position.

[0161] In some embodiments, the operation control method of the console further includes: during the process of controlling the control component of the second console to move from the current position to the target position, restricting the movement of the control component of the second console in directions other than a specified direction, wherein the specified direction is the direction from the current position to the target position.

[0162] In some embodiments, the second console includes a first mode and a second mode. In the first mode, the control components of the second console do not respond to user control operations, while in the second mode, the control components of the second console respond to user control operations.

[0163] In some embodiments, motion information includes one or more of the following: position, spatial orientation, motion speed, motion acceleration, and rotation angle.

[0164] In some embodiments, the surgical scene visual feedback data includes a recorded surgical video stream, and the movement speed of the control components of the second console changes with the playback speed of the recorded video stream.

[0165] In some embodiments, the visual feedback data of the surgical scene includes a live video stream, and the surgical robot is a surgical simulator.

[0166] In some embodiments, the surgical scene visual feedback data includes a live surgical video stream, and the surgical robot is a surgical robot that follows the movement of the first console.

[0167] It is not difficult to see that the above system embodiments and method embodiments correspond to the previous embodiments. The relevant technical details mentioned in the previous embodiments are still valid in the above system embodiments and method embodiments. In order to reduce repetition, they will not be repeated here.

[0168] The solutions in the above embodiments can be freely combined to obtain new solutions when there is no conflict. Due to space limitations, they will not be listed one by one here.

[0169] Accordingly, this application also provides a computer program product, which includes a computer program that, when executed, implements the methods mentioned in any of the above embodiments.

[0170] Furthermore, this application also provides an electronic device, as shown in FIG9. The electronic device 210 includes a processor 211, a memory 212, and a computer program stored in the memory 212 that can be executed by the processor 211. When the processor 211 executes the computer program, it implements the methods mentioned in any of the above embodiments. The electronic device can also be configured with various other hardware or software based on actual needs, and this application does not impose any limitations.

[0171] In some scenarios, the electronic device can be a console, and it may also include other hardware structures that implement the functions of a console, which will not be elaborated here.

[0172] Accordingly, this application also provides a computer storage medium storing a program, which, when executed by a processor, implements the methods mentioned in any of the above embodiments.

[0173] The embodiments of this application may take the form of a computer program product implemented on one or more storage media containing program code (including but not limited to disk storage, CD-ROM (Compact Disc Read-Only Memory), optical storage, etc.). Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to: phase-change memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, optical discs, CD-ROMs, digital video discs (DVDs) or other optical storage, magnetic tape, magnetic disks or other magnetic storage devices, or any other non-transfer medium that can be used to store information that can be accessed by a computing device.

[0174] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0175] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0176] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0177] The methods and apparatus provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this application should not be construed as a limitation of this application.

Claims

1. A surgical robot system, comprising: At least one surgical robot, each of the at least one surgical robot including a first console and a patient-side robot, the first console being used to control the patient-side robot in which the surgical robot performs surgery; At least one second console, each of the at least one second console being connected to at least a portion of the at least one surgical robot; Each of the at least one second console is used to acquire surgical operation data from the connected surgical robot and control the movement of its own control components based on the surgical operation data to provide tactile feedback.

2. The system of claim 1, wherein, The surgical robot system further includes a server, and at least a portion of the at least one second console is connected to at least a portion of the at least one surgical robot via the server; And / or, At least a portion of the at least one second console is directly connected to at least a portion of the at least one surgical robot.

3. The system of claim 1 or 2, wherein, Each of the at least one second console is connected to the first console of at least a portion of the surgical robots in the at least one surgical robot or the patient-side robot.

4. The system of claim 1 or 2, wherein, At least a portion of the surgical robots in the at least one surgical robot also includes a vision cart; Each of the at least one second console is connected to the first console of at least a portion of the at least one surgical robot, the patient-side robot, or the vision cart.

5. The system of any one of claims 1 to 4, wherein, At least a portion of the at least one second console is further configured to send a download instruction to the surgical robot in the at least one surgical robot, so that the surgical robot receiving the download instruction responds to the download instruction and sends the surgical operation data to the corresponding second console; And / or, The first console is also used to actively send the surgical operation data to at least a portion of the at least one second console.

6. The system of any one of claims 1 to 5, wherein, The surgical operation data includes information about the first console controlling the patient-side robot.

7. The system of claim 6, wherein, The information used by the first console to control the patient-side robot includes the motion information of the control components of the first console; At least a portion of the at least one second console is also used to control its own control components to move according to the motion information of the control components of the first console.

8. The system of claim 7, wherein, At least a portion of the second console is further configured to acquire motion information of its own control component when controlling the movement of its own control component, and to compensate for the movement of its own control component based on the difference between the motion information of its own control component and the motion information of the control component of the connected first console.

9. The system of claim 8, wherein, At least a portion of the second console is further configured to, when compensating for the movement of its own control component based on the difference between the movement information of its own control component and the movement information of the control component of the connected first console, determine the current position of its own control component based on the movement information of its own control component, determine the target position of its own control component based on the movement information of the control component of the connected first console, and control its own control component to move from the current position to the target position.

10. The system of claim 9, wherein, At least a portion of the second console is further configured to restrict the movement of its own control component in directions other than a specified direction during the process of controlling its own control component to move from the current position to the target position, wherein the specified direction is the direction from the current position to the target position.

11. The system of any one of claims 7 to 10, wherein, The difference between the motion information of the control components of at least a portion of the at least one second console and the motion information of the control components of the connected first console is also used for operational skill assessment.

12. The system of any one of claims 7 to 11, wherein, The motion information of the first console and / or at least a portion of the motion information of the second console includes one or more of the following: position, spatial orientation, motion speed, motion acceleration, and rotation angle.

13. The system of any one of claims 7 to 12, wherein, When the control component of the control device moves according to the motion information of the control component of the first console, at least a portion of the control components of the second console and the control components of the first console are the same type of control component or different control components.

14. A surgical robot system, comprising: server; At least one second console, each of the at least one second console being connected to the server, each of the at least one second console being used to acquire surgical operation data from the server and control the movement of its own control components according to the surgical operation data to provide tactile feedback.

15. The system of claim 14, wherein, At least a portion of the at least one second console is also used to upload the surgical operation data stored locally to the server.

16. The system of any one of claims 1 to 15, wherein, The system may also include a display device, or the second console may also include a display interface; The system is also used to output surgical scene visual feedback data corresponding to the surgical operation data on the display interface of the display device or the second console.

17. The system of claim 16, wherein, The surgical scene visual feedback data includes real-time images of the surgical scene during the actual surgical procedure, or pre-recorded and stored images of the surgical scene during the actual surgical procedure.

18. The system of claim 17, wherein, The surgical scene visual feedback data includes the actual surgical instruments held by the surgical robot during the surgery.

19. The system of claim 16, wherein, The surgical scene visual feedback data includes a video stream of the virtual surgery, which includes: a virtual surgical scene displayed through a surgical simulator.

20. The system of claim 19, wherein, The virtual surgical scene includes images obtained by the user simulating surgery using the surgical simulator, and / or the surgical operation data includes information about the user controlling the surgical simulator.

21. The system of any one of claims 16 to 20, wherein, The visual feedback data of the surgical scene and the surgical operation data are synchronized in time.

22. The system according to claim 21, wherein, At least a portion of the second console is also used to determine a set of target motion information corresponding to the content of the currently output surgical scene visual feedback data based on the time synchronization between the surgical scene visual feedback data and the surgical operation data, and to control the movement of its own control components according to the target motion information.

23. The system according to any one of claims 16 to 22, wherein, At least a portion of the second console is also used to overlay a virtual surgical tool onto the output screen of the surgical scene visual feedback data, the virtual surgical tool moving in the output screen of the surgical scene visual feedback data based on the movement of the control components of the second console.

24. The system according to any one of claims 16 to 23, wherein, The surgical scene visual feedback data also includes voice data, which includes details or explanations about the surgery.

25. The system according to any one of claims 16 to 24, wherein, The surgical scene visual feedback data includes a recorded video stream, and at least a portion of the at least one second console is also used to adjust the movement speed of its own control components according to the user's adjustment of the playback speed of the recorded video stream.

26. The system according to any one of claims 1 to 25, wherein, The second console includes an input component and the display interface; The second console is also used to acquire the user's control input to the input component and render a simulator on the display interface of the second console according to the control input. The simulator is used to simulate the surgical instruments held by the surgical robot so that the user can adjust the control input based on the pose difference between the simulator and the surgical instruments in the training materials.

27. The system according to any one of claims 1 to 26, wherein, At least a portion of the at least one second console is also used to prevent the user from operating its own control components. or, At least a portion of the at least one second console is also used to allow a user to operate its own control components; or, At least a portion of the at least one second console includes a first mode and a second mode, wherein in the first mode, the control component does not respond to user control operations, and in the second mode, the control component responds to user control operations.

28. The system according to any one of claims 1 to 27, wherein, At least a portion of the at least one second console includes multiple control modes, with different sensitivity and / or resistance of the control components in different control modes.

29. A method for controlling the operation of a console, the method comprising: The surgical scene visual feedback data and surgical operation data are acquired. The surgical operation data includes motion information of the control components of the first console. The motion information of the control components of the first console is used to control the surgical robot to perform the surgical operation indicated by the surgical scene visual feedback data. During the output of the surgical scene visual feedback data, the movement of the control components of the second console is controlled based on the surgical operation data to provide tactile feedback through the second console.

30. The method according to claim 29, wherein, The surgical scene visual feedback data and the surgical operation data are synchronized in time. During the output of the surgical scene visual feedback data, controlling the movement of the control components of the second console based on the surgical operation data includes: The surgical scene visual feedback data is played through the display device of the second console, and a set of target motion information corresponding to the currently output video frame is determined based on the time synchronization between the surgical scene visual feedback data and the surgical operation data. The movement of the control components of the second console is controlled based on the target motion information.

31. The method according to claim 29 or 30, wherein, The control of the movement of the manipulation components of the second console based on the surgical operation data includes: The control components of the second console move according to the motion information of the control components of the first console included in the surgical operation data.

32. The method according to any one of claims 29 to 31, wherein, The control of the movement of the manipulation components of the second console based on the surgical operation data includes: Obtain motion information of the control components of the second console; The motion of the control component of the second console is compensated based on the difference between the motion information of the control component of the second console and the motion information of the control component of the first console.

33. The method according to any one of claims 29 to 32, wherein, The surgical scene visual feedback data includes a surgical video stream, and the method further includes: The surgical video stream is played through the display device of the second console, and a virtual surgical tool is overlaid on the surgical video stream, the virtual surgical tool moving based on the movement of the control components of the second console.

34. The method according to any one of claims 29 to 33, wherein, The method further includes: Operational skills are assessed based on the difference between the motion information of the control components of the second console and the motion information of the control components of the first console.

35. The method according to claim 32, wherein, The compensation for the movement of the control component of the second console based on the difference between the motion information of the control component of the second console and the motion information of the control component of the first console includes: The current position of the control components of the second console is determined based on the motion information of the control components of the second console; The target position of the control component of the second console is determined based on the motion information of the control component of the first console. The control components of the second console are moved from the current position to the target position.

36. The method according to claim 35, wherein, The method further includes: During the process of controlling the operation component of the second console to move from the current position to the target position, the movement of the operation component of the second console in directions other than a specified direction is restricted, wherein the specified direction is the direction from the current position to the target position.

37. The method according to any one of claims 29 to 36, wherein, The second console includes a first mode and a second mode. In the first mode, the control components of the second console do not respond to user control operations. In the second mode, the control components of the second console respond to user control operations.

38. The method according to any one of claims 29 to 37, wherein, The motion information includes one or more of the following: position, spatial orientation, motion speed, motion acceleration, and rotation angle.

39. The method according to any one of claims 29 to 38, wherein, The surgical scene visual feedback data includes a recorded surgical video stream, and the movement speed of the control components of the second console changes with the playback speed of the recorded video stream.

40. The method according to any one of claims 29 to 38, wherein, The visual feedback data of the surgical scene includes a live video stream, and the surgical robot is a surgical simulator.

41. The method according to any one of claims 29 to 38, wherein, The surgical scene visual feedback data includes a live surgical video stream, and the surgical robot is a surgical robot that follows the movement of the first console.

42. An electronic device comprising a processor, a memory, and a computer program stored in the memory executable by the processor, wherein the processor executes the computer program to implement the method of any one of claims 29 to 41.

43. A computer program product comprising a computer program that, when executed, implements the method of any one of claims 29 to 41.

Images