Manipulation system, method for controlling manipulation system, and program
The manipulation system simplifies training data acquisition by integrating in-system data storage and mode switching, reducing effort and improving learning accuracy through retraining.
Patent Information
- Application Number
- PCT/JP2026/017642
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-16
- Filing Date
- 2026-05-08
- Publication Date
- 2026-11-19
AI Technical Summary
Existing manipulation systems require significant time and effort to acquire training data for generating a learned model, necessitating separate preparation or connection of signal lines from an existing manipulation system.
A manipulation system equipped with an input unit for storing operator operations as training data in a memory unit, allowing switching between autonomous and remote control modes to simplify data acquisition and improve learning accuracy through retraining.
Reduces the effort required to acquire training data by enabling in-system data storage and mode switching, facilitating easy reacquisition and enhancing the learning accuracy of the trained model.
Smart Images

Figure JP2026017642_19112026_PF_FP_ABST
Abstract
Description
Manipulation system, control method and program for manipulation system
[0001] The present disclosure relates to a manipulation system, a control method for the manipulation system, and a program.
[0002] Conventionally, a manipulation system including an operation device and a manipulation device is known. Japanese Patent No. 7334499 discloses a surgery support system including a master device and a slave device. In this surgery support system, the master device includes an operation device that an operator grips and operates. The slave device includes a surgical manipulator. When the operator operates the operation device, the surgical manipulator is actuated. Further, in Japanese Patent No. 7334499, the surgical manipulator can be switched between an autonomous drive mode and a manual operation mode. In the autonomous drive mode, a surgical field environment is sensed by a sensor, and the next operation of the surgical manipulator is determined based on the sensing result. In the manual operation mode, the surgical manipulator is actuated as the operator operates the operation device. Further, in Japanese Patent No. 7334499, the autonomous drive mode and the manual operation mode are switched based on the sensed situation of the operating room.
[0003] Japanese Patent No. 7334499
[0004] Here, in the autonomous drive mode of a manipulation system such as that disclosed in Japanese Patent No. 7334499, the autonomous drive mode may be executed using a learned model learned by machine learning. In this case, to generate the learned model, motion commands when the surgical manipulator is actually actuated by the operator operating the operation device, the surgical field environment sensed by the sensor, and the like are used as training data. At this time, to generate the training data, it is necessary to separately prepare a manipulation system, or connect signal lines for acquiring motion commands and the surgical field environment from an existing manipulation system. For this reason, there is a problem that acquisition of training data for generating a learned model takes time and effort.
[0005] This disclosure provides a manipulation system, a method for controlling the manipulation system, and a program that can reduce the effort required to acquire training data for generating a trained model.
[0006] A manipulation system according to the first aspect of this disclosure comprises: an operating device operated by an operator; a manipulation device that operates based on an operation command; a manual operation command generation unit that generates a manual operation command as an operation command based on the operator's operation of the operating device; a trained model that takes observational information of the environment surrounding the manipulation device and an operation command as input and outputs an inferred operation command as an inferred operation command; a switching unit that switches between an autonomous mode that operates based on the inferred operation command output from the trained model and a remote control mode that operates based on a manual operation command generated by the manual operation command generation unit; and an input unit that is disposed separately from the switching unit and inputs instructions to start and stop a storage operation that stores the operator's operation content as training data for generating the trained model in a storage unit.
[0007] The manipulation system according to the first aspect of this disclosure includes an input unit for inputting instructions to start and stop a memory operation that stores the operator's operations as training data for generating a trained model in a memory unit. As a result, the manipulation system itself is equipped with an input unit for storing training data in a memory unit, eliminating the need to prepare a separate manipulation system for generating training data or to connect signal lines to acquire operation commands and surgical field environments from an existing manipulation system. Therefore, the effort required to acquire training data for generating a trained model can be reduced. Furthermore, since a switching unit is provided to switch between autonomous mode and remote control mode, operation commands and surgical field environments can be acquired as training data from the current manipulation system simply by operating the input unit while in remote control mode using the switching unit. Therefore, it is possible to easily reacquire training data and improve the learning accuracy of the trained model through retraining.
[0008] A control method for a manipulation system according to the second aspect of the present disclosure is a control method for a manipulation system including an operating device operated by an operator and a manipulation device that operates based on an operation command, comprising: generating a manual operation command as an operation command based on the operator's operation of the operating device; switching between an autonomous mode that operates based on an inferred operation command output from a trained model that takes observation information of the environment surrounding the manipulation device and an operation command as input and outputs an inferred operation command as an inferred operation command, and a remote control mode that operates based on the generated manual operation command, by operating an input unit arranged separately from the switching unit, thereby starting and stopping a storage operation that stores the operator's operation content as training data for generating the trained model in a storage unit.
[0009] The control method for the manipulation system according to the second aspect of this disclosure, as described above, includes starting and stopping a storage operation in which the operator's operations as training data for generating a trained model are stored in the storage unit by operating an input unit which is arranged separately from the switching unit. As a result, the manipulation system itself is equipped with an input unit for storing training data in the storage unit, so there is no need to prepare a separate manipulation system for generating training data or to connect signal lines to acquire operation commands and surgical field environments from an existing manipulation system. Therefore, it is possible to provide a control method for the manipulation system that can suppress the effort required to acquire training data for generating a trained model. Furthermore, since a switching unit is provided to switch between autonomous mode and remote control mode, operation commands and surgical field environments can be acquired as training data from the current manipulation system simply by operating the input unit while the system is switched to remote control mode by the switching unit. Therefore, it is possible to easily reacquire training data and improve the learning accuracy of the trained model through retraining.
[0010] A program according to a third aspect of this disclosure is a program that performs a control method for a manipulation system including an operating device operated by an operator and a manipulation device that operates based on an operation command, the program performing a control method for a manipulation system comprising: generating a manual operation command as an operation command based on the operator's operation of the operating device; switching between an autonomous mode that operates based on an inferred operation command output from a trained model that takes observation information of the environment surrounding the manipulation device and an operation command as input and outputs an inferred operation command as an inferred operation command, and a remote control mode that operates based on the generated manual operation command, by operating an input unit located separately from the switching unit, thereby starting and stopping a storage operation that stores the operator's operation content as training data for generating the trained model in a storage unit.
[0011] The program according to the third aspect of this disclosure, as described above, executes a control method for a manipulation system that starts and stops a storage operation in which the operator's operations as training data for generating a trained model are stored in the storage unit, by operating an input unit which is arranged separately from the switching unit. As a result, the manipulation system itself is equipped with an input unit for storing training data in the storage unit, so there is no need to prepare a separate manipulation system for generating training data or to connect signal lines to acquire operation commands and surgical field environments from an existing manipulation system. Therefore, it is possible to provide a program that can suppress the effort required to acquire training data for generating a trained model. Furthermore, since a switching unit is provided to switch between autonomous mode and remote control mode, operation commands and surgical field environments can be acquired as training data from the current manipulation system simply by operating the input unit while the system is switched to remote control mode by the switching unit. Therefore, it is possible to easily reacquire training data and improve the learning accuracy of the trained model through retraining.
[0012] The manipulation system, control method for the manipulation system, and program of this disclosure can reduce the effort required to acquire training data for generating a trained model.
[0013] This is a diagram showing the configuration of a surgical support system according to one embodiment. This is a diagram showing the display unit of a medical trolley according to one embodiment. This is a diagram showing the configuration of a medical trolley according to one embodiment. This is a diagram showing the configuration of a robot arm according to one embodiment. This is a diagram showing the instruments. This is a perspective view showing the configuration of the arm operation unit according to one embodiment. This is a diagram for explaining the translational movement of the robot arm. This is a diagram for explaining the rotational movement of the robot arm. This is a diagram showing an endoscope. This is a diagram showing a pivot position setting device. This is a diagram showing the operation unit according to one embodiment. This is a diagram showing the right-hand wrist unit according to one embodiment. This is a diagram showing the left-hand wrist unit according to one embodiment. This is a perspective view showing a foot pedal according to one embodiment. This is a control block diagram of a surgical support system according to one embodiment. This is a control block diagram of a robot arm according to one embodiment. This is a control block diagram of a positioner and medical trolley according to one embodiment. This is a control block diagram of the operation unit according to one embodiment. This is a diagram for explaining the control of the first control device according to one embodiment. This is a diagram for explaining the configuration of an inference control device according to one embodiment. This is a diagram showing the patient's surgical area and instruments displayed on the monitor and display unit. This is a diagram showing the patient's surgical area, TCP coordinate system, etc., displayed on the monitor and display unit. This is a flowchart for explaining the control method of a surgical support system according to one embodiment. This is a flowchart illustrating the autonomous mode of a surgical support system according to one embodiment. This is a diagram illustrating the state in which the suture needle is inserted into the patient's body during suturing. This is a diagram illustrating the state in which the suture needle and thread are pulled during suturing. This is a diagram illustrating the state in which the thread is tied during suturing. This is a flowchart illustrating the operation of collecting training data. This is a diagram illustrating an example of operator operation. This is a diagram of a social robot as an example. This is a diagram of an industrial robot as an example.
[0014] (Configuration of the surgical support system) The configuration of the surgical support system 500 according to this embodiment will now be described. The surgical support system 500 comprises a surgical support robot 100, a remote control device 200, a vision unit 300, an image processing unit 400, and an inference control device 600. The surgical support robot 100 is an example of a surgical device. The remote control device 200 is an example of an operating device. The surgical support system 500 is an example of a manipulation system.
[0015] In this specification, as shown in Figure 4, the longitudinal direction of the surgical instrument 1 is defined as the Z direction. The tip side of the surgical instrument 1 is defined as the Z1 side, and the base side of the surgical instrument 1 is defined as the Z2 side. The direction perpendicular to the Z direction is defined as the X direction. The direction perpendicular to both the Z and X directions is defined as the Y direction.
[0016] As shown in Figure 1, the surgical robot 100 is positioned inside the operating room. The remote control device 200 is positioned at a distance from the surgical robot 100. The remote control device 200 and the surgical robot 100 may communicate via a wired connection or wirelessly. The remote control device 200 receives commands for the surgical instrument 1. Specifically, an operator, such as a physician, inputs commands to the remote control device 200 to cause the surgical robot 100 to perform a desired action. The remote control device 200 transmits the input commands to the surgical robot 100. The surgical robot 100 operates based on the received commands. The surgical robot 100 is positioned inside the operating room, which is a sterile field.
[0017] (Configuration of the surgical support robot) As shown in Figure 1, the surgical support robot 100 comprises a medical trolley 10, a trolley positioner operating unit 20, a positioner 30, an arm base 40, a plurality of robot arms 50, and an arm operating unit 60 provided on each robot arm 50. The plurality of robot arms 50 may be arranged on an operating table on which the patient P is placed.
[0018] As shown in Figure 3, the trolley positioner operation unit 20 is supported by a trolley positioner operation support unit 21 at the rear of the medical trolley 10, and the medical trolley 10 or positioner 30 is moved when the trolley positioner operation unit 20 is operated. The trolley positioner operation unit 20 includes an input device 22 and an operation handle 23. The input device 22 accepts operations for moving and changing the posture of the positioner 30, arm base 40, and multiple robot arms 50, mainly for preparing for surgery before the procedure. The medical trolley 10 includes an operation handle 23.
[0019] As shown in Figure 3, the input device 22 includes a display unit 22a, a joystick 22b, an enable switch 22c, an error reset button 22d, and a speaker 22e. The display unit 22a is, for example, a liquid crystal panel. As shown in Figure 2, the display unit 22a displays numbers corresponding to multiple robot arms 50. The display unit 22a also displays the type of surgical instrument 1 attached to each of the multiple robot arms 50. The display unit 22a displays a check mark CM indicating that the pivot position PP, which will be described later, has been set.
[0020] As shown in Figure 3, the joystick 22b is positioned near the display unit 22a of the input device 22. By selecting an operation mode displayed on the display unit 22a and operating the joystick 22b, the positioner 30 is moved in three dimensions.
[0021] The enable switch 22c is located near the joystick 22b. The enable switch 22c allows or disallows the movement of the positioner 30. When the enable switch 22c is pressed and the movement of the positioner 30 is permitted, the joystick 22b is operated, causing the positioner 30 to move.
[0022] The error reset button 22d clears errors in the surgical support system 500. The errors are, for example, deviation anomaly errors. A pair of speakers 22e are provided. The pair of speakers 22e are located near the location where the positioner 30 of the medical trolley 10 is positioned.
[0023] Furthermore, the operating handle 23 is located near the display unit 22a. The operating handle 23 has a throttle 23a that controls the movement of the medical cart 10 when grasped and rotated by an operator such as a nurse or technician. Specifically, the operating handle 23 is located below the input device 22. When the throttle 23a is rotated from the front to the back, the medical cart 10 moves forward. When the throttle 23a is rotated from the back to the front, the medical cart 10 moves backward. The speed of the medical cart 10 is also changed according to the amount of rotation of the throttle 23a. The operating handle 23 is configured to be rotatable left and right, as indicated by the R direction, and the medical cart 10 rotates along with the rotation of the operating handle 23.
[0024] Furthermore, an enable switch 23b is located on the operating handle 23 to allow or disallow the movement of the medical trolley 10. When the enable switch 23b is pressed and the movement of the medical trolley 10 is permitted, the throttle 23a on the operating handle 23 is operated, causing the medical trolley 10 to move.
[0025] As shown in Figure 1, the positioner 30 consists of a 7-axis articulated robot. The positioner 30 is positioned on a medical trolley 10. The positioner 30 is a 7-axis articulated robot that adjusts the position of the arm base 40. The positioner 30 moves the position of the arm base 40 in three dimensions. However, the positioner 30 is not limited to a 7-axis articulated robot; for example, it may be an articulated robot with 6 axes or less, or 8 axes or more.
[0026] The positioner 30 includes a base portion 31 and a plurality of link portions 32 connected to the base portion 31. The plurality of link portions 32 are connected to each other by joints 33.
[0027] The arm base 40 is attached to the tip of the positioner 30. The base of each of the multiple robot arms 50 is attached to the arm base 40. The multiple robot arms 50 can be folded and stored. The arm base 40 and the multiple robot arms 50 are used covered with sterile drapes. The robot arms 50 also support surgical instruments 1.
[0028] The arm base 40 is equipped with a status indicator 41 and an arm status indicator 42, as shown in Figure 15. The status indicator 41 displays the status of the surgical support system 500. The arm status indicator 42 displays the status of the robot arm 50.
[0029] Multiple robot arms 50 are arranged. Specifically, four robot arms 50a, 50b, 50c, and 50d are arranged. Robot arms 50a, 50b, 50c, and 50d have similar configurations to each other. Robot arms 50a, 50b, 50c, and 50d are operated independently based on motion commands. Note that robot arms 50 are examples of manipulation devices and articulated robot arms. Also, robot arms 50a, 50b, and 50d are examples of second robot arms. Also, robot arm 50c is an example of a first robot arm.
[0030] As shown in Figure 4, the robot arm 50 includes an arm section 51, a first link section 52, a second link section 53, and a translational movement mechanism section 54. The robot arm 50 has joints JT1, JT2, JT3, JT4, JT5, JT6, JT7, and JT8. Joints JT1, JT2, JT3, JT4, JT5, JT6, and JT7 each have axes A1, A2, A3, A4, A5, A6, and A7 as rotation axes. JT8 has an axis A8 as a linear motion axis. The arm section 51 includes a base section 51a and a link section 51b.
[0031] The arm section 51 consists of a 7-axis articulated robot arm. The first link section 52 is located at the tip of the arm section 51. The arm operation section 60, which will be described later, is attached to the second link section 53. The translational movement mechanism section 54 is located between the first link section 52 and the second link section 53. A holder 55 for holding the surgical instrument 1 is located on the second link section 53. The translational movement mechanism section 54 translates the holder 55, to which the surgical instrument 1 is attached, between a first position and a second position. The first position is the position of the end of the range of movement of the holder 55 by the translational movement mechanism section 54 along the A8 axis, on the Z2 side. The second position is the position of the end of the range of movement of the holder 55 by the translational movement mechanism section 54 along the A8 axis, on the Z1 side.
[0032] Each of the multiple robotic arms 50 has a surgical instrument 1 attached to its tip. The surgical instrument 1 includes, for example, a replaceable instrument 2, an endoscope 3 shown in Figure 9 for capturing images of the surgical site, and a pivot position setting device 4 shown in Figure 10 for setting the pivot position PP. The instrument 2 includes a driven unit 2a, an end effector 2b, a wrist joint 2c shown in Figure 5, and a shaft 2d. The end effector 2b is connected to the tip of the shaft 2d via the wrist joint 2c.
[0033] As shown in Figure 1, an endoscope 3 is attached to the tip of one of the multiple robot arms 50, for example, robot arm 50c, and instruments 2, which are surgical instruments 1 other than the endoscope 3, are attached to the tips of the remaining robot arms 50a, 50b, and 50d. Robot arms 50b and 50d are positioned adjacent to robot arm 50c. Robot arm 50a is positioned adjacent to robot arm 50b. Preferably, the endoscope 3 is attached to one of the two centrally positioned robot arms 50b and 50c, of the four robot arms 50 that are positioned adjacent to each other. The endoscope 3 observes the environment around robot arms 50a, 50b, and 50d and outputs the observation information. Specifically, the endoscope 3 images the surgical site of patient P as the environment around robot arms 50a, 50b, and 50d, and outputs the captured images to the monitor 140 of the remote control device 200 and the display unit 220 located on the cart 210. Note that endoscope 3 is an example of an environmental information acquisition unit and an imaging unit. Also, monitor 140 and display unit 220 are examples of display units.
[0034] (Instrument Configuration) As shown in Figure 5, an end effector 2b having jaw members 2g and 2h is attached to the tip of the instrument 2. Scissors, grapplers, needle holders, micro dissectors, stable applicators, tackers, suction cleaning tools, snare wires, and clip applicators can be used as the end effector 2b.
[0035] Instrument 2 includes a first support member 2e and a second support member 2f. The first support member 2e is attached to the shaft 2d. The second support member 2f is rotatably supported by the first support member 2e around the A10 axis and supports the end effector 2b so as to be rotatable around the A11 axis which intersects the A10 axis. The shaft 2d rotates around the A9 axis. The wrist joint 2c is positioned between the second support member 2f and the first support member 2e with the A10 axis as its axis of rotation.
[0036] (Configuration of the arm operating section) As shown in Figure 6, the arm operating section 60 is attached to the robot arm 50 and operates the robot arm 50. Specifically, the arm operating section 60 is attached to the second link section 53.
[0037] The arm operation unit 60 includes an enable switch 61, a joystick 62, a linear switch 63, a mode switching button 64, a mode indicator 65, a pivot button 66, and an adjustment button 67.
[0038] The enable switch 61, when pressed, allows or disallows the movement of the robot arm 50 by the joystick 62 and linear switch 63. When the arm control unit 60 is held by an operator such as a nurse or assistant, pressing the enable switch 61 allows the robot arm 50 to move the surgical instrument 1.
[0039] The joystick 62 is a control device for operating the movement of the surgical instrument 1 by the robot arm 50. The joystick 62 controls the direction and speed of movement of the robot arm 50. The robot arm 50 moves according to the direction and angle at which the joystick 62 is tilted.
[0040] The linear switch 63 is a switch for moving the surgical instrument 1 in the Z direction, which is the longitudinal direction of the surgical instrument 1. The linear switch 63 includes a linear switch 63a that moves the surgical instrument 1 in the direction of insertion into the patient P, and a linear switch 63b that moves the surgical instrument 1 away from the patient P. Both the linear switch 63a and the linear switch 63b are push-button switches.
[0041] The mode switching button 64 is a pushbutton switch for switching between a mode for translating the surgical instrument 1 and a mode for rotating and moving the surgical instrument 1. As shown in FIG. 7, in the mode for translating the robot arm 50, the robot arm 50 is moved such that the distal end 1a of the surgical instrument 1 moves on the X-Y plane. As shown in FIG. 8, in the mode for rotating and moving the robot arm 50, when the pivot position PP is not stored in the storage unit 351 shown in FIG. 15, the instrument 2 as the surgical instrument 1 is rotated and moved about the center of the end effector 2b on the A11 axis or the distal end of the end effector 2b as a fulcrum, and when the pivot position PP is stored in the storage unit 351, the robot arm 50 is moved such that the surgical instrument 1 is rotated and moved about the pivot position PP as a fulcrum. Note that the surgical instrument 1 is rotated and moved in a state where the shaft 1c of the surgical instrument 1 is inserted into the trocar T. The mode switching button 64 is disposed on the surface of the arm operation unit 60 on the Z direction side.
[0042] The mode indicator 65 displays the switched mode. Lighting of the mode indicator 65 indicates the rotational movement mode, and extinguishing of the mode indicator 65 indicates the translational movement mode. The mode indicator 65 also serves as a pivot position indicator that displays that the pivot position PP has been set. The mode indicator 65 is disposed on the surface of the arm operation unit 60 on the Z direction side.
[0043] The pivot button 66 is a pushbutton switch for setting a pivot position PP that serves as a fulcrum for movement of the surgical instrument 1 attached to the robot arm 50. When the pivot button 66 is pressed, the pivot position PP is stored in the storage unit 351.
[0044] The adjustment button 67 is a button for optimizing the position of the robot arm 50. After the pivot position PP for the robot arm 50 to which the endoscope 3 is attached is set, the positions of the other robot arms 50 and the arm base 40 are optimized by pressing the adjustment button 67. The adjustment button 67 is a button different from the enable switch 61.
[0045] (Remote Control Device) As shown in Figure 1, the remote control device 200 is located, for example, inside or outside the operating room. The remote control device 200 includes an operating unit 110, a foot pedal 120, a touch panel 130, a monitor 140, a support arm 150, a support bar 160, and an error reset button 161. The operating unit 110 constitutes an operating handle for an operator, such as a doctor, to input commands. The operating unit 110 receives commands for the robot arm 50.
[0046] As shown in Figure 11, the operating section 110 is a handle for operating the surgical instrument 1. The operating section 110 also receives input from the surgical instrument 1. The operating section 110 includes an operating section 110L located on the left side as viewed from the operator, such as a doctor, and operated by the operator's left hand, and an operating section 110R located on the right side and operated by the operator's right hand. The operating section 110 includes an arm section 111 and a wrist section 112. The operating section 110R includes an arm section 111R and a wrist section 112R. The operating section 110L includes an arm section 111L and a wrist section 112L.
[0047] The arm portion 111 has joints JT21, JT22, and JT23 as shown in Figure 11, and JT24, JT25, JT26, and JT27 as shown in Figures 12 and 13. The rotation axes of joints JT21, JT22, JT23, JT24, JT25, JT26, and JT27 are designated as axes A21, A22, A23, A24, A25, A26, and A27, respectively.
[0048] As shown in FIG. 11, the arm portion 111R includes a link portion 111a, a link portion 111b, and a link portion 111c. The upper end side of the link portion 111a is attached to the remote operation device 200 so as to be rotatable about an A21 axis extending along the vertical direction. The upper end side of the link portion 111b is attached to the lower end side of the link portion 111a so as to be rotatable about an A22 axis extending along the horizontal direction. One end side of the link portion 111c is attached to the lower end side of the link portion 111b so as to be rotatable about an A23 axis extending along the horizontal direction. A wrist portion 112 is attached to the other end side of the link portion 111c so as to be rotatable about an A24 axis. The link portion 111a is connected to the remote operation device 200 via a joint JT21. The link portion 111a and the link portion 111b are connected via a joint JT22. The link portion 111b and the link portion 111c are connected via a joint JT23. The arm portion 111 supports the wrist portion 112. The arm portion 111L also has the same configuration as the arm portion 111R.
[0049] The wrist portion 112 includes a wrist portion 112R operated by the operator's right hand shown in FIG. 12, and a wrist portion 112L operated by the operator's left hand shown in FIG. 13. FIG. 12 shows the reference posture of the operation portion 110R, and FIG. 13 shows the reference posture of the operation portion 110L. The configuration of the wrist portion 112R and the configuration of the wrist portion 112L are the same.
[0050] The wrist portion 112 includes a link portion 112a, a link portion 112b, a link portion 112c, and a grip support member 112d operated by an operator such as a doctor. The base end portion of the link portion 112a is connected to the distal end portion of the arm portion 111, and rotates about the A24 axis. The base end portion of the link portion 112b is connected to the distal end portion of the link portion 112a, and rotates about the A25 axis relative to the link portion 112a. The base end portion of the link portion 112c is connected to the distal end portion of the link portion 112b, the grip support member 112d is connected to the distal end portion of the link portion 112c, and the link portion 112c rotates about the A26 axis relative to the link portion 112b. The grip support member 112d rotates about the A27 axis relative to the link portion 112c. Each of the link portion 112a, the link portion 112b, and the link portion 112c has an L shape. The grip support member 112d is an example of a link portion.
[0051] The wrist section 112 includes a pair of grip members 112e that are opened and closed by the operator. The grip members 112e consist of elongated plate-shaped lever members, and the proximal ends of the pair of grip members 112e are rotatably connected to the proximal end of the grip support member 112d. The grip members 112e have cylindrical finger insertion portions 112f. The operator operates the wrist section 112 by inserting their fingers into the pair of finger insertion portions 112f. The base ends of the pair of grip members 112e are connected to the grip support member 112d, and the opening angle between the jaw members 2g and 2h is changed by increasing or decreasing the angle between the pair of grip members 112e. A magnet is placed on one of the grip members 112e, and a Hall sensor is placed on the grip support member 112d. When the operator opens and closes the grip members 112e, the magnet and Hall sensor function as angle detection sensors, and the Hall sensor outputs the opening angle. Alternatively, a Hall sensor may be placed on the grip member 112e, and a magnet may be placed on the grip support member 112d.
[0052] As shown in Figure 1, the monitor 140 is a scope-type display device for displaying images captured by the endoscope 3. The monitor 140 also has a notification unit 141, which emits an error sound. The support arm 150 supports the monitor 140 so that its height is at the same height as the operator's face, such as a doctor. The touch panel 130 is located on the support bar 160. The surgical support robot 100 can be operated by the remote control device 200 by detecting the operator's head using a sensor located near the monitor 140. The operator operates the control unit 110 and foot pedal 120 while viewing the surgical area on the monitor 140. This inputs a command to the remote control device 200. The command input to the remote control device 200 is transmitted to the surgical support robot 100.
[0053] (Foot pedals) As shown in Figure 14, the foot pedals 120 are provided in multiple units to perform functions related to the surgical instrument 1. The multiple foot pedals 120 are arranged on the base unit 121. The foot pedals 120 include a switching pedal 122, a clutch pedal 123, a camera pedal 124, an incision pedal 125, a coagulation pedal 126, and a foot detection unit 127. The switching pedal 122, clutch pedal 123, camera pedal 124, incision pedal 125, and coagulation pedal 126 are operated by the operator's feet. The incision pedal 125 includes an incision pedal 125R for the right robot arm 50 and an incision pedal 125L for the left robot arm 50. The coagulation pedal 126 includes a coagulation pedal 126R for the right robot arm 50 and a coagulation pedal 126L for the left robot arm 50. The coagulation pedal 126L is an example of a switching unit.
[0054] The switching pedal 122 switches the robot arm 50 to be operated by the control unit 110. The clutch pedal 123 performs a clutch operation that temporarily disconnects the control connection between the robot arm 50 and the control unit 110. While the clutch pedal 123 is pressed down by the operator, the operation of the control unit 110 is not transmitted to the robot arm 50. Also, while the camera pedal 124 is pressed down by the operator, the control unit 110 can operate the robot arm 50 to which the endoscope 3 is attached. While the incision pedal 125 or coagulation pedal 126 is pressed down by the operator, the electrosurgical device is activated.
[0055] (Vision Unit and Image Processing Unit) As shown in Figure 1, the vision unit 300 and the image processing unit 400 are mounted on the cart 210. The image processing unit 400 processes the images captured by the endoscope 3. A display unit 220 is located on the cart 210. The display unit 220 displays the images captured by the endoscope 3.
[0056] (Control System Configuration) As shown in Figure 15, the surgical support system 500 comprises a first control device 310, an arm control unit 320, a positioner control unit 330, an operation control unit 340, and a second control device 350. The surgical support system 500 also comprises a storage unit 311 connected to the first control device 310 and a storage unit 351 connected to the second control device 350.
[0057] The first control device 310 is positioned inside the medical trolley 10 to communicate with the arm control unit 320 and the positioner control unit 330, and controls the entire surgical support system 500. Specifically, the first control device 310 communicates with the arm control unit 320, the positioner control unit 330, and the operation control unit 340, and controls these units. The first control device 310 is connected to the arm control unit 320, the positioner control unit 330, and the operation control unit 340 by a LAN or the like. The first control device 310 is located inside the medical trolley 10.
[0058] An arm control unit 320 is located for each of the multiple robot arms 50. In other words, multiple arm control units 320 corresponding to the number of robot arms 50 are located inside the medical trolley 10.
[0059] As shown in Figure 15, the input device 22 is connected to the first control device 310 by LAN or the like. The status indicator 41, arm status indicator 42, operating handle 23, throttle 23a and joystick 22b and the positioner control unit 330 are connected by serial communication via a communication network that allows them to share information with each other, through wiring 360. In Figure 15, it is shown that all of the status indicator 41, arm status indicator 42, etc. are connected to a single wiring 360, but in reality, there is a separate wiring 360 for each of the status indicator 41, arm status indicator 42, operating handle 23, throttle 23a, joystick 22b and electric cylinder 25.
[0060] As shown in Figure 16, the arm section 51 is equipped with multiple servo motors SM1, encoders EN1, and a reduction gear, each corresponding to a joint JT1, JT2, JT3, JT4, JT5, JT6, and JT7. The encoder EN1 detects the rotation angle of the servo motor SM1. The reduction gear reduces the rotation of the servo motor SM1 to increase the torque. Inside the medical trolley 10, a servo control unit SC1 for controlling the servo motor SM1 is located adjacent to the arm control unit 320. The encoder EN1 for detecting the rotation angle of the servo motor SM1 is electrically connected to the servo control unit SC1.
[0061] The second link section 53 is equipped with a servo motor SM2, an encoder EN2, and a reduction gear for rotating a driven member located on the driven unit 2a of the surgical instrument 1. The encoder EN2 detects the rotation angle of the servo motor SM2. The reduction gear reduces the rotation of the servo motor SM2 to increase the torque. The medical trolley 10 is also equipped with a servo control unit SC2 for controlling the servo motor SM2 that drives the surgical instrument 1. The encoder EN2 for detecting the rotation angle of the servo motor SM2 is electrically connected to the servo control unit SC2. Note that multiple servo motors SM2, encoders EN2, and servo control units SC2 are provided.
[0062] The translational movement mechanism 54 is equipped with a servo motor SM3 for translating the surgical instrument 1, an encoder EN3, and a reduction gear. The encoder EN3 detects the rotation angle of the servo motor SM3. The reduction gear reduces the rotation of the servo motor SM3 to increase the torque. The medical trolley 10 also has a servo control unit SC3 for controlling the servo motor SM3 that translates the surgical instrument 1. The encoder EN3 for detecting the rotation angle of the servo motor SM3 is electrically connected to the servo control unit SC3. The encoders EN1, EN2, and EN3 are, for example, magnetic, mechanical, or electromagnetic induction type. The high resolution of the encoders EN1, EN2, and EN3 themselves improves the positional accuracy of the tip of the robot arm 50 and the tip of the surgical instrument 1. Furthermore, by reducing the rotational speed transmitted from the servo motors SM1, SM2, and SM3 using a reduction gear, the accuracy of the rotational angle of the joints of the robot arm 50, which is the output shaft, is further improved.
[0063] The first control device 310 generates command values that command the positions (rotation angles of each servo motor) of servo motors SM1, SM2, and SM3 based on the operation received by the remote control device 200, and drives the servo motors SM1, SM2, and SM3 based on the command values. The first control device 310 then detects a deviation anomaly error if the difference between the command values and the positions of servo motors SM1, SM2, and SM3 detected by the sensors exceeds an acceptable range.
[0064] As shown in Figure 17, the positioner 30 is equipped with multiple servo motors SM4, an encoder EN4, and a reduction gear, corresponding to the multiple joints 33 of the positioner 30. The encoder EN4 is configured to detect the rotation angle of the servo motors SM4. The reduction gear is configured to reduce the rotation of the servo motors SM4 and increase the torque.
[0065] The medical trolley 10 is equipped with wheels, including front wheels that serve as drive wheels and rear wheels that are steered by an operating handle 23. The rear wheels are positioned closer to the operating handle 23 than the front wheels. The medical trolley 10 is also equipped with servo motors SM5, encoders EN5, reduction gears, and brakes BRK, each of which drives one of the multiple front wheels of the medical trolley 10. The reduction gear is configured to reduce the rotation of the servo motors SM5 and increase the torque. A potentiometer P1, as shown in Figure 3, is positioned on the operating handle 23, and the servo motors SM5 of the front wheels are driven based on the rotation angle detected by the potentiometer P1 in response to the twisting of the throttle 23a. The rear wheels of the medical trolley 10 are of the twin-wheel type, and the rear wheels are steered based on the left and right rotation of the operating handle 23. Furthermore, a potentiometer P2, as shown in Figure 3, is positioned on the rotation axis of the operating handle 23, and a servo motor SM6, encoder EN6, and reduction gear are positioned on the rear wheels of the medical trolley 10. The reduction gear is configured to reduce the rotation of the servo motor SM6 and increase the torque. The servo motor SM6 is driven based on the rotation angle detected by the potentiometer P2 in response to the left and right rotation of the operating handle 23. In other words, the steering of the rear wheels by the left and right rotation of the operating handle 23 is power-assisted by the servo motor SM6.
[0066] The medical trolley 10 moves in the forward and backward direction when its front wheels are driven. In addition, when the operating handle 23 is rotated, the rear wheels are steered, causing the medical trolley 10 to rotate in the left and right direction.
[0067] As shown in Figure 17, the medical trolley 10 is equipped with a servo control unit SC4 for controlling the servo motor SM4 that moves the positioner 30. An encoder EN4 for detecting the rotation angle of the servo motor SM4 is electrically connected to the servo control unit SC4. The medical trolley 10 is also equipped with a servo control unit SC5 for controlling the servo motor SM5 that drives the front wheels of the medical trolley 10. An encoder EN5 for detecting the rotation angle of the servo motor SM5 is electrically connected to the servo control unit SC5. The medical trolley 10 is also equipped with a servo control unit SC6 for controlling the servo motor SM6 that power assists the steering of the rear wheels of the medical trolley 10. An encoder EN6 for detecting the rotation angle of the servo motor SM6 is electrically connected to the servo control unit SC6.
[0068] As shown in Figures 16 and 17, each of the joints JT1, JT2, JT3, JT4, JT5, JT6, and JT7 of the arm section 51 and the joint 33 of the positioner 30 are equipped with brakes. Brakes are also equipped on the front wheels of the medical trolley 10, the arm base 40, and the translational movement mechanism 54. Control signals are transmitted unidirectionally from the arm control unit 320 to each of the brakes mounted on the joints JT1, JT2, JT3, JT4, JT5, JT6, and JT7 of the arm section 51 and the translational movement mechanism 54. The control signals are signals to turn the brakes on and off. The signal to turn on the brakes includes a signal to maintain the state in which the brakes are engaged. The same applies to the control signals from the positioner control unit 330 to each of the brakes mounted on the joint 33 of the positioner 30 and the arm base 40. At startup, all brakes (BRK) on the arm base 40, arm section 51, and translational movement mechanism 54 are released, and the servo motor SM is driven to counteract gravity, thereby maintaining the posture of the robot arm 50 and the arm base 40. When an error occurs in the surgical support system 500, the brakes (BRK) on the arm base 40, arm section 51, and translational movement mechanism 54 are turned on. When the error in the surgical support system 500 is cleared, the brakes (BRK) on the arm base 40, arm section 51, and translational movement mechanism 54 are turned off. The brakes (BRK) on the arm base 40, arm section 51, and translational movement mechanism 54 are turned on when the surgical support system 500 is shut down. In addition, the front wheels of the medical trolley 10 always have the brakes (BRK) turned on, and the brakes (BRK) are released only while the enable switch 23b is pressed. Furthermore, each joint 33 of the positioner 30 has the brake BRK always on, and the brake BRK is released only while the enable switch 22c is pressed.
[0069] As shown in Figure 18, servo motors SM7a, SM7b, SM7c, SM7d, SM7e, SM7f, and SM7g are arranged on the joints JT21, JT22, JT23, JT24, JT25, JT26, and JT27 of the operating section 110, respectively. Servo motor SM7a rotates the link section 111a around the A21 axis. Servo motor SM7b rotates the link section 111b around the A22 axis. Servo motor SM7c rotates the link section 111c around the A23 axis. Servo motor SM7d rotates the link section 112a around the A24 axis. Servo motor SM7e rotates the link section 112b around the A25 axis. Servo motor SM7f rotates the link section 112c around the A26 axis. Furthermore, the servo motor SM7g rotates the grip support member 112d around the A27 axis. Servo control units SC7a, SC7b, SC7c, SC7d, SC7e, SC7f, and SC7g are also provided to control each servo motor. Each servo control unit is electrically connected to encoders EN7a, EN7b, EN7c, EN7d, EN7e, EN7f, and EN7g for detecting the rotation angle of each servo motor. Each servo motor, each servo control unit, and each encoder are provided in the operation unit 110L and the operation unit 110R, respectively. The servo motors SM7a, SM7b, SM7c, SM7d, SM7e, SM7f, and SM7g move the respective link sections of the operation unit 110. Note that servo motors SM7a, SM7b, SM7c, SM7d, SM7e, SM7f, and SM7g are examples of drive units.
[0070] The first control device 310, via the operation control unit 340, controls each servo motor to generate a torque that counteracts the gravitational torque generated on the rotation axis of each servo motor, according to the posture of the operation unit 110. This makes it possible for the operator to operate the operation unit 110 with relatively little force.
[0071] Furthermore, when the operator rotates the grip support member 112d around the A27 axis of the operating section 110 shown in Figures 12 and 13, the shaft 2d of the instrument 2 rotates around the A9 axis shown in Figure 5. Also, when the operator rotates the joints JT24, JT25, and JT26 of the operating section 110 shown in Figures 12 and 13, the end effector 2b bends around the A10 axis or A11 axis shown in Figure 5.
[0072] The operation control unit 340 is located in the main body of the remote control device 200. The operation control unit 340 controls the operation unit 110. As shown in Figure 15, the operation control unit 340 is positioned to correspond to the left-hand operation unit 110L and the right-hand operation unit 110R, respectively.
[0073] As shown in Figure 15, the vision unit 300 and the image processing unit 400 are connected to the first control device 310 via a LAN or the like. The display unit 220 is connected to the vision unit 300.
[0074] (Control operation of the first control device) The control of the first control device 310 when the operation unit 110 receives an operation from the operator will be described below. The following control is performed by a functional block of the first control device 310 that operates based on the program 311a stored in the storage unit 311.
[0075] As shown in Figure 19, when the operator operates the operating unit 110, the axis values of the operating unit 110 are input to the first control device 310. The first control device 310 calculates the position and orientation of the operating unit 110 and the opening angle of the pair of grip members 112e from the axis values of the operating unit 110. In Figure 19, the operating unit 110 is denoted as HC. The position and orientation refer to the coordinate values in the TCP coordinate system, and the opening angle of the pair of grip members 112e refers to the opening angle of the jaw member 2g and jaw member 2h. TCP is an abbreviation for Tool Center Point. TCP is set, for example, at the tip of the instrument 2. The first control device 310 also calculates the difference between the current position and orientation of the operating unit 110 and the previous position and orientation of the operating unit 110, and calculates the amount of operation of the operating unit 110 based on the difference. Furthermore, the position and orientation of the endoscope 3 are input to the first control device 310.
[0076] The first control device 310 converts the operating amount of the control unit 110 into the coordinate system of the endoscope 3 based on the input position and orientation of the endoscope 3. Then, the first control device 310 calculates the operating amount of the instrument 2 based on the operating amount of the control unit 110 converted into the coordinate system of the endoscope 3. The first control device 310 also calculates the sum of the current operating amount of the instrument 2 and the previous operating amount of the instrument 2, and calculates the position and orientation of the instrument 2 based on the sum. Note that "this time" means the current control cycle, and "previous time" means the previous control cycle.
[0077] Furthermore, the first control device 310 performs a process to limit the axis speed of the robot arm 50 based on the calculated position and orientation of the instrument 2. The maximum value of the axis speed for each axis is set in advance, and if the axis speed of the robot arm 50 exceeds the maximum value, the first control device 310 limits the axis speed so that the axis speed becomes the maximum value. The functional block of the first control device 310 that limits the axis speed is called the limiting unit.
[0078] The first control device 310 performs inverse kinematics calculations for the limited velocities of each axis. The first control device 310 calculates command values for each axis of the robot arm 50 based on the inverse kinematics calculations. The first control device 310 then performs a process to apply acceleration limits to the calculated command values for each axis. Note that acceleration includes acceleration in the direction of increasing velocity and acceleration in the direction of decreasing velocity. The maximum value of the acceleration for each axis is set in advance, and if the acceleration of the robot arm 50 exceeds the maximum value, the first control device 310 limits the acceleration so that the acceleration becomes the maximum value. Then, it outputs command values to the servo control unit SC1 that drives the robot arm 50 and the servo control unit SC2 that drives the instrument 2, so that the servo motor rotates while the velocity and acceleration are limited. Furthermore, the signals that indicate the position, orientation, and opening angles of the jaw members 2g and 2h included in each of the instrument 2, which are generated based on the operation of the operation unit 110 of the remote control device 200 by the operator, will be referred to as manual operation commands below. The signals indicating the position, orientation, and opening angles of the jaw members 2g and 2h included in each of the instrument 2, inferred by the inference unit 613 of the inference control device 600 (described later), will be referred to below as inference operation commands.
[0079] (Inference Control Device) As shown in Figure 1, the inference control device 600 is located separately from the surgical support robot 100, the remote control device 200, the vision unit 300, and the image processing unit 400. The inference control device 600 is, for example, a PC (personal computer). As shown in Figure 15, the inference control device 600 is connected to the first control device 310 by a LAN or the like. As shown in Figure 20, the inference control device 600 includes a processing unit 610 such as a CPU (Central Processing Unit). The inference control device 600 also includes a storage unit 620. The storage unit 620 includes, for example, ROM (Read Only Memory), RAM (Random Access Memory), and HDD (hard disk drive). Furthermore, the inference control device 600 includes an input unit 630. The input unit 630 is, for example, a keyboard. Here, the inference control device 600 may be located inside a surgical support robot 100, a remote control device 200, a vision unit 300, or an image processing unit 400 as a device having a processing unit 610 and a storage unit 620.
[0080] In this embodiment, the processing unit 610 includes a data collector 611, a learner 612, and an inference unit 613. The data collector 611, learner 612, and inference unit 613 are functional blocks executed by the CPU of the processing unit 610 based on a program 621 stored in the storage unit 620. The inference unit 613 is an example of a learning model.
[0081] The data acquisition unit 611 receives images G captured by the endoscope 3 attached to the robot arm 50c. As shown in Figure 21, images G include the surgical site of the patient P and the tip of the instrument 2. The data acquisition unit 611 collects and stores the images G captured by the endoscope 3 in chronological order. The data acquisition unit 611 also collects and stores manual operation commands and inference operation commands processed by the first control device 310 in chronological order. If the sampling periods of the images G captured by the endoscope 3 and the manual operation commands and inference operation commands are different, the data acquisition unit 611 adjusts for the time difference between the images G captured by the endoscope 3 and the manual operation commands and inference operation commands. The data acquired by the data acquisition unit 611 is stored, for example, in the data acquisition unit 611 itself or in the storage unit 620.
[0082] The learner 612 generates an inference 613, which is a trained model that has been trained by machine learning, using observational information of the environment surrounding multiple robot arms 50, each operated independently based on motion commands, and motion commands as input. Specifically, the learner 612 generates an inference 613, which is a trained model, by machine learning, using images G captured by the endoscope 3 collected by the data collector 611 and manual motion commands as training data. The machine learning is, for example, deep learning.
[0083] In this embodiment, the manual operation commands learned by the learner 612 include suturing operation commands for suturing the surgical site of patient P. Specifically, the suturing operation commands are manual operation commands generated based on the operator's operation of the control unit 110 to suture the surgical site of patient P. The image G learned by the learner 612 is an image G captured by the endoscope 3 when suturing is performed on the surgical site of patient P. As shown in Figure 21, this image G shows the surgical site of patient P, the jaw members 2g and 2h at the tip of the instrument 2, the suture needle 641 held by the jaw members 2g and 2h, and the thread 642 threaded through the suture needle 641. In other words, the learner 612 learns the suturing operation of the surgical site of patient P and generates an inferencer 613 for inferring the suturing operation of the surgical site of patient P.
[0084] In this embodiment, the data acquisition and storage operation in which the data acquisition unit 611 acquires and stores images G captured by the endoscope 3 and manual operation commands as training data for generating the inference unit 613 is initiated and stopped, for example, by the coagulation pedal 126R. The coagulation pedal 126R is located separately from the coagulation pedal 126L, which switches between autonomous mode and remote operation mode, as described later. The coagulation pedal 126R is pressed by the operator's foot, which switches the start and stop of the data acquisition and storage operation of the data acquisition unit 611. Pressing by the foot means stepping on it with the foot.
[0085] Furthermore, when the coagulation pedal 126R is pressed, an alert sound is emitted. For example, the alert sound is emitted from the notification unit 141 of the monitor 140 shown in Figure 1 or from the speaker 22e of the input device 22.
[0086] Furthermore, the learning device 612 can learn not only the images G captured by the endoscope 3 and manual operation commands, but also the performance level of the robot arm 50 and instrument 2. The performance level includes, for example, who the operator is, the execution time of the suturing operation on the patient P's surgical site, and the score of the quality of the suturing. The score of the quality of the suturing is a score that quantifies, for example, the spacing of the sutures, deformation of the suture site, and the amount of bleeding. The performance level is input, for example, using the input unit 630 of the inference control device 600.
[0087] In this embodiment, the inference unit 613 takes observational information and operation commands of the environment surrounding the robot arm 50 and instrument 2 as input, and outputs an inferred operation command, which is the inferred operation command. As shown in Figure 19, the inference control device 600 receives, for example, an image G captured by the endoscope 3, the position and orientation of the tip of the endoscope 3, the position and orientation of the left and right instrument 2, and the opening angles of the jaw members 2g and 2h included in each of the left and right instrument 2 as input. The left and right instrument 2 refers to the two robot arms 50 selected by the operator as the target of operation from among the robot arms 50a, 50b, and 50d to which the instrument 2 is attached. The operator switches the robot arm 50 to be operated by the operation unit 110 by operating the switching pedal 122. The inference control device 600 also receives information on whether it is in autonomous mode or remote control mode. This information may be stored in the storage unit 620, storage unit 311, storage unit 351, or on the cloud. The information indicating whether the system is in autonomous mode or remote control mode refers to whether the coagulation pedal 126L is being pressed by the operator, as will be described later. The inference control device 600 also receives information on the start or stop of the collection and storage operation, which collects and stores images G captured by the endoscope 3 and manual operation commands as training data for generating the inference unit 613. This information may also be stored in the storage unit 620, storage unit 311, storage unit 351, or on the cloud. This information may also be stored for each robot arm 50.
[0088] In this embodiment, the inference unit 613 takes observation information and operation commands from multiple past points in time stored in the data collector 611 or the storage unit 620 as input and outputs inference operation commands for the next and subsequent times. The next time may be immediately after the current time, or a little while after the current time. The next inference operation command may be an inference operation command for one step in the next time, or an inference operation command for several steps after the next time. For example, the inference unit 613 receives images G captured by the endoscope 3 for several steps of the control cycle of the surgical support robot 100 as observation information. The inference unit 613 also receives manual operation commands, inference operation commands, or operation commands that are a mixture of manual operation commands and inference operation commands for several steps of the control cycle of the surgical support robot 100 as input. The images G, manual operation commands, inference operation commands, etc., are input to the inference control device 600 via the input port 640. The inference unit 613 outputs inference operation commands based on this input information.
[0089] In addition, additional information related to surgery may be included as input information to the inference unit 613 in autonomous mode. This additional information may include, for example, information specifying a surgical procedure or a specific step in a particular surgical procedure, or information specifying a particular technique such as suturing, or information on steps that further subdivide that technique. The input information may be, for example, voice instructions, input via an interface other than voice instructions such as a touch panel, input via a hard switch, or input via the operator's gestures. Furthermore, if, for example, a suturing technique is input as additional information related to surgery, the inference unit 613 can further subdivide the suturing technique into lower-level steps for inference. Lower-level steps refer to individual actions such as, for example, holding the suture needle 641, passing the suture needle 641 through the suture site of patient P, and tying the thread 642. It is also possible to directly input information on these lower-level steps via voice, an interface, or a hard switch.
[0090] Furthermore, in this embodiment, the duration of the image G and the duration of the operation command input to the inference unit 613 may be set. For example, by operating the touch panel 130 of the remote control device 200, the duration of the image G and the duration of the operation command input to the inference unit 613 can be set to 5 steps, 10 steps, 15 steps, etc. of the control cycle. Note that the touch panel 130 is an example of a second setting unit.
[0091] Furthermore, in this embodiment, in addition to the image G and operation commands, the performance level of the instrument 2's operation may be input to the inference unit 613, and the inference unit 613 may be trained according to the performance level. Then, the performance level of the inference unit 613 to be used for inference may be selected. In other words, multiple inference units 613 may be prepared according to their performance levels. Then, by operating the touch panel 130 of the remote control device 200, the performance level of the inference unit 613 is selected, and inference is performed by the selected inference unit 613. Note that the touch panel 130 is an example of a third setting unit.
[0092] In this embodiment, each set of multiple robot arms 50 and instruments 2 is switched between an autonomous mode, which operates based on inference operation commands output from the inference unit 613, and a remote operation mode, which operates based on manual operation commands from the remote control device 200. In this embodiment, all sets of robot arms 50a, 50b, and 50d, and the instruments 2 attached to them, are switched between autonomous mode and remote operation mode. It is also possible to set some of the sets of multiple robot arms 50 and instruments 2 to autonomous mode and others to remote operation mode. Furthermore, the robot arm 50c to which the endoscope 3 is attached may also be switched between autonomous mode and remote operation mode.
[0093] In this embodiment, in autonomous mode, the operation command and the image G acquired by the endoscope 3 are input to the inference unit 613, and the inference unit 613 outputs an inference operation command, and the robot arm 50 and instrument 2 operate based on the inference operation command. The inference operation command includes, for example, the position and orientation of the left and right instrument 2, and the opening angles of the jaw members 2g and 2h included in each of the left and right instrument 2. Here, as shown in Figure 19, the first control device 310 performs a process to limit the axial speed and axial acceleration of the robot arm 50 and instrument 2 as described above. The reason for limiting the axial speed and axial acceleration is to use mechanical elements such as servo motors and reducers within the allowable specification range. Specifically, whether in remote operation mode or autonomous mode, the mechanisms of the operated robot arm 50 and instrument 2 are common, so it is necessary to use them below the limiting conditions imposed by the strength and durability of mechanical elements such as servo motors and reducers. Furthermore, when multiple robot arm 50 and instrument 2 sets include both sets operating in remote control mode and sets operating in autonomous mode, the characteristics of the command values may differ between the remote control mode and the autonomous mode. Therefore, it is important to consider how to impose common restrictions. The inferred motion command replaces the manual motion command before the calculation of the position and orientation of the instrument 2, and before the process of applying the axis velocity. In other words, in autonomous mode, the manual motion command is replaced with an inferred motion command before the first control device 310 restricts the manual motion command.
[0094] As described above, the switching pedal 122 switches between the robot arms 50 operated by the left and right control units 110. For example, the switching pedal 122 selects two robot arms 50 to be operated by the left and right control units 110 from among the robot arms 50a, 50b, and 50d to which the instrument 2 is attached. Here, when the mode is switched, if both the left robot arm 50 and instrument 2 set and the right robot arm 50 and instrument 2 set are switched modes, the autonomous mode and remote control mode are switched by the solidification pedal 126L. The solidification pedal 126L is operated by the operator's foot. The autonomous mode is executed while the operator is pressing down on the solidification pedal 126L with their foot. The remote control mode is executed while the operator is releasing the pressure on the solidification pedal 126L. Pressing down with the foot means that the operator is pressing down on the solidification pedal 126L. Releasing the pressure means that the operator has stopped pressing down on the coagulation pedal 126L. On the other hand, when switching modes for each pair of left and right feet, two switching units are required: one for switching the right mode and another for switching the left mode. For example, in addition to the coagulation pedal 126L, another foot pedal 120 may be assigned as a switching unit.
[0095] In this embodiment, the inference unit 613 continues to infer inference operation commands even in remote control mode, using manual operation commands in remote control mode as input. The autonomous mode is started when the operator presses down on the solidification pedal 126L. In autonomous mode, the robot arm 50 and instrument 2 operate based on the inference operation commands output from the inference unit 613. The remote control mode is started when the operator releases the pressure on the solidification pedal 126L. In remote control mode, manual operation commands are generated when the operator operates the control unit 110, and the robot arm 50 and instrument 2 operate based on the manual operation commands. Here, even after switching from autonomous mode to remote control mode, the inference unit 613 continues to infer inference operation commands. Specifically, at the initial point after switching from autonomous mode to remote control mode, the inference operation commands and manual operation commands are included in several steps of the past control cycle, so the inference unit 613 infers inference operation commands using past observation information, past inference operation commands, and past manual operation commands as input. After some time has passed since switching from autonomous mode to remote control mode, the past few steps of the control cycle will only contain past manual operation commands. Therefore, the inference unit 613 uses past observation information and past manual operation commands as input to infer an inference operation command.
[0096] Furthermore, in this embodiment, as shown in Figure 22, the monitor 140 of the remote control device 200 and the display unit 220 located on the cart 210 display operation information of the instrument 2 based on inferred operation commands from the present moment onward. In this embodiment, the monitor 140 of the remote control device 200 and the display unit 220 located on the cart 210 display the current operation position of the instrument 2 as operation information, and the operation position of the instrument 2 based on inferred operation commands from the present moment onward. Note that the current operation position of the instrument 2 as operation information may not be displayed, and only the operation position of the instrument 2 based on inferred operation commands from the present moment onward may be displayed. Also, "present moment" is a concept that includes not only the present moment, but also several steps of the control cycle before and after the present moment.
[0097] For example, the TCP coordinate system CS as operational information is displayed on the monitor 140 and the display unit 220. In Figure 22, the coordinate system CS of the instrument 2 attached to the robot arm 50 operated by the left operation unit 110 and the coordinate system CS of the instrument 2 attached to the robot arm 50 operated by the right operation unit 110 are displayed, respectively. The coordinate system CS is the coordinate system that serves as the operational reference for the robot arm 50 and the instrument 2, and is, for example, the coordinate system of the endoscope 3. In the coordinate system CS, the X, Y, and Z axes are displayed in different colors. In addition, the current TCP coordinate system CS and the TCP coordinate system CS from the present time onward are displayed on the monitor 140 and the display unit 220. In Figure 22, the current TCP coordinate system CS is displayed as a solid line, and the TCP coordinate system CS from the present time onward is displayed as a dotted line.
[0098] Furthermore, in this embodiment, a time point after the current time may be set. For example, a time point after the current time can be set by the operator operating the touch panel 130 of the remote control device 200. For example, a time point after the current time can be set to 1 second later, 5 seconds later, 10 seconds later, etc. If a time point after the current time is set to 1 second later, the coordinate system CS of the TCP after the current time is the coordinate system CS of the TCP 1 second later inferred by the inference unit 613. The touch panel 130 is an example of a first setting unit.
[0099] Furthermore, the monitor 140 and the display unit 220 display the opening angle OA as operational information for the jaw members 2g and 2h of the instrument 2 at the present time, and the opening angle OA as operational information for the jaw members 2g and 2h of the instrument 2 from the present time onward. In Figure 22, the current opening angle OA is shown by a solid line, and the opening angle OA from the present time onward is shown by a dotted line. The opening angle OA from the present time onward is, for example, the opening angle OA one second later inferred by the inference unit 613. The opening angle OA is also displayed, for example, by a diagram that simulates a pair of jaw members. In addition, the monitor 140 and the display unit 220 may also display the orientation of the instrument 2 as operational information.
[0100] Furthermore, the monitor 140 and the display unit 220 display operation information of the instrument 2 based on inference operation commands from the present time onward, at least in remote control mode. In this embodiment, the monitor 140 and the display unit 220 display operation information of the instrument 2 based on inference operation commands from the present time onward, and operation information of the instrument 2 based on inference operation commands from the present time onward, in both remote control mode and autonomous mode.
[0101] Furthermore, in this embodiment, the monitor 140 of the remote control device 200 and the display unit 220 located on the cart 210 superimpose the current operating position of the instrument 2 and the operating position of the instrument 2 based on the inferred operation command onto the image G captured by the endoscope 3. In Figure 22, the image G of the suture position of patient P, captured by the endoscope 3, is displayed on the monitor 140 and the display unit 220. The coordinate system CS and opening angle OA, representing the operating position of the instrument 2, are superimposed on the image G of the suture position of patient P.
[0102] In this embodiment, the monitor 140 of the remote control device 200 and the display unit 220 located on the cart 210 display an indicator M indicating whether the current mode is autonomous mode or remote control mode. For example, as shown in Figure 22, the monitor 140 and the display unit 220 display the characters "Mode: Dr" as indicator M when the current mode is remote control mode. The monitor 140 and the display unit 220 also display the characters "Mode: AI" as indicator M when the current mode is autonomous mode. When both the left robot arm 50 and instrument 2 set and the right robot arm 50 and instrument 2 set can switch modes, one indicator M common to both the left and right sets is displayed. On the other hand, when the modes can be switched for each of the left and right sets, an indicator M is displayed for each of the left and right sets.
[0103] (Operation of the control unit in autonomous mode) In this embodiment, as described above, the control unit 110 includes link parts 111a, 111b, 111c, 112a, 112b, 112c and a grip support member 112d. In the following description, the grip support member 112d is also referred to as a link part. The control unit 110 also includes servo motors SM7a to SM7g that drive the above-mentioned plurality of link parts. By changing the posture of the plurality of link parts, the posture of the robot arm 50 and instrument 2 is changed. In autonomous mode, the posture of the plurality of link parts is changed by servo motors SM7a to SM7g to follow the posture of the robot arm 50 and instrument 2. That is, the plurality of link parts of the control unit 110 rotate automatically and the posture of the plurality of link parts is changed to correspond to the posture of the robot arm 50 and instrument 2 in autonomous mode. In autonomous mode, link sections 111a, 111b, and 111c may not follow the posture of the robot arm 50 and instrument 2, while only link sections 112a, 112b, 112c, and grip support member 112d may follow.
[0104] (Protection function in autonomous mode) In this embodiment, as shown in Figure 19, the first control device 310 monitors the operation of the robot arm 50 and instrument 2 in autonomous mode. For example, the first control device 310 receives inference operation commands from the inference control device 600, and the first control device 310 monitors the inference operation commands. When the first control device 310 determines that the operation of the robot arm 50 and instrument 2 in autonomous mode satisfies predetermined conditions, it switches to remote control mode. Switching to remote control mode means that the inference operation commands are replaced with manual operation commands. The functional block of the first control device 310 that monitors the operation of the robot arm 50 and instrument 2 in autonomous mode is called the monitoring unit. That is, the monitoring unit as a functional block of the first control device 310 is located between the inference unit 613 and the limiting unit as a functional block of the first control device 310.
[0105] Furthermore, if the first control device 310 determines that predetermined conditions are met, an alert sound is emitted from, for example, the notification unit 141 of the monitor 140 shown in Figure 1 or the speaker 22e of the input device 22. Note that this alert sound is different from the alert sound emitted when the coagulation pedal 126R is pressed down for the purpose of collecting the training data mentioned above.
[0106] In this embodiment, the above-mentioned predetermined conditions include the case where the robot arm 50 and instrument 2 are not in a state where they can be operated by the remote control device 200. Operating the robot arm 50 and instrument 2 by the remote control device 200 is called following operation. If either the left or right robot arm 50 is not capable of following operation, the autonomous mode is switched to remote control mode. Following operation becomes possible after a matching grip is performed, which is an operation that synchronizes the operating unit 110 and the tip of the instrument 2.
[0107] In this embodiment, the above-mentioned predetermined conditions include cases where the robot arm 50 and instrument 2 move outside the preset monitoring range. The first control device 310 has pre-set a monitoring range as the space in which the robot arm 50 and instrument 2 can operate. Then, if the robot arm 50 and instrument 2 move outside the preset monitoring range based on the position and orientation of either the left or right robot arm 50 according to the inference operation command, the autonomous mode is switched to remote control mode.
[0108] In this embodiment, the above-mentioned predetermined conditions include cases where the speed of the robot arm 50 and instrument 2 based on the inference operation command exceeds a predetermined speed threshold. For example, if the speed of the robot arm 50 and instrument 2 based on the inference operation command exceeds a predetermined speed threshold for two consecutive cycles, or if it becomes twice or more the predetermined speed threshold in one cycle, the autonomous mode is switched to the remote control mode. Note that the above-mentioned cycle refers to the control cycle.
[0109] Next, the control method for the surgical support system 500 will be described. In the following control method for the surgical support system 500, the operations performed by the inference control device 600 are performed based on the program 621 stored in the memory unit 620, and the operations performed by the first control device 310 are performed based on the program 311a stored in the memory unit 311. Furthermore, in the following control method for the surgical support system 500, the case in which the left robot arm 50 and instrument 2 set maintains its current mode, and the mode of the right robot arm 50 set is switched by the coagulation pedal 126L will be described. Note that the control method for the surgical support system 500 is an example of a control method for a manipulation system.
[0110] As shown in Figure 23, in step S1, in remote operation mode where the operator does not press the coagulation pedal 126L, the operation unit 110 accepts operations from the operator to prepare for suturing. For example, the operator makes the suture needle 641, through which the thread 642 is threaded, grasped by the jaw members 2g and 2h of the right robot arm 50, and moves the instruments 2 attached to the left and right robot arms 50 to the vicinity of the suturing position of the patient P. The monitor 140 and the display unit 220 display an indicator M indicating the current mode.
[0111] In step S2, the data collector 611 collects manual operation commands output to the servo control unit SC2, and the inference unit 613 starts inferring inference operation commands based on the collected manual operation commands. The monitor 140 and the display unit 220 display the TCP coordinate system CS and the opening angles OA of the jaw members 2g and 2h of the instrument 2, both currently and from this point in time.
[0112] In step S3, the processing unit 610 of the inference control device 600 determines whether or not the coagulation pedal 126L has been pressed. The operation of step S3 is repeated until it is determined that the coagulation pedal 126L has been pressed. The information on whether or not the coagulation pedal 126L has been pressed is transmitted from the first control device 310 to the inference control device 600.
[0113] In step S3, if the processing unit 610 of the inference control device 600 determines that the coagulation pedal 126L has been pressed, in step S4 the remote control mode is switched to autonomous mode. Specifically, the left robot arm 50 and instrument 2 set maintains its current mode, and the mode of the right robot arm 50 and instrument 2 set is switched to autonomous mode. Then, as shown in Figure 24, in step S41 the operation command for a control cycle of approximately 1 second and the image G captured by the endoscope 3 are collected by the data collector 611. The image G from the endoscope 3 is based on the Full HD standard, and the resolution is 1920 x 1080 pixels. Next, in step S42 the inference unit 613 infers an inference operation command based on the collected operation command and image G. Initially, after switching from remote control mode to autonomous mode, the inference unit 613 infers inference operation commands for approximately one second following the operation commands of the past control cycle, which are included in the past few steps of the control cycle. After some time has passed since switching from remote control mode to autonomous mode, the inference unit 613 infers inference operation commands for approximately one second following the operation commands of the past few steps of the control cycle, which are included in the past observation information and which are included in the past inference operation commands. Furthermore, the image G of the endoscope 3 used by the inference unit 613 for inferring inference operation commands has been reduced in resolution to 256 x 128 pixels. In addition, the inference operation commands inferred by the inference unit 613 infer the inference operation commands for approximately one second following the operation commands of approximately one second of the control cycle collected by the data collector 611. Furthermore, the time required for the inference unit 613 to infer the inference operation command is approximately 0.1 seconds, which is shorter than the approximately 1 second duration of the inference operation command that the inference unit 613 infers. Also, in step S42, the data collector 611 collects the operation command and the image G captured by the endoscope 3 in parallel with the inference unit 613 performing the inference. In step S43, the inference control device 600 outputs the inferred inference operation command to the first control device 310.As a result, the robot arm 50 and instrument 2 operate autonomously, and the posture of the operating unit 110 is also changed to follow the changes in the posture of the robot arm 50 and instrument 2.
[0114] In addition, during suturing in autonomous mode, for example, as shown in Figure 25, the suture needle 641 held by the right instrument 2 is passed through the patient P's body, as shown in Figure 26, the suture needle 641 is then transferred to the left instrument 2, the left instrument 2 pulls the suture needle 641 and thread 642, and as shown in Figure 27, the thread 642 is tied by the left and right instruments 2.
[0115] In step S5, the processing unit 610 of the inference control device 600 determines whether the solidification pedal 126L has been released. If it is determined in step S5 that the solidification pedal 126L has been released, in step S6 the mode of the right robot arm 50 and instrument 2 set is switched from autonomous mode to remote control mode. Even after switching from autonomous mode to remote control mode, the inference unit 613 continues to infer inference operation commands.
[0116] In step S5, if the processing unit 610 of the inference control device 600 determines that the solidification pedal 126L has not been released, in step S7, the first control device 310 determines whether the operation of the right robot arm 50 and instrument 2 in autonomous mode satisfies predetermined conditions. The predetermined conditions are, as described above, when the right robot arm 50 and instrument 2 are not in a state that can be operated by the remote control device 200, when the right robot arm 50 and instrument 2 move outside the preset monitoring range, and when the speed of the right robot arm 50 and instrument 2 based on the inference operation command exceeds a predetermined speed threshold. If at least one of these is met, the first control device 310 determines that the predetermined conditions are met. If the first control device 310 determines that the predetermined conditions are met, in step S6, the autonomous mode is switched to remote control mode and the process returns to step S3. In step S7, if the first control device 310 determines that the predetermined conditions are not met, the process returns to step S5. The operations described in steps S1 through S7 above are performed until the suturing operation is completed.
[0117] (Collection of training data) The collection of training data will now be explained. Training data is collected, for example, in remote operation mode. The following operation of the inference control device 600 to collect training data is performed based on the program 621 stored in the storage unit 620. As shown in Figure 28, in step S11, in remote operation mode, the processing unit 610 of the inference control device 600 determines whether or not the coagulation pedal 126R has been pressed. The information on whether or not the coagulation pedal 126R has been pressed is transmitted from the first control device 310 to the inference control device 600. If the processing unit 610 determines in step S11 that the coagulation pedal 126R has been pressed, in step S12, the data collector 611 starts collecting images G captured by the endoscope 3 and manual operation commands as training data to generate the inferencer 613. The operation in step S11 is repeated until the coagulation pedal 126R is pressed. In step S13, the processing unit 610 of the inference control device 600 determines whether the solidification pedal 126R has been pressed down again. If the processing unit 610 determines in step S13 that the solidification pedal 126R has been pressed down again, the collection of training data is stopped in step S14. The process then returns to step S11. The operation in step S13 is repeated until the solidification pedal 126R is pressed down again. The operations from step S11 to step S14 are performed in remote control mode. Here, if manual operation commands for both the left robot arm 50 and instrument 2 set and the right robot arm 50 and instrument 2 set are collected by the data collector 611, it is possible to start and stop the storage operation of the manual operation commands using only the solidification pedal 126R. On the other hand, when manually starting and stopping the memory operation of the operation command for each left and right pair, it is necessary to assign other foot pedals 120 to start and stop the memory operation for the right pair and the left pair, in addition to the coagulation pedal 126R.
[0118] In the example above, the solidification pedal 126L, or another foot pedal 120 different from the solidification pedal 126L, is set as the pedal for switching between autonomous mode and remote control mode. In addition, the solidification pedal 126R, or another foot pedal 120 different from the solidification pedal 126R, is set as the pedal for starting and stopping the memory operation of manual operation commands. The assignment of these pedals can be set, for example, by the touch panel 130 of the remote control device 200. Note that the touch panel 130 is an example of a setting unit.
[0119] (Example of operator operation) An example of operator operation will be explained with reference to Figure 29. At time t0, the operator refrains from pressing the coagulation pedal 126L. This puts the surgical support system 500 into remote operation mode, and the operator performs the operation to prepare for suturing by operating the control unit 110.
[0120] At time t1, the operator presses down on the coagulation pedal 126L. This causes the right robot arm 50 and instrument 2 to enter autonomous mode.
[0121] At time t2, the operator determines that the operation of the right robot arm 50 and instrument 2, which are operating in autonomous mode, is inappropriate, and the operator presses down on the coagulation pedal 126L. As a result, the surgical support system 500 enters remote control mode, but this remote control mode is maintained only while the coagulation pedal 126L is pressed down.
[0122] At time t3, the operator determines that instrument 2 has returned to an appropriate state and presses the coagulation pedal 126L. As a result, the right robot arm 50 and instrument 2 set up in autonomous mode.
[0123] At time t4, if the operator determines that the suturing operation is complete, the operator presses the coagulation pedal 126L. This puts the right-hand robot arm 50 and instrument 2 into remote control mode. The operator then operates the control unit 110 to prepare for the next suturing operation.
[0124] Furthermore, at time t1, the operator can press the coagulation pedal 126R to initiate the data collection by the data collector 611 of images G captured by the endoscope 3 and manual operation commands, which will serve as training data for generating the inference unit 613. At time t4, the operator can press the coagulation pedal 126R again to stop the data collection by the data collector 611 of images G captured by the endoscope 3 and manual operation commands. This type of training data collection is also possible. As a result, inappropriate movements of the robot arm 50 and instrument 2 before time t2, as well as movements that correct inappropriate manual movements after time t2, are also collected as training data. Consequently, the inference unit 613 learns not only appropriate movements but also inappropriate movements, enabling it to learn suturing operations more accurately.
[0125] Furthermore, a trained model other than the inference unit 613 may predict the probability of an abnormality occurring in the future based on the image G captured by the endoscope 3 and the inference operation commands inferred by the inference unit 613. If the probability of an abnormality occurring is high, the surgical support system 500 may be switched from autonomous mode to remote control mode. For example, the image G captured by the endoscope 3 may be an image G showing bleeding at the surgical site of patient P. By predicting the probability of an abnormality occurring, it is possible to prevent future abnormalities from occurring.
[0126] [Effects of this Embodiment] The surgical support system 500 is equipped with a coagulation pedal 126R for inputting the start and stop of a storage operation that stores the operator's operations as training data for generating the inference unit 613 in the storage unit 620. As a result, the surgical support system 500 itself is equipped with a coagulation pedal 126R for storing training data in the storage unit 620, so there is no need to prepare a separate surgical support system 500 for generating training data or to connect signal lines to acquire operation commands and surgical field environments from an existing surgical support system 500. Therefore, the effort required to acquire training data for generating the inference unit 613 can be suppressed. In addition, since a coagulation pedal 126L is provided to switch between autonomous mode and remote control mode, operation commands and surgical field environments can be acquired as training data from the current surgical support system 500 simply by operating the coagulation pedal 126R while in remote control mode using the coagulation pedal 126L. Therefore, it is possible to easily reacquire training data and improve the learning accuracy of the inference unit 613 through relearning.
[0127] The coagulation pedal 126L is pressed by the operator. The autonomous mode is performed while the operator is pressing the coagulation pedal 126L, and the remote control mode is performed while the operator is releasing the coagulation pedal 126L. This allows for easy switching between the autonomous mode and the remote control mode with the simple operation of pressing and releasing the coagulation pedal 126L.
[0128] The coagulation pedal 126R is pressed by the operator. Pressing the coagulation pedal 126R switches between starting and stopping the memory operation. This allows for easy switching between starting and stopping the memory operation of training data with the simple operation of pressing the coagulation pedal 126R.
[0129] The coagulation pedals 126L and 126R are pressed down by the operator's foot. This eliminates the need for the operator to take their hands off the control unit 110 to switch between autonomous mode and remote control mode, and to switch between starting and stopping memory operations, thus minimizing interruptions to the suturing operation.
[0130] Multiple sets of robot arms 50 and instruments 2 are arranged. A switching unit and an input unit are provided for each set of robot arms 50 and instruments 2. This improves the convenience of operation for the operator when switching modes and switching the input of training data.
[0131] The remote control device 200 includes a touch panel 130 for setting the assignment of switching units and input units, which are located for each set of multiple robot arms 50 and instruments 2. This allows the operator to easily change the settings by operating the touch panel 130.
[0132] The system includes a remote control device 200 which includes an operating unit 110 that receives input for the surgical instrument 1. The switching unit and input unit are located on the remote control device 200. This allows the operator to operate the switching unit and input unit while operating the operating unit 110.
[0133] [Variations] It should be noted that the embodiments disclosed herein are illustrative and not restrictive in all respects. The scope of this disclosure is indicated by the claims rather than the description of the embodiments above, and further includes all modifications or variations within the meaning and scope equivalent to the claims.
[0134] In the above embodiment, the inference unit 613 is shown to take observation information and operation commands from multiple past time points as input and output the next inference operation command, but the disclosure is not limited to this. For example, the inference unit 613 may take observation information and operation commands from one past time point as input and output the next inference operation command. This reduces the control burden on the inference control device 600.
[0135] In the above embodiment, an example was shown in which an image G captured by the endoscope 3 is input to the inference unit 613 as observation information, but the disclosure is not limited to this. For example, a camera may be placed on the ceiling or wall of the operating room, and the image G captured by this camera may be input to the inference unit 613 as observation information. This makes it possible to generate inference operation commands that take into account the environment around the patient P and the robot arm 50. Alternatively, a force sensor may be placed as an environmental information acquisition unit instead of the endoscope 3. This makes it possible to detect interference between the instrument 2 and the surgical area of the patient P using the force sensor, and thus generate inference operation commands that suppress interference.
[0136] In the above embodiment, an example was shown in which the current operating position of the instrument 2 and the operating position of the instrument 2 based on the inference operation command from the present time onward are displayed on both the monitor 140 and the display unit 220, but the disclosure is not limited to this. For example, these operating positions may be displayed only on the monitor 140. When the operator is operating the device, their head is immersed in the monitor 140, so it may be acceptable for the operating positions to be displayed only on the monitor 140.
[0137] In the above embodiment, an example was shown in which the orientation of the multiple link parts of the operating unit 110 is changed to follow the orientation of the robot arm 50 and the instrument 2 in autonomous mode, but the disclosure is not limited thereto. For example, in autonomous mode, the orientation of the multiple link parts of the operating unit 110 may remain stationary without being changed to follow the orientation of the robot arm 50 and the instrument 2. This reduces the control burden on the first control device 310.
[0138] In the above embodiment, an example was shown in which the robot arm 50 and the instrument 2 attached thereto operate in autonomous mode, but the disclosure is not limited thereto. For example, the robot arm 50 may operate in autonomous mode, and the instrument 2 may operate in remote control mode. This allows the operator to concentrate on operating the instrument 2.
[0139] In the above embodiment, an example of applying the present disclosure to a surgical support system 500 comprising a surgical support robot 100 and a remote control device 200 was shown, but the present disclosure is not limited thereto. For example, as shown in the first modified example in Figure 30, the present disclosure may be applied to a social robot 700. The social robot 700 is equipped with an interface for dialogue and is capable of communicating with people. The social robot 700 also comprises a pair of articulated robot arms 701 and a trolley 702. In other words, the social robot 700 has multiple degrees of freedom. For example, the social robot 700 normally operates in autonomous mode. For example, the switching unit 703 makes it possible to keep the pair of articulated robot arms 701 in autonomous mode and switch the trolley 702 to remote control mode. The remote control mode is activated by an operator operating the control unit 704. In this way, by switching a part of the multi-degree-of-freedom social robot 700 between autonomous mode and remote control mode, even if the social robot 700 performs an inappropriate action while operating in autonomous mode, it is possible to switch to remote control mode and correct the action.
[0140] Furthermore, as shown in the second modified example in Figure 31, the present disclosure may also be applied to an industrial robot 710 placed in a factory or the like. The industrial robot 710 includes, for example, a vertical articulated robot 711 and a horizontal articulated robot 712. For example, the industrial robot 710 normally operates in autonomous mode. The switching unit 713 makes it possible to, for example, keep the vertical articulated robot 711 in autonomous mode and switch the horizontal articulated robot 712 to remote control mode. The remote control mode is activated by an operator operating the control unit 714. Here, when there are many robots, many operators are required to operate all robots in remote control mode. Therefore, by switching some of the robots between autonomous mode and remote control mode, even when there are many robots, a small number of operators can select and operate only the robots that require remote control mode.
[0141] The above embodiment shows an example in which multiple robot arms 50 are arranged, but the disclosure is not limited thereto. For example, the disclosure may be applied to a surgical support system in which only one robot arm 50 is arranged. In this case, either the robot arm 50 or the instrument 2 can be switched between autonomous mode and remote control mode. This makes it possible to smoothly switch between autonomous mode and remote control mode even in a surgical support system in which only one robot arm 50 is arranged.
[0142] In the above embodiment, an example was shown in which the operation command included a suturing operation command for suturing the surgical site of patient P, but the present disclosure is not limited thereto. For example, the operation command may be a command other than a suturing operation command. For example, the operation command may be an excision command for excising the surgical site of patient P. Furthermore, the operation command may include multiple surgical operations such as a suturing operation command and an excision command. In this way, the present disclosure enables smooth switching between autonomous mode and remote control mode even for surgical operations other than suturing.
[0143] In the above embodiment, an example was shown where the switching unit for switching between autonomous mode and remote control mode is the coagulation pedal 126L and another foot pedal 120, but the present disclosure is not limited thereto. For example, the switching unit of the present disclosure may be a switching unit that switches between autonomous mode and remote control mode by detecting the movement of the operator's gaze. In this case, for example, a sensor that detects the movement of the operator's gaze is arranged on the monitor 140. Alternatively, the switching unit of the present disclosure may be a switching unit that switches between autonomous mode and remote control mode based on the detection of a part of the operator's body. In this case, a sensor that detects the movement of a part of the operator's body is arranged. For example, the sensor detects the movement of the operator's arm or head. By switching between autonomous mode and remote control mode with such a switching unit, the operator does not need to take their hands off the control unit to switch between autonomous mode and remote control mode, so that the operator can switch between autonomous mode and remote control mode while continuing the suturing operation. Alternatively, the switching unit of the present disclosure may be a button separately arranged on the control unit 110. This allows the operator to operate the button by hand, making it easy to switch between autonomous mode and remote control mode. Furthermore, a switching unit for switching between autonomous mode and remote control mode may be provided as a separate pedal from the solidification pedal 126L and the other foot pedals 120. Alternatively, a button may be provided instead of a pedal or sensor as the switching unit. These pedals, sensors, and buttons may also be provided for each of the multiple robot arms.
[0144] In the above embodiment, an example was shown in which, in autonomous mode, a manual operation command is replaced with an inference operation command before the limiting unit of the first control device 310 limits the speed of the manual operation command. However, the disclosure is not limited thereto. For example, in autonomous mode, a manual operation command may be replaced with an inference operation command before the limiting unit of the first control device 310 limits the acceleration of the manual operation command. This makes it possible to limit the acceleration of the operation of the surgical support system in the same way in both autonomous mode and remote control mode.
[0145] In the above embodiment, an example was shown in which, in autonomous mode, the monitoring unit of the first control device 310 determines that the operation of the robot arm 50 and instrument 2 in autonomous mode satisfies predetermined conditions and switches to remote control mode. However, this disclosure is not limited to this. For example, the monitoring unit of the first control device 310 does not need to determine predetermined conditions. This makes it possible to suppress interference with the operation of the robot arm 50 and instrument 2 in autonomous mode.
[0146] In the above embodiment, the predetermined conditions include, but are not limited to, the following: when the robot arm 50 and instrument 2 are not in a state where they can be operated by the remote control device 200; when the robot arm 50 and instrument 2 move outside a preset monitoring range; and when the speed of the robot arm 50 and instrument 2 based on the inference operation command exceeds a predetermined speed threshold. For example, the predetermined conditions may include only one or two of these conditions. This reduces the control burden on the first control device 310.
[0147] In the above embodiment, an example was shown in which the autonomous mode is executed while the operator is pressing the cutting pedal 125L and the remote control mode is executed while the operator is releasing the cutting pedal 125L, but the disclosure is not limited thereto. For example, conversely to the above operation, the remote control mode may be executed while the cutting pedal 125L is pressed, and the autonomous mode may be executed while the cutting pedal 125L is released. Alternatively, the autonomous mode and the remote control mode may be switched each time the cutting pedal 125L is pressed. In this way, even if the operation for switching between the autonomous mode and the remote control mode is different, the autonomous mode and the remote control mode can be easily switched with the simple operation of pressing the coagulation pedal 126L. Furthermore, depending on the learning proficiency of the inference unit 613, initially, the autonomous mode is executed while the operator is pressing the cutting pedal 125L, and the remote control mode is executed while the operator is releasing the cutting pedal 125L. Once the proficiency exceeds a predetermined standard, the remote control mode may be executed while the cutting pedal 125L is pressed, and the autonomous mode may be executed while the cutting pedal 125L is released.
[0148] In the above embodiment, an example was shown in which the cutting pedal 125L switches between autonomous mode and remote control mode, and the cutting pedal 125R switches between starting the teacher data storage operation and other operations. However, this disclosure is not limited to this. For example, the above switching may be performed by detecting the movement of the operator's gaze or by detecting a part of the operator's body other than their hands. This allows the above switching to be performed while the operator continues to operate the control unit 110.
[0149] In the above embodiment, an example was shown in which the current operating position of Instrument 2 and the operating position of Instrument 2 one second later based on the inference operation command from the current point in time are displayed, but the disclosure is not limited to this. In the disclosure, the operating positions of Instrument 2 at multiple points in time from the current point in time may be displayed. For example, the current operating position of Instrument 2, the operating position of Instrument 2 one second later based on the inference operation command from the current point in time, and the operating position of Instrument 2 two seconds later may be displayed. As a result, the operating positions of Instrument 2 at multiple points in time from the current point in time are displayed, so the operator can easily understand how Instrument 2 will operate in the future based on the inference operation command.
[0150] In the above embodiment, an example was shown in which the current operating position of instrument 2 and the operating position of instrument 2 based on inferred operation commands from that point onward are superimposed on the image G captured by the endoscope 3. However, this disclosure is not limited to this. For example, the current operating position of instrument 2 and the operating position of instrument 2 based on inferred operation commands from that point onward may be displayed outside the image G captured by the endoscope 3. This improves the visibility of the operating position.
[0151] In the above embodiment, an example was shown in which the current mode, whether autonomous mode or remote control mode, is displayed in text, but the disclosure is not limited thereto. For example, an alert sound corresponding to the current mode may be provided. This prevents text from being superimposed on the image G captured by the endoscope 3, thereby improving the visibility of the image G captured by the endoscope 3.
[0152] In the above embodiment, an example was shown in which the first control device 310 and the inference control device 600 are arranged separately, but the disclosure is not limited thereto. For example, the first control device 310 may also serve as the inference control device 600. This simplifies the configuration of the surgical support system 500.
[0153] Furthermore, although the above embodiment shows an example in which four robot arms 50 are provided, this disclosure is not limited to this. In this disclosure, the number of robot arms 50 may be other than four.
[0154] Furthermore, although the above embodiment shows an example in which the arm portion 51 and the positioner 30 are composed of a 7-axis articulated robot, this disclosure is not limited to this. For example, the arm portion 51 and the positioner 30 may be composed of an articulated robot with an axis configuration other than a 7-axis articulated robot. An axis configuration other than a 7-axis articulated robot would be, for example, 6 axes or 8 axes.
[0155] Furthermore, while the above embodiment shows an example in which the surgical assistance robot 100 includes a medical cart 10, a positioner 30, an arm base 40, and a robot arm 50, this disclosure is not limited to this. For example, the medical cart 10, the positioner 30, and the arm base 40 are not necessarily required, and the surgical assistance robot 100 may include only the robot arm 50.
[0156] The functions of elements such as the data collector 611, learner 612, and inference unit 613, and other control units disclosed herein may be implemented using one or more circuits or processing circuits, including general-purpose processors, special-purpose processors, integrated circuits, ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and / or conventional circuits. The functions of elements disclosed herein may be implemented using one or more circuits or processing circuits, including combinations of general-purpose processors, special-purpose processors, integrated circuits, ASICs, FPGAs, and conventional circuits. One or more circuits or processing circuits may be programmed using one or more programs stored together or individually in one or more memories, or otherwise configured to perform the disclosed functions. A processor is considered a processing circuit or circuit because it includes transistors and other circuits. A processor may be a programmed processor that executes programs stored in memory. In this disclosure, a circuit, unit, or means is hardware that performs the enumerated functions individually or in combination with each other, or hardware programmed to perform the enumerated functions individually or in combination with each other. Hardware may be any hardware disclosed herein that is programmed or configured to perform the enumerated functions. A computer program, including computer instructions, is stored in memory. Computer instructions provide logic and routines that enable the hardware to perform the methods disclosed herein. Hardware includes, for example, processing circuits or circuits. A computer program may be implemented in known formats on computer-readable storage media, computer program products, memory devices, recording media such as CD-ROMs or DVDs, and / or in the memory of FPGAs or ASICs.
[0157] [Embodiments] The exemplary embodiments described above will be understood by those skilled in the art to be specific examples of the following embodiments.
[0158] (Aspect 1) A manipulation system comprising: an operating device operated by an operator; a manipulation device that operates based on an operation command; a manual operation command generation unit that generates a manual operation command as the operation command based on the operator's operation of the operating device; a trained model that takes observational information of the environment surrounding the manipulation device and the operation command as inputs and outputs an inferred operation command as the inferred operation command; a switching unit that switches between an autonomous mode that operates based on the inferred operation command output from the trained model and a remote control mode that operates based on the manual operation command generated by the manual operation command generation unit; and an input unit that is arranged separately from the switching unit and inputs instructions to start and stop a storage operation that stores the operator's operation content as training data for generating the trained model in a storage unit.
[0159] (Aspect 2) The manipulation system according to aspect 1, wherein the switching unit includes a switching press that is pressed by the operator, the autonomous mode is performed while the operator is pressing the switching press, and the remote operation mode is performed while the operator is releasing the pressing of the switching press.
[0160] (Aspect 3) The manipulation system according to aspect 1 or aspect 2, wherein the input unit includes an input press that is pressed by the operator, and the start and stop of the memory operation are switched when the input press is pressed.
[0161] (Aspect 4) The manipulation system according to any one of aspects 1 to 3, wherein the switching unit includes a switching press that is pressed by the operator, the input unit includes an input press that is pressed by the operator, and the switching press and the input press include a pedal that is pressed by the operator's foot.
[0162] (Aspect 5) The manipulation system according to any one of aspects 1 to 4, wherein a plurality of manipulation devices are arranged, and the switching unit and the input unit are arranged for each of the plurality of manipulation devices.
[0163] (Aspect 6) The manipulation system according to aspect 5, further comprising a setting unit for setting the assignment of the switching unit and the input unit, which are arranged for each of the multiple manipulation devices.
[0164] (Aspect 7) The manipulation system according to any one of aspects 1 to 6, wherein the operating device includes an operating unit for receiving operations on surgical instruments, and the switching unit and the input unit are located in the operating device.
[0165] (Aspect 8) A method for controlling a manipulation system, comprising: generating a manual operation command as the operation command based on the operation of the operation device by the operator; switching between an autonomous mode that operates based on the inferred operation command output from a trained model that takes observation information of the environment surrounding the manipulation system and the operation command as inputs and outputs an inferred operation command as the inferred operation command, and a remote control mode that operates based on the generated manual operation command, by operating an input unit arranged separately from the switching unit, thereby starting and stopping a storage operation that stores the operator's operation content as training data for generating the trained model in a storage unit.
[0166] (Aspect 9) A program for executing a control method for a manipulation system, comprising: generating a manual operation command as the operation command based on the operator's operation of the operating device; switching between an autonomous mode, which operates based on the inferred operation command output from a trained model that takes observation information of the environment surrounding the manipulation device and the operation command as inputs and outputs an inferred operation command as the inferred operation command, and a remote control mode, which operates based on the generated manual operation command, by operating a switching unit; and starting and stopping a storage operation, which stores the operator's operation content as training data for generating the trained model, by operating an input unit arranged separately from the switching unit.
[0167] 2 Instrument (Manipulation Device) 50 Robot Arm (Manipulation Device) 126L Coagulation Pedal (Switching Unit, Switching Pressing Unit, Pedal) 126R Coagulation Pedal (Input Unit, Input Pressing Unit, Pedal) 130 Touch Panel (Setting Unit) 310 First Control Device (Manual Operation Command Generation Unit) 311a Program 500 Surgical Support System (Manipulation System) 613 Inference Unit (Trained Model) 620 Memory Unit 621 Program
Claims
1. A manipulation system comprising: an operating device operated by an operator; a manipulation device that operates based on an operation command; a manual operation command generation unit that generates a manual operation command as the operation command based on the operator's operation of the operating device; a trained model that takes observational information of the environment surrounding the manipulation device and the operation command as inputs and outputs an inferred operation command as the inferred operation command; a switching unit that switches between an autonomous mode that operates based on the inferred operation command output from the trained model and a remote control mode that operates based on the manual operation command generated by the manual operation command generation unit; and an input unit that is disposed separately from the switching unit and inputs instructions to start and stop a storage operation that stores the operator's operation content as training data for generating the trained model in a storage unit.
2. The manipulation system according to claim 1, wherein the switching unit includes a switching press that is pressed by the operator, the autonomous mode is performed while the operator is pressing the switching press, and the remote control mode is performed while the operator is releasing the press on the switching press.
3. The manipulation system according to claim 1, wherein the input unit includes an input press that is pressed by the operator, and the start and stop of the memory operation are switched when the input press is pressed.
4. The manipulation system according to claim 1, wherein the switching unit includes a switching press that is pressed by the operator, the input unit includes an input press that is pressed by the operator, and the switching press and the input press include pedals that are pressed by the operator's foot.
5. The manipulation system according to claim 1, wherein a plurality of manipulation devices are arranged, and the switching unit and the input unit are arranged for each of the plurality of manipulation devices.
6. The manipulation system according to claim 5, further comprising a setting unit for setting the assignment of the switching unit and the input unit, which are arranged for each of the multiple manipulation devices.
7. The manipulation system according to claim 1, wherein the operating device includes an operating unit for receiving operations on surgical instruments, and the switching unit and the input unit are located in the operating device.
8. A control method for a manipulation system, comprising an operating device operated by an operator and a manipulation device that operates based on an operation command, the method comprising: generating a manual operation command as the operation command based on the operator's operation of the operating device; switching between an autonomous mode that operates based on the inferred operation command output from a trained model that takes observational information of the environment surrounding the manipulation device and the operation command as inputs and outputs an inferred operation command as the inferred operation command, and a remote control mode that operates based on the generated manual operation command, by operating an input unit arranged separately from the switching unit, thereby starting and stopping a storage operation that stores the operator's operation content as training data for generating the trained model in a storage unit.
9. A program for executing a control method for a manipulation system, comprising an operating device operated by an operator and a manipulation device that operates based on an operation command, the program comprising: generating a manual operation command as the operation command based on the operator's operation of the operating device; switching between an autonomous mode, which operates based on the inferred operation command output from a trained model that takes observational information of the environment surrounding the manipulation device and the operation command as inputs and outputs an inferred operation command as the inferred operation command, and a remote control mode, which operates based on the generated manual operation command, by operating a separate input unit, which starts and stops a storage operation that stores the operator's operation content as training data for generating the trained model in a storage unit.
