Manipulation system, control method for manipulation system, and program
The manipulation system addresses risks in autonomous operation by identifying and responding to multiple risk factors, improving safety and reliability.
Patent Information
- Application Number
- PCT/JP2026/017784
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-16
- Filing Date
- 2026-05-11
- Publication Date
- 2026-11-19
AI Technical Summary
Existing manipulation systems face risks of abnormalities during autonomous operation modes, necessitating improved risk management strategies.
A manipulation system with an operating device, manipulation device, and control device that determines multiple types of risk elements and performs differentiated risk responses based on these determinations during autonomous mode operation.
Enables appropriate responses to abnormalities during autonomous mode, enhancing system safety and reliability.
Smart Images

Figure JP2026017784_19112026_PF_FP_ABST
Abstract
Description
Manipulation system, control method for manipulation system, and program
[0001] The present disclosure relates to a manipulation system, a control method for a manipulation system, and a program.
[0002] Conventionally, a manipulation system including an operation device and a manipulation device is known. Japanese Patent No. 7334499 discloses a surgery support system as a manipulation system, which includes an operation device and a surgical manipulator as the manipulation device. In this surgery support system, the surgical manipulator is operated by displacing the position and orientation of the operation device. Further, in this surgery support system, the surgical manipulator can be switched between an autonomous drive mode and a manual operation mode.
[0003] Japanese Patent No. 7334499
[0004] In the surgery support system described in Japanese Patent No. 7334499, the surgical manipulator can be switched between the autonomous drive mode and the manual operation mode. Here, when the surgical manipulator operates in the autonomous drive mode, risks such as an abnormality or a sign of an abnormality caused by the operation of the surgical manipulator in the autonomous drive mode may occur. Further, when a manipulation device other than the surgical manipulator operates in the autonomous mode, similarly, risks such as an abnormality or a sign of an abnormality caused by the operation of the manipulation device in the autonomous mode may occur. Therefore, when the manipulation device operates in the autonomous mode, it is desired to appropriately cope with risks such as an abnormality or a sign of an abnormality caused by the operation of the manipulation device in the autonomous mode.
[0005] The present disclosure is to provide a manipulation system, a control method for a manipulation system, and a program that can appropriately cope with risks such as an abnormality or a sign of an abnormality caused by the operation of the manipulation device in the autonomous mode when the manipulation device operates in the autonomous mode.
[0006] The first aspect of the present disclosure of the manipulation system comprises: an operating device that accepts operations from an operator; a manipulation device that can switch between a remote operation mode that operates based on the operation of the operating device and an autonomous mode that operates autonomously; and a control device that, when the manipulation device is operating in autonomous mode, determines multiple types of risk elements and performs different risk responses based on the determination results of the multiple types of risk elements.
[0007] The manipulation system according to the first aspect of this disclosure includes a control device that, when the manipulation device is operating in autonomous mode, determines multiple types of risk factors and performs different risk responses based on the determination results of the multiple types of risk factors. This allows for the determination of multiple types of risk factors and the performance of appropriate risk responses based on the determination results of the multiple types of risk factors, unlike when only one type of risk factor is determined. As a result, when the manipulation device is operating in autonomous mode, it is possible to appropriately respond to risks such as abnormalities or signs of abnormalities caused by the operation of the manipulation device in autonomous mode.
[0008] A control method for a manipulation system according to the second aspect of this disclosure comprises an operating device that accepts operations from an operator, and a manipulation device that can switch between a remote operation mode that operates based on the operation of the operating device and an autonomous mode that operates autonomously, wherein when the manipulation device operates in autonomous mode, it includes determining multiple types of risk elements and performing different risk responses based on the determination results of the multiple types of risk elements.
[0009] The control method for a manipulation system according to the second aspect of this disclosure comprises, when the manipulation device is operating in autonomous mode, determining multiple types of risk elements and performing different risk responses based on the determination results of the multiple types of risk elements. This allows for the determination of multiple types of risk elements and the performance of appropriate risk responses based on the determination results of the multiple types of risk elements, unlike when only one type of risk element is determined. As a result, it is possible to provide a control method for a manipulation system that can appropriately respond to risks such as abnormalities or signs of abnormalities caused by the operation of the manipulation device in autonomous mode when the manipulation device is operating in autonomous mode.
[0010] The program according to the third aspect of this disclosure is a program that executes a control method for a manipulation system comprising an operating device that accepts operator input, a manipulation device that can switch between a remote operation mode that operates based on the operation of the operating device, and an autonomous mode that operates autonomously, wherein when the manipulation device is operating in autonomous mode, the program executes a control method for a manipulation system comprising: determining multiple types of risk elements, and performing different risk responses based on the determination results of the multiple types of risk elements.
[0011] The program according to the third aspect of this disclosure executes a control method for a manipulation system, which, when the manipulation device is operating in autonomous mode, includes determining multiple types of risk factors and performing different risk responses based on the determination results of the multiple types of risk factors. This allows for the determination of multiple types of risk factors and the performance of appropriate risk responses based on the determination results of the multiple types of risk factors, unlike when only one type of risk factor is determined. As a result, when the manipulation device is operating in autonomous mode, a program is provided that can appropriately respond to risks such as abnormalities or signs of abnormalities caused by the operation of the manipulation device in autonomous mode.
[0012] According to this disclosure, when a manipulation device operates in autonomous mode, it is possible to appropriately respond to abnormalities or signs of abnormalities caused by the operation of the manipulation device in autonomous mode.
[0013] This is a diagram showing the configuration of a surgical support system according to one embodiment. This is a diagram showing the display unit of a medical trolley according to one embodiment. This is a diagram showing the configuration of a medical trolley according to one embodiment. This is a diagram showing the configuration of a robot arm according to one embodiment. This is a diagram showing the instruments. This is a perspective view showing the configuration of the arm operation unit according to one embodiment. This is a diagram for explaining the translational movement of the robot arm. This is a diagram for explaining the rotational movement of the robot arm. This is a diagram showing an endoscope. This is a diagram showing a pivot position setting device. This is a diagram showing the operation unit according to one embodiment. This is a diagram showing the right-hand wrist unit according to one embodiment. This is a diagram showing the left-hand wrist unit according to one embodiment. This is a perspective view showing a foot pedal according to one embodiment. This is a control block diagram of a surgical support system according to one embodiment. This is a control block diagram of a robot arm according to one embodiment. This is a control block diagram of a positioner and medical trolley according to one embodiment. This is a control block diagram of the operation unit according to one embodiment. This is a diagram for explaining the control of the first control device according to one embodiment. This is a diagram for explaining the configuration of the inference control device according to one embodiment. This is a diagram showing the patient's surgical unit and instruments displayed on the monitor and display unit. This is a flowchart for explaining the autonomous mode of a surgical support system according to one embodiment. This is a diagram for explaining the configuration of a risk determination control device according to one embodiment. This is a flowchart for explaining the control method of a surgical support system according to one embodiment. This is Figure (1) illustrating the switching from autonomous mode to remote control mode. This is Figure (2) illustrating the switching from autonomous mode to remote control mode. This is a diagram illustrating the configuration of the risk determination control device when determining a dangerous state. This is a diagram illustrating the configuration of the risk determination control device when determining an input image. This is a diagram illustrating the configuration of the risk determination control device when determining an output operation. This is a diagram illustrating the configuration of the risk determination control device when determining the XYZ region. This is a diagram illustrating the configuration of the risk determination control device when determining the semantic region. This is a diagram illustrating the configuration of the risk determination control device when determining the speed range. This is a diagram illustrating the configuration of the risk determination control device when determining the estimated remaining time for work.This diagram illustrates the configuration of a risk assessment control device used to determine the accuracy of work. It also shows a modified example of a social robot and an industrial robot.
[0014] (Configuration of the surgical support system) The configuration of the surgical support system 500 according to this embodiment will now be described. The surgical support system 500 comprises a surgical support robot 100, a remote control device 200, a vision unit 300, an image processing unit 400, an inference control device 600, and a risk determination control device 700. The surgical support robot 100 is an example of a surgical device. The remote control device 200 is an example of an operating device. The surgical support system 500 is an example of a manipulation system. The risk determination control device 700 is an example of a control device.
[0015] In this specification, as shown in Figure 4, the longitudinal direction of the surgical instrument 1 is defined as the Z direction. The tip side of the surgical instrument 1 is defined as the Z1 side, and the base side of the surgical instrument 1 is defined as the Z2 side. The direction perpendicular to the Z direction is defined as the X direction. The direction perpendicular to both the Z and X directions is defined as the Y direction.
[0016] As shown in Figure 1, the surgical robot 100 is positioned inside the operating room. The remote control device 200 is positioned at a distance from the surgical robot 100. The remote control device 200 receives commands for the surgical instrument 1. Specifically, an operator, such as a physician, inputs commands to the remote control device 200 to cause the surgical robot 100 to perform a desired action. The remote control device 200 transmits the input commands to the surgical robot 100. The surgical robot 100 operates based on the received commands. The surgical robot 100 is positioned inside the operating room, which is a sterile field.
[0017] (Configuration of the surgical support robot) As shown in Figure 1, the surgical support robot 100 comprises a medical trolley 10, a trolley positioner operating unit 20, a positioner 30, an arm base 40, a plurality of robot arms 50, and an arm operating unit 60 provided on each robot arm 50.
[0018] As shown in Figure 3, the trolley positioner operation unit 20 is supported by a trolley positioner operation support unit 21 at the rear of the medical trolley 10, and the medical trolley 10 or positioner 30 is moved when the trolley positioner operation unit 20 is operated. The trolley positioner operation unit 20 includes an input device 22 and an operation handle 23. The input device 22 accepts operations for moving and changing the posture of the positioner 30, arm base 40, and multiple robot arms 50, mainly for preparing for surgery before the procedure. The medical trolley 10 includes an operation handle 23.
[0019] As shown in Figure 3, the input device 22 includes a display unit 22a, a joystick 22b, an enable switch 22c, an error reset button 22d, and a speaker 22e. The display unit 22a is, for example, a liquid crystal panel. As shown in Figure 2, the display unit 22a displays numbers corresponding to multiple robot arms 50. The display unit 22a also displays the type of surgical instrument 1 attached to each of the multiple robot arms 50. The display unit 22a displays a check mark CM indicating that the pivot position PP, which will be described later, has been set.
[0020] As shown in Figure 3, the joystick 22b is positioned near the display unit 22a of the input device 22. By selecting an operation mode displayed on the display unit 22a and operating the joystick 22b, the positioner 30 is moved in three dimensions.
[0021] The enable switch 22c is located near the joystick 22b. The enable switch 22c allows or disallows the movement of the positioner 30. When the enable switch 22c is pressed and the movement of the positioner 30 is permitted, the joystick 22b is operated, causing the positioner 30 to move.
[0022] The error reset button 22d clears errors in the surgical support system 500. The errors are, for example, deviation anomaly errors. A pair of speakers 22e are provided. The pair of speakers 22e are located near the location where the positioner 30 of the medical trolley 10 is positioned.
[0023] Furthermore, the operating handle 23 is located near the display unit 22a. The operating handle 23 has a throttle 23a that controls the movement of the medical cart 10 when grasped and rotated by an operator such as a nurse or technician. Specifically, the operating handle 23 is located below the input device 22. When the throttle 23a is rotated from the front to the back, the medical cart 10 moves forward. When the throttle 23a is rotated from the back to the front, the medical cart 10 moves backward. The speed of the medical cart 10 is also changed according to the amount of rotation of the throttle 23a. The operating handle 23 is configured to be rotatable left and right, as indicated by the R direction, and the medical cart 10 rotates along with the rotation of the operating handle 23.
[0024] Furthermore, an enable switch 23b is located on the operating handle 23 to allow or disallow the movement of the medical trolley 10. When the enable switch 23b is pressed and the movement of the medical trolley 10 is permitted, the throttle 23a on the operating handle 23 is operated, causing the medical trolley 10 to move.
[0025] As shown in Figure 1, the positioner 30 consists of a 7-axis articulated robot. The positioner 30 is positioned on a medical trolley 10. Because the positioner 30 is a 7-axis articulated robot, it adjusts the position of the arm base 40. The positioner 30 moves the position of the arm base 40 in three dimensions. However, the positioner 30 is not limited to a 7-axis articulated robot; for example, it may be an articulated robot with 6 axes or less, or 8 axes or more.
[0026] The positioner 30 includes a base portion 31 and a plurality of link portions 32 connected to the base portion 31. The plurality of link portions 32 are connected to each other by joints 33.
[0027] The arm base 40 is attached to the tip of the positioner 30. The base of each of the multiple robot arms 50 is attached to the arm base 40. The multiple robot arms 50 can be folded and stored. The arm base 40 and the multiple robot arms 50 are used covered with sterile drapes. The robot arms 50 also support surgical instruments 1.
[0028] The arm base 40 is equipped with a status indicator 41 and an arm status indicator 42, as shown in Figure 15. The status indicator 41 displays the status of the surgical support system 500. The arm status indicator 42 displays the status of the robot arm 50.
[0029] Multiple robot arms 50 are arranged. Specifically, four robot arms 50a, 50b, 50c, and 50d are arranged. Robot arms 50a, 50b, 50c, and 50d have similar configurations to each other. Robot arms 50a, 50b, 50c, and 50d are operated independently based on motion commands. Furthermore, robot arms 50a, 50b, and 50d are examples of second robot arms. Also, robot arm 50c is an example of a first robot arm.
[0030] As shown in Figure 4, the robot arm 50 includes an arm section 51, a first link section 52, a second link section 53, and a translational movement mechanism section 54. The robot arm 50 has joints JT1, JT2, JT3, JT4, JT5, JT6, JT7, and JT8. Joints JT1, JT2, JT3, JT4, JT5, JT6, and JT7 each have axes A1, A2, A3, A4, A5, A6, and A7 as rotation axes. JT8 has an axis A8 as a linear motion axis. The arm section 51 includes a base section 51a and a link section 51b.
[0031] The arm section 51 consists of a 7-axis articulated robot arm. The first link section 52 is located at the tip of the arm section 51. The arm operation section 60, which will be described later, is attached to the second link section 53. The translational movement mechanism section 54 is located between the first link section 52 and the second link section 53. A holder 55 for holding the surgical instrument 1 is located on the second link section 53. The translational movement mechanism section 54 translates the holder 55, to which the surgical instrument 1 is attached, between a first position and a second position. The first position is the position of the end of the range of movement of the holder 55 by the translational movement mechanism section 54 along the A8 axis, on the Z2 side. The second position is the position of the end of the range of movement of the holder 55 by the translational movement mechanism section 54 along the A8 axis, on the Z1 side.
[0032] Each of the multiple robot arms 50 has a surgical instrument 1 attached to its tip. The surgical instrument 1 includes, for example, a replaceable instrument 2, an endoscope 3 shown in Figure 9 for capturing images of the surgical site, and a pivot position setting device 4 shown in Figure 10 for setting the pivot position PP. The instrument 2 includes a driven unit 2a, an end effector 2b, a wrist joint 2c shown in Figure 5, and a shaft 2d. The end effector 2b is connected to the tip of the shaft 2d via the wrist joint 2c. Note that the configuration including the robot arms 50 and instruments 2 is just one example of a manipulation device.
[0033] As shown in Figure 1, an endoscope 3 is attached to the tip of one of the multiple robot arms 50, for example, robot arm 50c, and instruments 2, which are surgical instruments 1 other than the endoscope 3, are attached to the tips of the remaining robot arms 50a, 50b, and 50d. Robot arms 50b and 50d are positioned adjacent to robot arm 50c. Robot arm 50a is positioned adjacent to robot arm 50b. Preferably, the endoscope 3 is attached to one of the two centrally positioned robot arms 50b and 50c, of the four robot arms 50 that are positioned adjacent to each other. The endoscope 3 observes the environment around robot arms 50a, 50b, and 50d and outputs the observation information. Specifically, the endoscope 3 images the surgical site of patient P as the environment around robot arms 50a, 50b, and 50d, and outputs the captured images to the monitor 140 of the remote control device 200 and the display unit 220 located on the cart 210.
[0034] (Instrument Configuration) As shown in Figure 5, an end effector 2b having jaw members 2g and 2h is attached to the tip of the instrument 2. Scissors, grapplers, needle holders, micro dissectors, stable applicators, tackers, suction cleaning tools, snare wires, and clip applicators can be used as the end effector 2b.
[0035] Instrument 2 includes a first support member 2e and a second support member 2f. The first support member 2e is attached to the shaft 2d. The second support member 2f is rotatably supported by the first support member 2e around the A10 axis and supports the end effector 2b so as to be rotatable around the A11 axis which intersects the A10 axis. The shaft 2d rotates around the A9 axis. The wrist joint 2c is positioned between the second support member 2f and the first support member 2e with the A10 axis as its axis of rotation.
[0036] (Configuration of the arm operating section) As shown in Figure 6, the arm operating section 60 is attached to the robot arm 50 and operates the robot arm 50. Specifically, the arm operating section 60 is attached to the second link section 53.
[0037] The arm operation unit 60 includes an enable switch 61, a joystick 62, a linear switch 63, a mode switching button 64, a mode indicator 65, a pivot button 66, and an adjustment button 67.
[0038] The enable switch 61, when pressed, allows or disallows the movement of the robot arm 50 by the joystick 62 and linear switch 63. When the arm control unit 60 is held by an operator such as a nurse or assistant, pressing the enable switch 61 allows the robot arm 50 to move the surgical instrument 1.
[0039] The joystick 62 is a control device for operating the movement of the surgical instrument 1 by the robot arm 50. The joystick 62 controls the direction and speed of movement of the robot arm 50. The robot arm 50 moves according to the direction and angle at which the joystick 62 is tilted.
[0040] The linear switch 63 is a switch for moving the surgical instrument 1 in the Z direction, which is the longitudinal direction of the surgical instrument 1. The linear switch 63 includes a linear switch 63a that moves the surgical instrument 1 in the direction of insertion into the patient P, and a linear switch 63b that moves the surgical instrument 1 away from the patient P. Both the linear switch 63a and the linear switch 63b are push-button switches.
[0041] The mode switching button 64 is a push button switch for switching between a mode for translating the surgical instrument 1 and a mode for rotationally moving the surgical instrument 1. As shown in FIG. 7, in the mode for translating the robot arm 50, the robot arm 50 is moved such that the distal end 1a of the surgical instrument 1 moves on the X-Y plane. As shown in FIG. 8, in the mode for rotationally moving the robot arm 50, when the pivot position PP is not stored in the storage unit 351 shown in FIG. 15, rotational movement is performed about the center of the end effector 2b of the instrument 2 as the surgical instrument 1 on the A11 axis or the distal end of the end effector 2b as a fulcrum, and when the pivot position PP is stored in the storage unit 351, the robot arm 50 is moved such that the surgical instrument 1 rotationally moves about the pivot position PP as a fulcrum. The surgical instrument 1 is rotationally moved in a state where the shaft 1c of the surgical instrument 1 is inserted into the trocar T. The mode switching button 64 is disposed on the surface of the arm operation unit 60 on the Z direction side.
[0042] The mode indicator 65 displays the switched mode. Lighting of the mode indicator 65 represents the rotational movement mode, and extinction represents the translational movement mode. The mode indicator 65 also serves as a pivot position indicator that displays that the pivot position PP has been set. The mode indicator 65 is disposed on the surface of the arm operation unit 60 on the Z direction side.
[0043] The pivot button 66 is a push button switch for setting the pivot position PP that serves as the fulcrum for movement of the surgical instrument 1 attached to the robot arm 50. When the pivot button 66 is pressed, the pivot position PP is stored in the storage unit 351.
[0044] The adjustment button 67 is a button for optimizing the position of the robot arm 50. After the pivot position PP for the robot arm 50 to which the endoscope 3 is attached is set, the positions of the other robot arms 50 and the arm base 40 are optimized by pressing the adjustment button 67. The adjustment button 67 is a button different from the enable switch 61.
[0045] (Remote Control Device) As shown in Figure 1, the remote control device 200 is located, for example, inside or outside the operating room. The remote control device 200 includes an operating unit 110, a foot pedal 120, a touch panel 130, a monitor 140, a support arm 150, a support bar 160, and an error reset button 161. The operating unit 110 constitutes an operating handle for an operator, such as a doctor, to input commands. The operating unit 110 receives the operator's input to the robot arm 50.
[0046] As shown in Figure 11, the operating unit 110 is a handle for operating the surgical instrument 1. The operating unit 110 also receives input from the surgical instrument 1. The operating unit 110 operates the robot arm 50 and the instrument 2 by changing its position and orientation. When the position and orientation of the operating unit 110 are changed, the position and orientation of the robot arm 50 and the instrument 2 are changed in accordance with the change in the position and orientation of the operating unit 711. The operating unit 110 includes an operating unit 110L located on the left side as viewed from the operator, such as a doctor, and operated by the operator's left hand, and an operating unit 110R located on the right side and operated by the operator's right hand. The operating unit 110 includes an arm section 111 and a wrist section 112. The operating unit 110R includes an arm section 111R and a wrist section 112R. The operating unit 110L includes an arm section 111L and a wrist section 112L.
[0047] The arm portion 111 has joints JT21, JT22, and JT23 as shown in Figure 11, and JT24, JT25, JT26, and JT27 as shown in Figures 12 and 13. The rotation axes of joints JT21, JT22, JT23, JT24, JT25, JT26, and JT27 are designated as axes A21, A22, A23, A24, A25, A26, and A27, respectively.
[0048] As shown in FIG. 11, the arm unit 111R includes a link unit 111a, a link unit 111b, and a link unit 111c. The upper end side of the link unit 111a is attached to the remote operation device 200 so as to be rotatable around an A21 axis extending along the vertical direction. The upper end side of the link unit 111b is attached to the lower end side of the link unit 111a so as to be rotatable around an A22 axis extending along the horizontal direction. One end side of the link unit 111c is attached to the lower end side of the link unit 111b so as to be rotatable around an A23 axis extending along the horizontal direction. A wrist unit 112 is attached to the other end side of the link unit 111c so as to be rotatable around an A24 axis. The link unit 111a is connected to the remote operation device 200 via a joint JT21. The link unit 111a and the link unit 111b are connected via a joint JT22. The link unit 111b and the link unit 111c are connected via a joint JT23. The arm unit 111 supports the wrist unit 112. Note that the arm unit 111L also has the same configuration as the arm unit 111R.
[0049] The wrist unit 112 includes a wrist unit 112R operated by the operator's right hand shown in FIG. 12, and a wrist unit 112L operated by the operator's left hand shown in FIG. 13. FIG. 12 shows the reference posture of the operation unit 110R, and FIG. 13 shows the reference posture of the operation unit 110L. The configuration of the wrist unit 112R and the configuration of the wrist unit 112L are the same.
[0050] The wrist unit 112 includes a link unit 112a, a link unit 112b, a link unit 112c, and a grip support member 112d operated by an operator such as a doctor. The base end portion of the link unit 112a is connected to the distal end portion of the arm unit 111, and rotates around the A24 axis. The base end portion of the link unit 112b is connected to the distal end portion of the link unit 112a, and rotates around the A25 axis relative to the link unit 112a. The base end portion of the link unit 112c is connected to the distal end portion of the link unit 112b, the grip support member 112d is connected to the distal end portion thereof, and the link unit 112c rotates around the A26 axis relative to the link unit 112b. The grip support member 112d rotates around the A27 axis relative to the link unit 112c. Each of the link unit 112a, the link unit 112b, and the link unit 112c has an L-shape.
[0051] The wrist section 112 includes a pair of grip members 112e that are opened and closed by the operator. The grip members 112e consist of elongated plate-shaped lever members, and the proximal ends of the pair of grip members 112e are rotatably connected to the proximal end of the grip support member 112d. The grip members 112e have cylindrical finger insertion portions 112f. The operator operates the wrist section 112 by inserting their fingers into the pair of finger insertion portions 112f. The base ends of the pair of grip members 112e are connected to the grip support member 112d, and the opening angle between the jaw members 2g and 2h is changed by increasing or decreasing the angle between the pair of grip members 112e. A magnet is placed on one of the grip members 112e, and a Hall sensor is placed on the grip support member 112d. When the operator opens and closes the grip members 112e, the magnet and Hall sensor function as angle detection sensors, and the Hall sensor outputs the opening angle. Alternatively, a Hall sensor may be placed on the grip member 112e, and a magnet may be placed on the grip support member 112d.
[0052] As shown in Figure 1, the monitor 140 is a scope-type display device for displaying images captured by the endoscope 3. The monitor 140 also has a notification unit 141, which emits an error sound. The support arm 150 supports the monitor 140 so that its height is at the same height as the operator's face, such as a doctor. The touch panel 130 is located on the support bar 160. The surgical support robot 100 can be operated by the remote control device 200 by detecting the operator's head using a sensor located near the monitor 140. The operator operates the control unit 110 and foot pedal 120 while viewing the surgical area on the monitor 140. This inputs a command to the remote control device 200. The command input to the remote control device 200 is transmitted to the surgical support robot 100.
[0053] (Foot pedals) As shown in Figure 14, the foot pedals 120 are provided in multiple units to perform functions related to the surgical instrument 1. The multiple foot pedals 120 are arranged on the base unit 121. The foot pedals 120 include a switching pedal 122, a clutch pedal 123, a camera pedal 124, an incision pedal 125, a coagulation pedal 126, and a foot detection unit 127. The switching pedal 122, clutch pedal 123, camera pedal 124, incision pedal 125, and coagulation pedal 126 are operated by the operator's feet. The incision pedal 125 includes an incision pedal 125R for the right robot arm 50 and an incision pedal 125L for the left robot arm 50. The coagulation pedal 126 includes a coagulation pedal 126R for the right robot arm 50 and a coagulation pedal 126L for the left robot arm 50.
[0054] The switching pedal 122 switches the robot arm 50 to be operated by the control unit 110. The clutch pedal 123 performs a clutch operation that temporarily disconnects the control connection between the robot arm 50 and the control unit 110. While the clutch pedal 123 is pressed down by the operator, the operation of the control unit 110 is not transmitted to the robot arm 50. Also, while the camera pedal 124 is pressed down by the operator, the control unit 110 can operate the robot arm 50 to which the endoscope 3 is attached. While the incision pedal 125 or coagulation pedal 126 is pressed down by the operator, the electrosurgical device is activated.
[0055] (Vision Unit and Image Processing Unit) As shown in Figure 1, the vision unit 300 and the image processing unit 400 are mounted on the cart 210. The image processing unit 400 processes the images captured by the endoscope 3. A display unit 220 is located on the cart 210. The display unit 220 displays the images captured by the endoscope 3.
[0056] (Control System Configuration) As shown in Figure 15, the surgical support system 500 comprises a first control device 310, an arm control unit 320, a positioner control unit 330, an operation control unit 340, and a second control device 350. The surgical support system 500 also comprises a storage unit 311 connected to the first control device 310 and a storage unit 351 connected to the second control device 350. Note that the first control device 310 is an example of a control device.
[0057] The first control device 310 is positioned inside the medical trolley 10 to communicate with the arm control unit 320 and the positioner control unit 330, and controls the entire surgical support system 500. Specifically, the first control device 310 communicates with the arm control unit 320, the positioner control unit 330, and the operation control unit 340, and controls these units. The first control device 310 is connected to the arm control unit 320, the positioner control unit 330, and the operation control unit 340 by a LAN or the like. The first control device 310 is located inside the medical trolley 10.
[0058] An arm control unit 320 is located for each of the multiple robot arms 50. In other words, multiple arm control units 320 corresponding to the number of robot arms 50 are located inside the medical trolley 10.
[0059] As shown in Figure 15, the input device 22 is connected to the first control device 310 by a LAN or the like. The status indicator 41, arm status indicator 42, operating handle 23, throttle 23a, and joystick 22b are connected to the positioner control unit 330 by serial communication via a communication network that allows them to share information with each other, through wiring 360. In Figure 15, it is shown that all of the status indicators, including the status indicator 41 and arm status indicator 42, are connected to a single wiring 360, but in reality, there is a separate wiring 360 for each of the status indicator 41, arm status indicator 42, operating handle 23, throttle 23a, and joystick 22b.
[0060] As shown in Figure 16, the arm section 51 is equipped with multiple servo motors SM1, encoders EN1, and a reduction gear, each corresponding to a joint JT1, JT2, JT3, JT4, JT5, JT6, and JT7. The encoder EN1 detects the rotation angle of the servo motor SM1. The reduction gear reduces the rotation of the servo motor SM1 to increase the torque. Inside the medical trolley 10, a servo control unit SC1 for controlling the servo motor SM1 is located adjacent to the arm control unit 320. The encoder EN1 for detecting the rotation angle of the servo motor SM1 is electrically connected to the servo control unit SC1.
[0061] The second link section 53 is equipped with a servo motor SM2, an encoder EN2, and a reduction gear for rotating a driven member located on the driven unit 2a of the surgical instrument 1. The encoder EN2 detects the rotation angle of the servo motor SM2. The reduction gear reduces the rotation of the servo motor SM2 to increase the torque. The medical trolley 10 is also equipped with a servo control unit SC2 for controlling the servo motor SM2 that drives the surgical instrument 1. The encoder EN2 for detecting the rotation angle of the servo motor SM2 is electrically connected to the servo control unit SC2. Note that multiple servo motors SM2, encoders EN2, and servo control units SC2 are provided.
[0062] The translational movement mechanism 54 is equipped with a servo motor SM3 for translating the surgical instrument 1, an encoder EN3, and a reduction gear. The encoder EN3 detects the rotation angle of the servo motor SM3. The reduction gear reduces the rotation of the servo motor SM3 to increase the torque. The medical trolley 10 also has a servo control unit SC3 for controlling the servo motor SM3 that translates the surgical instrument 1. The encoder EN3 for detecting the rotation angle of the servo motor SM3 is electrically connected to the servo control unit SC3. The encoders EN1, EN2, and EN3 are, for example, magnetic, mechanical, or electromagnetic induction type. The high resolution of the encoders EN1, EN2, and EN3 themselves improves the positional accuracy of the tip of the robot arm 50 and the tip of the surgical instrument 1. Furthermore, by reducing the rotational speed transmitted from the servo motors SM1, SM2, and SM3 using a reduction gear, the accuracy of the rotational angle of the joints of the robot arm 50, which is the output shaft, is further improved.
[0063] The first control device 310 generates command values that command the positions (rotation angles of each motor) of the servo motors SM1, SM2, and SM3 based on the operation received by the remote control device 200, and drives the servo motors SM1, SM2, and SM3 based on the command values. The first control device 310 then detects an error indicating a deviation abnormality if the difference between the command values and the positions of the servo motors SM1, SM2, and SM3 detected by the sensors exceeds an acceptable range.
[0064] As shown in Figure 17, the positioner 30 is equipped with multiple servo motors SM4, an encoder EN4, and a reduction gear, corresponding to the multiple joints 33 of the positioner 30. The encoder EN4 is configured to detect the rotation angle of the servo motors SM4. The reduction gear is configured to reduce the rotation of the servo motors SM4 and increase the torque.
[0065] The medical trolley 10 is equipped with wheels, including front wheels that serve as drive wheels and rear wheels that are steered by an operating handle 23. The rear wheels are positioned closer to the operating handle 23 than the front wheels. The medical trolley 10 is also equipped with servo motors SM5, encoders EN5, reduction gears, and brakes BRK, each of which drives one of the multiple front wheels of the medical trolley 10. The reduction gear is configured to reduce the rotation of the servo motors SM5 and increase the torque. A potentiometer P1, as shown in Figure 3, is positioned on the operating handle 23, and the servo motors SM5 of the front wheels are driven based on the rotation angle detected by the potentiometer P1 in response to the twisting of the throttle 23a. The rear wheels of the medical trolley 10 are of the twin-wheel type, and the rear wheels are steered based on the left and right rotation of the operating handle 23. Furthermore, a potentiometer P2, as shown in Figure 3, is positioned on the rotation axis of the operating handle 23, and a servo motor SM6, encoder EN6, and reduction gear are positioned on the rear wheels of the medical trolley 10. The reduction gear is configured to reduce the rotation of the servo motor SM6 and increase the torque. The servo motor SM6 is driven based on the rotation angle detected by the potentiometer P2 in response to the left and right rotation of the operating handle 23. In other words, the steering of the rear wheels by the left and right rotation of the operating handle 23 is power-assisted by the servo motor SM6.
[0066] The medical trolley 10 moves in the forward and backward direction when its front wheels are driven. In addition, when the operating handle 23 is rotated, the rear wheels are steered, causing the medical trolley 10 to rotate in the left and right direction.
[0067] As shown in Figure 17, the medical trolley 10 is equipped with a servo control unit SC4 for controlling the servo motor SM4 that moves the positioner 30. An encoder EN4 for detecting the rotation angle of the servo motor SM4 is electrically connected to the servo control unit SC4. The medical trolley 10 is also equipped with a servo control unit SC5 for controlling the servo motor SM5 that drives the front wheels of the medical trolley 10. An encoder EN5 for detecting the rotation angle of the servo motor SM5 is electrically connected to the servo control unit SC5. The medical trolley 10 is also equipped with a servo control unit SC6 for controlling the servo motor SM6 that power assists the steering of the rear wheels of the medical trolley 10. An encoder EN6 for detecting the rotation angle of the servo motor SM6 is electrically connected to the servo control unit SC6.
[0068] As shown in Figures 16 and 17, each of the joints JT1, JT2, JT3, JT4, JT5, JT6, and JT7 of the arm section 51 and the joint 33 of the positioner 30 are equipped with brakes. Brakes are also equipped on the front wheels of the medical trolley 10, the arm base 40, and the translational movement mechanism 54. Control signals are transmitted unidirectionally from the arm control unit 320 to each of the brakes mounted on the joints JT1, JT2, JT3, JT4, JT5, JT6, and JT7 of the arm section 51 and the translational movement mechanism 54. The control signals are signals to turn the brakes on and off. The signal to turn on the brakes includes a signal to maintain the state in which the brakes are engaged. The same applies to the control signals from the positioner control unit 330 to each of the brakes mounted on the joint 33 of the positioner 30 and the arm base 40. At startup, all brakes (BRK) on the arm base 40, arm section 51, and translational movement mechanism 54 are released, and the servo motor SM is driven to counteract gravity, thereby maintaining the posture of the robot arm 50 and the arm base 40. When an error occurs in the surgical support system 500, the brakes (BRK) on the arm base 40, arm section 51, and translational movement mechanism 54 are turned on. When the error in the surgical support system 500 is cleared, the brakes (BRK) on the arm base 40, arm section 51, and translational movement mechanism 54 are turned off. The brakes (BRK) on the arm base 40, arm section 51, and translational movement mechanism 54 are turned on when the surgical support system 500 is shut down. In addition, the front wheels of the medical trolley 10 always have the brakes (BRK) turned on, and the brakes (BRK) are released only while the enable switch 23b is pressed. Furthermore, each joint 33 of the positioner 30 has the brake BRK always on, and the brake BRK is released only while the enable switch 22c is pressed.
[0069] As shown in Figure 18, servo motors SM7a, SM7b, SM7c, SM7d, SM7e, SM7f, and SM7g are arranged on the joints JT21, JT22, JT23, JT24, JT25, JT26, and JT27 of the operating section 110, respectively. Servo motor SM7a rotates the link section 111a around the A21 axis. Servo motor SM7b rotates the link section 111b around the A22 axis. Servo motor SM7c rotates the link section 111c around the A23 axis. Servo motor SM7d rotates the link section 112a around the A24 axis. Servo motor SM7e rotates the link section 112b around the A25 axis. Servo motor SM7f rotates the link section 112c around the A26 axis. Furthermore, the servo motor SM7g rotates the grip support member 112d around the A27 axis. Servo control units SC7a, SC7b, SC7c, SC7d, SC7e, SC7f, and SC7g are also provided to control each servo motor. Each servo control unit is electrically connected to encoders EN7a, EN7b, EN7c, EN7d, EN7e, EN7f, and EN7g for detecting the rotation angle of each servo motor. Each servo motor, each servo control unit, and each encoder are provided in the operation unit 110L and the operation unit 110R, respectively. The servo motors SM7a, SM7b, SM7c, SM7d, SM7e, SM7f, and SM7g move the respective link sections of the operation unit 110. Note that servo motors SM7a, SM7b, SM7c, SM7d, SM7e, SM7f, and SM7g are examples of actuators.
[0070] The first control device 310, via the operation control unit 340, controls each servo motor to generate a torque that counteracts the gravitational torque generated on the rotation axis of each servo motor, according to the posture of the operation unit 110. This makes it possible for the operator to operate the operation unit 110 with relatively little force.
[0071] Furthermore, when the operator rotates the grip support member 112d around the A27 axis of the operating section 110 shown in Figures 12 and 13, the shaft 2d of the instrument 2 rotates around the A9 axis shown in Figure 5. Also, when the operator rotates the joints JT24, JT25, and JT26 of the operating section 110 shown in Figures 12 and 13, the end effector 2b bends around the A10 axis or A11 axis shown in Figure 5.
[0072] The operation control unit 340 is located in the main body of the remote control device 200. The operation control unit 340 controls the operation unit 110. As shown in Figure 15, the operation control unit 340 is positioned to correspond to the left-hand operation unit 110L and the right-hand operation unit 110R, respectively.
[0073] As shown in Figure 15, the vision unit 300 and the image processing unit 400 are connected to the first control device 310 via a LAN or the like. The display unit 220 is connected to the vision unit 300.
[0074] (Control operation of the first control device) The control of the first control device 310 when the operation unit 110 receives an operation from the operator will be described below. The following control is performed by a functional block of the first control device 310 that operates based on the program 311a stored in the storage unit 311.
[0075] As shown in Figure 19, when the operator operates the operating unit 110, the axis values of the operating unit 110 are input to the first control device 310. The first control device 310 calculates the position and orientation of the operating unit 110 and the opening angle of the pair of grip members 112e from the axis values of the operating unit 110. In Figure 19, the operating unit 110 is denoted as HC. The position and orientation refer to the coordinate values in the TCP coordinate system, and the opening angle of the pair of grip members 112e refers to the opening angle of the jaw member 2g and jaw member 2h. TCP is an abbreviation for Tool Center Point. TCP is set, for example, at the tip of the instrument 2. The first control device 310 also calculates the difference between the current position and orientation of the operating unit 110 and the previous position and orientation of the operating unit 110, and calculates the amount of operation of the operating unit 110 based on the difference. Furthermore, the position and orientation of the endoscope 3 are input to the first control device 310.
[0076] The first control device 310 converts the operating amount of the control unit 110 into the coordinate system of the endoscope 3 based on the input position and orientation of the endoscope 3. Then, the first control device 310 calculates the operating amount of the instrument 2 based on the operating amount of the control unit 110 converted into the coordinate system of the endoscope 3. The first control device 310 also calculates the sum of the current operating amount of the instrument 2 and the previous operating amount of the instrument 2, and calculates the position and orientation of the instrument 2 based on the sum. Note that "this time" means the current control cycle, and "previous time" means the previous control cycle.
[0077] Furthermore, the first control device 310 performs a process to limit the axis speed of the robot arm 50 based on the calculated position and orientation of the instrument 2. The maximum value of the axis speed for each axis is set in advance, and if the axis speed of the robot arm 50 exceeds the maximum value, the first control device 310 limits the axis speed so that the axis speed becomes the maximum value. The functional block of the first control device 310 that limits the axis speed is called the limiting unit.
[0078] The first control device 310 performs inverse kinematics calculations for the limited axial velocities. The first control device 310 calculates command values for each axis of the robot arm 50 based on the inverse kinematics calculations. The first control device 310 then performs a process to apply acceleration limits to the calculated axial command values. Note that acceleration includes acceleration in the direction of increasing velocity and acceleration in the direction of decreasing velocity. The maximum value of the acceleration for each axis is set in advance, and if the acceleration of the robot arm 50 exceeds the maximum value, the first control device 310 limits the acceleration so that the acceleration becomes the maximum value. The limited axial velocities are then output to the servo control unit SC1 that drives the robot arm 50 and the servo control unit SC2 that drives the instrument 2. Furthermore, the signals indicating the position, orientation, and opening angles of the jaw members 2g and 2h included in the instrument 2, generated based on the operator's operation of the control unit 110 of the remote control device 200, will be referred to as manual operation commands below. The signals indicating the position, orientation, and opening angles of the jaw members 2g and 2h included in the instrument 2, inferred by the inference unit 613 of the inference control device 600 (described later), will be referred to below as inference operation commands.
[0079] (Inference Control Device) As shown in Figure 1, the inference control device 600 is located separately from the surgical support robot 100, the remote control device 200, the vision unit 300, and the image processing unit 400. The inference control device 600 is, for example, a PC (personal computer). As shown in Figure 15, the inference control device 600 is connected to the first control device 310 by a LAN or the like. As shown in Figure 20, the inference control device 600 includes a processing unit 610 such as a CPU (Central Processing Unit). The inference control device 600 also includes a storage unit 620. The storage unit 620 includes, for example, ROM (Read Only Memory), RAM (Random Access Memory), and HDD (hard disk drive). The inference control device 600 also includes an input unit 630. The input unit 630 is, for example, a keyboard.
[0080] In this embodiment, the processing unit 610 includes a data collector 611, a learner 612, and an inference unit 613. The data collector 611, learner 612, and inference unit 613 are functional blocks executed by the processing unit 610 based on a program 621 stored in the storage unit 620. The inference unit 613 is an example of a trained model.
[0081] The data acquisition unit 611 receives images G captured by the endoscope 3 attached to the robot arm 50c. As shown in Figure 21, images G include the surgical site of the patient P and the tip of the instrument 2. The data acquisition unit 611 collects and stores the images G captured by the endoscope 3 in chronological order. The data acquisition unit 611 also collects and stores manual operation commands and inference operation commands processed by the first control device 310 in chronological order. If the sampling periods of the images G captured by the endoscope 3 and the manual operation commands and inference operation commands are different, the data acquisition unit 611 adjusts for the time difference between the images G captured by the endoscope 3 and the manual operation commands and inference operation commands. The data acquired by the data acquisition unit 611 is stored, for example, in the data acquisition unit 611 itself or in the storage unit 620.
[0082] The learner 612 generates an inference 613, which is a trained model that has been trained by machine learning, using observational information of the environment surrounding multiple robot arms 50, each operated independently based on motion commands, and motion commands as input. Specifically, the learner 612 generates an inference 613, which is a trained model, by machine learning, using images G captured by the endoscope 3 collected by the data collector 611 and manual motion commands as training data. Machine learning can be, for example, deep learning using a neural network model, imitation learning that mimics human methods, or deep imitation learning that learns human methods using a neural network model.
[0083] In this embodiment, the manual operation commands learned by the learner 612 include suturing operation commands for suturing the surgical site of patient P. Specifically, the suturing operation commands are manual operation commands generated based on the operator's operation of the control unit 110 in order for the operator to suture the surgical site of patient P. The image G learned by the learner 612 is an image G captured by the endoscope 3 when suturing is performed on the surgical site of patient P. As shown in Figure 21, this image G shows the surgical site of patient P, the jaw members 2g and 2h at the tip of the instrument 2, the suture needle 641 held by the jaw members 2g and 2h, and the thread 642 threaded through the suture needle 641. In other words, the learner 612 learns the suturing operation of the surgical site of patient P and generates an inferencer 613 for inferring the suturing operation of the surgical site of patient P.
[0084] In this embodiment, the data acquisition and storage operation in which the data acquisition unit 611 acquires and stores images G captured by the endoscope 3 and manual operation commands as training data for generating the inference unit 613 is initiated and stopped, for example, by the coagulation pedal 126R. The coagulation pedal 126R is located separately from the coagulation pedal 126L, which switches between autonomous mode and remote operation mode, as described later. The coagulation pedal 126R is pressed by the operator's foot, which switches the start and stop of the data acquisition and storage operation of the data acquisition unit 611. Pressing by the foot means stepping on it with the foot.
[0085] Furthermore, when the coagulation pedal 126R is pressed, an alert sound is emitted. For example, the alert sound is emitted from the notification unit 141 of the monitor 140 shown in Figure 1 or from the speaker 22e of the input device 22.
[0086] (Performance Level) In addition to the images G captured by the endoscope 3 and manual operation commands, the learning device 612 can also learn the performance level of the robot arm 50 and instrument 2. The performance level includes, for example, who the operator is, the execution time of the suturing operation on the patient P's surgical site, and the score of the quality of the suturing. The performance level is input, for example, using the input unit 630 of the inference control device 600.
[0087] In this embodiment, the inference unit 613 takes observational information and operation commands of the environment surrounding the robot arm 50 and instrument 2 as input, and outputs an inferred operation command, which is the inferred operation command. As shown in Figure 19, the inference control device 600 receives, for example, an image G captured by the endoscope 3, the position and orientation of the tip of the endoscope 3, the position and orientation of the left and right instrument 2, and the opening angles of the jaw members 2g and 2h included in each of the left and right instrument 2 as input. The left and right instrument 2 refers to the two robot arms 50 selected by the operator as the target of operation from among the robot arms 50a, 50b, and 50d to which the instrument 2 is attached. The operator switches the robot arm 50 to be operated by the operation unit 110 by operating the switching pedal 122. The inference control device 600 also receives information on whether it is in autonomous mode or remote control mode. This information may be stored in the storage unit 620, storage unit 311, storage unit 351, or on the cloud. The information indicating whether the system is in autonomous mode or remote control mode refers to whether the coagulation pedal 126L is being pressed by the operator, as will be described later. The inference control device 600 also receives information on the start or stop of the collection and storage operation, which collects and stores images G captured by the endoscope 3 and manual operation commands as training data for generating the inference unit 613. This information may also be stored in the storage unit 620, storage unit 311, storage unit 351, or on the cloud. This information may also be stored for each robot arm 50.
[0088] In this embodiment, the inference unit 613 takes observation information and operation commands from multiple past points in time stored in the data collector 611 or the storage unit 620 as input and outputs inference operation commands for the next and subsequent times. The next time may be immediately after the current time, or a little while after the current time. The next inference operation command may be an inference operation command for one step in the next time, or an inference operation command for several steps after the next time. For example, the inference unit 613 receives images G captured by the endoscope 3 for several steps of the control cycle of the surgical support robot 100 as observation information. The inference unit 613 also receives manual operation commands, inference operation commands, or operation commands that are a mixture of manual operation commands and inference operation commands for several steps of the control cycle of the surgical support robot 100 as input. The images G, manual operation commands, inference operation commands, etc., are input to the inference control device 600 via the input port 640. The inference unit 613 outputs inference operation commands based on this input information.
[0089] In this embodiment, the robot arm 50 and instrument 2 are configured to be switchable between a remote control mode, which operates based on the operation of the control unit 110 of the remote control device 200, and an autonomous mode, which operates autonomously. Specifically, at least one set of the multiple robot arm 50 and instrument 2 sets can be switched between an autonomous mode, which operates based on inference operation commands output from the inference unit 613, and a remote control mode, which operates based on manual operation commands, which are operation commands from the remote control device 200. In this embodiment, all sets of robot arms 50a, 50b, and 50d, and the instruments 2 attached to them, can be switched between autonomous mode and remote control mode. It is also possible to set some of the sets of multiple robot arm 50 and instrument 2 to autonomous mode and others to remote control mode. Furthermore, the robot arm 50c to which the endoscope 3 is attached may also be switchable between autonomous mode and remote control mode. The operation in autonomous mode includes suturing operations for suturing the surgical site of patient P.
[0090] In addition, additional information related to surgery may be included as input information to the inference unit 613 in autonomous mode. This additional information may include, for example, information specifying a surgical procedure or a specific step in a particular surgical procedure, or information specifying a particular technique such as suturing, or information on steps that further subdivide that technique. The input information may be, for example, voice instructions, input via an interface other than voice instructions such as a touch panel, input via a hard switch, or input via the operator's gestures. Furthermore, if, for example, a suturing technique is input as additional information related to surgery, the inference unit 613 can further subdivide the suturing technique into lower-level steps for inference. Lower-level steps refer to individual actions such as, for example, holding the suture needle 641, passing the suture needle 641 through the suture site of patient P, and tying the thread 642. It is also possible to directly input information on these lower-level steps via voice, an interface, or a hard switch.
[0091] Furthermore, the duration of the image G input to the inference unit 613 and the duration of the operation command may be set. For example, by operating the touch panel 130 of the remote control device 200, the duration of the image G input to the inference unit 613 and the duration of the operation command may be set to 5 steps, 10 steps, 15 steps, etc., of the control cycle.
[0092] Furthermore, in addition to the image G and the operation command, the performance level of the instrument 2 may be input to the inference unit 613, and the inference unit 613 may be trained according to the performance level. Then, the performance level of the inference unit 613 to be used for inference may be selected. In other words, multiple inference units 613 may be prepared according to their performance levels. Then, by operating the touch panel 130 of the remote control device 200, the performance level of the inference unit 613 is selected, and inference is performed by the selected inference unit 613.
[0093] In this embodiment, in autonomous mode, the operation command and the image G acquired by the endoscope 3 are input to the inference unit 613, and the inference unit 613 outputs an inference operation command, and the robot arm 50 and instrument 2 operate based on the inference operation command. The inference operation command includes, for example, the position and orientation of the left and right instrument 2, and the opening angles of the jaw members 2g and 2h included in each of the left and right instrument 2. Here, as shown in Figure 19, the first control device 310 performs a process to limit the axial speed of the robot arm 50 and instrument 2 as described above. The inference operation command is replaced with the manual operation command before the process of applying axial speed and before the calculation of the position and orientation of the instrument 2. That is, in autonomous mode, the manual operation command is replaced with the inference operation command before the manual operation command is limited by the first control device 310.
[0094] As described above, the switching pedal 122 switches between the robot arms 50 operated by the left and right control units 110. For example, the switching pedal 122 selects two robot arms 50 to be operated by the left and right control units 110 from among the robot arms 50a, 50b, and 50d to which the instrument 2 is attached. Here, when the mode is switched, if both the left robot arm 50 and instrument 2 set and the right robot arm 50 and instrument 2 set are switched modes, the autonomous mode and remote control mode are switched by the solidification pedal 126L. The solidification pedal 126L is operated by the operator's foot. The autonomous mode is executed while the operator is pressing down on the solidification pedal 126L with their foot. The remote control mode is executed while the operator is releasing the pressure on the solidification pedal 126L. Pressing down with the foot means that the operator is pressing down on the solidification pedal 126L. Releasing the pressure means that the operator has stopped pressing the coagulation pedal 126L. On the other hand, when switching modes for each left-hand pair, two switching units are required: one for switching the right-hand mode and another for switching the left-hand mode. For example, in addition to the coagulation pedal 126L, another foot pedal 120 may be assigned as a switching unit.
[0095] Even in remote control mode, the inference unit 613 continues to infer inference operation commands using manual operation commands in remote control mode as input. The autonomous mode is started when the operator presses down on the solidification pedal 126L. In autonomous mode, the robot arm 50 and instrument 2 operate based on the inference operation commands output from the inference unit 613. The remote control mode is started when the operator releases the pressure on the solidification pedal 126L. In remote control mode, manual operation commands are generated when the operator operates the control unit 110, and the robot arm 50 and instrument 2 operate based on the manual operation commands. Here, even after switching from autonomous mode to remote control mode, the inference unit 613 continues to infer inference operation commands. Specifically, at the initial point after switching from autonomous mode to remote control mode, the inference operation commands and manual operation commands are included in several steps of the past control cycle, so the inference unit 613 infers inference operation commands using past observation information, past inference operation commands, and past manual operation commands as input. After some time has passed since switching from autonomous mode to remote control mode, the past few steps of the control cycle will only contain past manual operation commands. Therefore, the inference unit 613 uses past observation information and past manual operation commands as input to infer an inference operation command.
[0096] Next, the control process in the autonomous mode of the surgical support system 500 will be described. In the following control method of the surgical support system 500, the operations performed by the inference control device 600 are performed based on the program 621 stored in the memory unit 620, and the operations performed by the first control device 310 are performed based on the program 311a stored in the memory unit 311.
[0097] As shown in Figure 22, when autonomous mode is started, in step S41, the operation commands for a control cycle of approximately 1 second and the image G captured by the endoscope 3 are collected by the data collector 611. The image G from the endoscope 3 is based on the Full HD standard, and the resolution is 1920 x 1080 pixels. Next, in step S42, the inference unit 613 infers an inferred operation command based on the collected operation commands and image G. At the initial point after switching from remote control mode to autonomous mode, the past few steps of the control cycle contain both inferred operation commands and manual operation commands, so the inference unit 613 uses past observation information, past inferred operation commands, and past manual operation commands as input to infer an inferred operation command. After some time has passed since switching from remote control mode to autonomous mode, the past few steps of the control cycle contain only past inferred operation commands, so the inference unit 613 uses past observation information and past inferred operation commands as input to infer an inferred operation command. Furthermore, the image G of the endoscope 3 used by the inference unit 613 for inferring the inference operation command has been reduced in size to a resolution of 256 x 128 pixels. The inference operation command that the inference unit 613 infers is an inference operation command for approximately one second following the operation command for approximately one second of the control cycle collected by the data collector 611. The time required for the inference unit 613 to infer the inference operation command is approximately 0.1 seconds, which is shorter than the period of the inference operation command that the inference unit 613 infers. In step S42, the data collector 611 collects operation commands and the image G captured by the endoscope 3 in parallel while the inference unit 613 is performing the inference. In step S43, the inference control device 600 outputs the inferred inference operation command to the first control device 310. As a result, the robot arm 50 and instrument 2 operate autonomously, and the posture of the operating unit 110 is also changed to follow the changes in the posture of the robot arm 50 and instrument 2.
[0098] (Risk Determination Control Device) As shown in Figure 1, the risk determination control device 700 is located separately from the surgical support robot 100, the remote control device 200, the vision unit 300, and the image processing unit 400. The risk determination control device 700 is, for example, a PC (personal computer). As shown in Figure 15, the risk determination control device 700 is connected to the first control device 310 by a LAN or the like. As shown in Figure 23, the risk determination control device 700 includes a processing unit 710 such as a CPU (Central Processing Unit). The risk determination control device 700 also includes a storage unit 720. The storage unit 720 is, for example, ROM (Read Only Memory), RAM (Random Access Memory), and HDD (hard disk drive).
[0099] The processing unit 710 includes at least one of the determination units 710a, 710b, 710c, 710d, 710e, 710f, 710g, and 710h, which will be described later. The determination units 710a, 710b, 710c, 710d, 710e, 710f, 710g, and 710h are functional blocks executed by the processing unit 710 based on the program 721 stored in the storage unit 720.
[0100] (Control Method for Surgical Support System) Next, the control method for the surgical support system 500 will be described. In the following control method for the surgical support system 500, the actions performed by the risk determination control device 700 are performed based on the program 721 stored in the memory unit 720, and the actions performed by the first control device 310 are performed based on the program 311a stored in the memory unit 311. Note that the control method for the surgical support system 500 is an example of a control method for a manipulation system.
[0101] In this embodiment, as shown in Figure 24, in step S1, the risk determination control device 700 determines multiple types of risk elements when the robot arm 50 and instrument 2 are operating in autonomous mode. Then, in step S2, the first control device 310 takes different risk responses based on the determination results of the multiple types of risk elements by the risk determination control device 700. Specifically, the first control device 310 takes at least one of the following risk responses based on the determination results of the multiple types of risk elements: issuing a warning, stopping the robot arm 50 and instrument 2, and switching from autonomous mode to remote control mode. The first control device 310 may take only two of the following actions: issuing a warning, stopping the robot arm 50 and instrument 2, and switching from autonomous mode to remote control mode, or it may take all of them.
[0102] When issuing a warning as a risk response, the first control device 310 controls the notification of the warning by at least one of display and sound. For example, the first control device 310 controls the display to show a warning screen on at least one of the display unit 22a, touch panel 130, monitor 140, and display unit 220. Alternatively, for example, the first control device 310 controls the announcement of a warning sound on at least one of the speaker 22e and notification unit 141.
[0103] When stopping the robot arm 50 and instrument 2 as a risk mitigation measure, the first control device 310 gradually slows down the movement of the robot arm 50 and instrument 2 and controls them to stop. This makes it possible to safely stop the robot arm 50 and instrument 2.
[0104] When switching from autonomous mode to remote control mode as a risk response, the switching may be performed for each set of multiple robot arms 50 and instruments 2. Each set of multiple robot arms 50 and instruments 2 is switchable between autonomous mode and remote control mode. Based on the determination results of multiple types of risk elements, the first control device 310 switches at least one of the sets of multiple robot arms 50 and instruments 2 from autonomous mode to remote control mode as a risk response. For example, as shown in Figure 25, the first control device 310 controls both the left and right sets of robot arms 50 and instruments 2 that are operating in autonomous mode to switch from autonomous mode to remote control mode.
[0105] When switching from autonomous mode to remote control mode as a risk response, the robot arm 50 and some parts of the instrument 2 may be switched from autonomous mode to remote control mode. The robot arm 50 and instrument 2 can be partially switched between autonomous mode and remote control mode. Based on the determination results of multiple types of risk elements, the first control device 310 switches parts of the robot arm 50 and instrument 2 from autonomous mode to remote control mode as a risk response. In other words, the percentage of the part left to the inference unit 613 is changed from 100% to less than 100%, such as 50% or 80%. For example, the first control device 310 switches the end effector 2b, wrist joint 2c, and shaft 2d of the instrument 2 from autonomous mode to remote control mode, while maintaining the autonomous mode of the other parts.
[0106] Furthermore, in this embodiment, the first control device 310, based on the determination results of multiple types of risk elements, will either automatically switch the robot arm 50 and instrument 2 from autonomous mode to remote control mode as a risk response, or provide a notification prompting the user to switch the robot arm 50 and instrument 2 from autonomous mode to remote control mode. For example, the first control device 310 will provide a notification prompting the user to switch the robot arm 50 and instrument 2 from autonomous mode to remote control mode by display and at least one of audio.
[0107] (Determination of Risk Elements) For example, as shown in Figure 27, the risk determination control device 700 may determine a dangerous state as a determination of multiple types of risk elements. The determination of a dangerous state is performed by the dangerous state determination device 710a of the processing unit 710. The dangerous state determination device 710a is a functional block executed by the processing unit 710 based on the program 721 stored in the memory unit 720. The dangerous state determination device 710a may be a machine learning trained model, or it may be a logical determination device other than a trained model.
[0108] The hazard detection device 710a, for example, determines whether there is an increase in the amount of blood based on the image G captured by the endoscope 3, and determines whether or not a hazardous condition exists. If the hazard detection device 710a determines that a hazardous condition exists, risk response measures such as issuing a warning or switching from autonomous mode to remote control mode are taken.
[0109] Furthermore, as shown in Figure 28, for example, the risk determination control device 700 may perform input image determination as a determination of multiple types of risk elements. The input image is an image input to the inference unit 613, and specifically, is the image G captured by the endoscope 3. The input image determination is performed by the input image determination unit 710b of the processing unit 710. The input image determination unit 710b is a functional block executed by the processing unit 710 based on the program 721 stored in the memory unit 720. The input image determination unit 710b may be a machine learning trained model, or it may be a logical determination unit other than a trained model.
[0110] The input image classifier 710b determines whether the input image input to the inference unit 613 as observation information deviates from the input image used by the inference unit 613 during training. If the input image classifier 710b determines that there is a deviation in the input image, risk management measures such as issuing a warning or switching from autonomous mode to remote control mode are taken.
[0111] Furthermore, as shown in Figure 29, for example, the risk determination control device 700 may determine output operations as a determination of multiple types of risk elements. An output operation is an operation output by the inference unit 613 as an inference operation command. The determination of output operations is performed by the output operation determination unit 710c of the processing unit 710. The output operation determination unit 710c is a functional block executed by the processing unit 710 based on the program 721 stored in the memory unit 720. The output operation determination unit 710c may be a machine learning trained model, or it may be a logical determination unit other than a trained model.
[0112] The output operation determination unit 710c determines whether the output operation output by the inference unit 613 as an inference operation command deviates from the output operation performed by the inference unit 613 during its learning process. For example, the output operation determination unit 710c sets a threshold value based on the position, velocity, acceleration, etc., of the inference unit 613 during its learning process, determines whether the output operation exceeds the threshold value, and determines whether the output operation output by the inference unit 613 as an inference operation command deviates from the output operation performed by the inference unit 613 during its learning process. Alternatively, for example, the output operation determination unit 710c determines whether the output operation output by the inference unit 613 belongs to one of a plurality of clusters created in advance based on the pattern of the output operation output by the inference unit 613, thereby determining whether the output operation output by the inference unit 613 as an inference operation command deviates from the output operation performed by the inference unit 613 during its learning process. A cluster is a group of output operations divided according to the pattern of the output operation. When the inference unit 613 performs training, multiple clusters are created according to the output operation patterns. If the output operation judger 710c determines that there is a discrepancy in the output operation, risk management measures such as issuing a warning or switching from autonomous mode to remote control mode are taken.
[0113] Furthermore, as shown in Figure 30, for example, the risk determination control device 700 may perform XYZ region determination as a determination of multiple types of risk elements. The XYZ region is the three-dimensional region in which the robot arm 50 and instrument 2 move. The XYZ region determination is performed by the XYZ region determination unit 710d of the processing unit 710. The XYZ region determination unit 710d is a functional block executed by the processing unit 710 based on the program 721 stored in the memory unit 720. The XYZ region determination unit 710d may be a machine learning trained model, or it may be a logical determination unit other than a trained model.
[0114] The XYZ region determination unit 710d, for example, determines whether the positions of the robot arm 50 and instrument 2 exceed a predetermined threshold, and determines whether the robot arm 50 and instrument 2 have moved outside the permissible XYZ region. If the XYZ region determination unit 710d determines that the robot arm 50 and instrument 2 have moved outside the permissible XYZ region, the robot arm 50 and instrument 2 are stopped as a risk response.
[0115] Furthermore, as shown in Figure 31, for example, the risk determination control device 700 may perform semantic region determination as a determination of multiple types of risk elements. A semantic region is a region identified by semantic segmentation, which is a region where the robot arm 50 and instrument 2 want to avoid interference, such as blood vessels or organs. The semantic region determination is performed by the semantic region determination device 710e of the processing unit 710. The semantic region determination device 710e is a functional block executed by the processing unit 710 based on a program 721 stored in the memory unit 720. The semantic region determination device 710e may be a machine learning-trained model, or it may be a logical determination device other than a trained model.
[0116] The semantic region determination unit 710e determines, for example, whether the robot arm 50 and instrument 2 interfere with the semantic region based on the image G captured by the endoscope 3. If the semantic region determination unit 710e determines that the robot arm 50 and instrument 2 interfere with the semantic region, the robot arm 50 and instrument 2 are stopped as a risk response.
[0117] Furthermore, as shown in Figure 32, for example, the risk determination control device 700 may determine speed or acceleration as one of several types of risk elements. The speed or acceleration determination is performed by the speed determination device 710f of the processing unit 710. The speed determination device 710f is a functional block executed by the processing unit 710 based on the program 721 stored in the memory unit 720. The speed determination device 710f may be a machine learning trained model, or it may be a logical determination device other than a trained model.
[0118] The speed detector 710f determines, for example, whether the speed or acceleration of the robot arm 50 and instrument 2 exceeds a predetermined threshold, and whether the robot arm 50 and instrument 2 have deviated from the allowable speed or acceleration range. If the speed detector 710f determines that the robot arm 50 and instrument 2 have deviated from the allowable speed or acceleration range, the robot arm 50 and instrument 2 are stopped as a risk response.
[0119] Furthermore, as shown in Figure 33, for example, the risk determination control device 700 may determine the predicted remaining time as a determination of multiple types of risk elements. The predicted remaining time is the predicted remaining time for the work of the robot arm 50 and instrument 2, which is calculated from information such as the image G captured by the endoscope 3. The determination of the predicted remaining time is performed by the predicted remaining time determination device 710g of the processing unit 710. The predicted remaining time determination device 710g is a functional block executed by the processing unit 710 based on the program 721 stored in the memory unit 720. The predicted remaining time determination device 710g may be a machine learning trained model, or it may be a logical determination device other than a trained model.
[0120] The predicted remaining time detector 710g determines whether there is an anomaly in the predicted remaining time, for example, based on the magnitude or change in the predicted remaining time. For example, the predicted remaining time detector 710g determines that there is an anomaly in the predicted remaining time if the predicted remaining time does not change for a predetermined time, if the predicted remaining time increases rapidly, if the predicted remaining time is excessively large, or if the predicted remaining time is excessively small. If the predicted remaining time detector 710g determines that there is an anomaly in the predicted remaining time, risk response measures such as issuing a warning or switching from autonomous mode to remote control mode are taken. The predicted remaining time can be obtained using a machine learning-trained model. The predicted remaining time detector 710g may include a trained model for obtaining the predicted remaining time, or a separate trained model for obtaining the predicted remaining time may be provided.
[0121] Furthermore, as shown in Figure 34, for example, the risk determination control device 700 may also perform accuracy determination as a determination of multiple types of risk elements. Accuracy refers to the accuracy of the work of the robot arm 50 and instrument 2, which is calculated from information such as the image G captured by the endoscope 3. Accuracy determination is performed by the accuracy determination device 710h of the processing unit 710. The accuracy determination device 710h is a functional block executed by the processing unit 710 based on the program 721 stored in the memory unit 720. The accuracy determination device 710h may be a machine learning trained model, or it may be a logical determination device other than a trained model.
[0122] The accuracy judging device 710h determines whether there is an accuracy anomaly, for example, based on the magnitude or change in accuracy. For example, the accuracy judging device 710h determines that there is an accuracy anomaly if the accuracy is excessively low or if the accuracy decreases rapidly. If the accuracy judging device 710h determines that there is an accuracy anomaly, risk response measures such as issuing a warning or switching from autonomous mode to remote control mode are taken. Accuracy can be acquired using a machine learning-trained model. The accuracy judging device 710h may include a trained model for acquiring accuracy, or a separate trained model for acquiring accuracy may be provided.
[0123] Furthermore, in this embodiment, the risk determination control device 700 may use at least one trained model to determine multiple types of risk elements. That is, at least one of the determination devices 710a, 710b, 710c, 710d, 710e, 710f, 710g, and 710h may be a trained model.
[0124] Furthermore, each of the 710a, 710b, 710c, 710d, 710e, 710f, 710g, and 710h judging devices may perform a stepwise judgment. In this case, a stepwise risk response will be implemented. For example, in the first stage, a warning may be issued; in the second stage, a switch from autonomous mode to remote control mode may be performed; and in the third stage, the robot arm 50 and instrument 2 may be stopped. Alternatively, as a stepwise risk response, the part of switching the robot arm 50 and instrument 2 from autonomous mode to remote control mode may be increased in stages. It is also possible for judging devices that perform stepwise judgments and those that do not to be mixed. In addition, different risk responses may be implemented not only according to the individual judgment results of each judging device, but also according to the combination of judgment results of each judging device.
[0125] (Switching from autonomous mode to remote control mode) In this embodiment, when the first control device 310 switches the robot arm 50 and instrument 2 from autonomous mode to remote control mode, it controls the operation of the robot arm 50 and instrument 2 to facilitate the switch from autonomous mode to remote control mode. Specifically, the first control device 310 controls the operation of the robot arm 50 and instrument 2 to gradually slow down and stop. This makes it possible to smoothly switch the robot arm 50 and instrument 2 from autonomous mode to remote control mode.
[0126] [Effects of this embodiment] When the robot arm 50 and instrument 2 are operating in autonomous mode, the risk determination control device 700 determines multiple types of risk elements, and the first control device 310 performs different risk responses based on the determination results of the multiple types of risk elements. This makes it possible to determine multiple types of risk elements and perform appropriate risk responses based on the determination results of the multiple types of risk elements, unlike when only one type of risk element is determined. As a result, when the robot arm 50 and instrument 2 are operating in autonomous mode, it is possible to appropriately respond to risks such as abnormalities and signs of abnormalities caused by the operation of the robot arm 50 and instrument 2 in autonomous mode.
[0127] The first control device 310 performs at least one of the following risk response actions based on the determination results of multiple types of risk elements: issuing a warning, stopping the robot arm 50 and instrument 2, and switching from autonomous mode to remote control mode. This allows the user to appropriately respond to risks such as abnormalities or signs of abnormalities caused by the operation of the robot arm 50 and instrument 2 in autonomous mode when a warning is issued as a risk response. Furthermore, stopping the robot arm 50 and instrument 2 as a risk response can suppress the progression of risks such as abnormalities or signs of abnormalities caused by the operation of the robot arm 50 and instrument 2 in autonomous mode. Additionally, switching from autonomous mode to remote control mode suppresses the progression of risks such as abnormalities or signs of abnormalities caused by the operation of the robot arm 50 and instrument 2 in autonomous mode, while allowing the user to appropriately respond to risks such as abnormalities or signs of abnormalities caused by the operation of the robot arm 50 and instrument 2 in autonomous mode through remote control.
[0128] Multiple sets of robot arms 50 and instruments 2 can be switched between autonomous mode and remote control mode. The risk determination control device 700, based on the determination results of multiple types of risk elements, switches at least one of the sets of robot arms 50 and instruments 2 from autonomous mode to remote control mode as a risk response. This allows for switching at least one set of robot arms 50 and instruments 2 that is related to risks such as abnormalities or signs of abnormalities caused by the operation of the autonomous robot arms 50 and instruments 2 to remote control mode as a risk response. As a result, risks such as abnormalities or signs of abnormalities caused by the operation of the autonomous robot arms 50 and instruments 2 can be effectively suppressed from progressing, while the user can appropriately respond to risks such as abnormalities or signs of abnormalities caused by the operation of the autonomous robot arms 50 and instruments 2 through remote control.
[0129] The robot arm 50 and instrument 2 can be partially switched between autonomous mode and remote control mode. The first control device 310 switches a portion of the robot arm 50 and instrument 2 from autonomous mode to remote control mode as a risk response, based on the determination results of multiple types of risk elements. This allows for the switching of a portion of the robot arm 50 and instrument 2 from autonomous mode to remote control mode in response to risks such as abnormalities or signs of abnormalities caused by the operation of the robot arm 50 and instrument 2 in autonomous mode. As a result, risks such as abnormalities or signs of abnormalities caused by the operation of the robot arm 50 and instrument 2 in autonomous mode can be effectively suppressed, while the user can appropriately respond to risks such as abnormalities or signs of abnormalities caused by the operation of the robot arm 50 and instrument 2 in autonomous mode through remote control.
[0130] Based on the determination results of multiple types of risk factors, the first control device 310 either automatically switches the robot arm 50 and instrument 2 from autonomous mode to remote control mode as a risk response, or issues a notification prompting the user to switch the robot arm 50 and instrument 2 from autonomous mode to remote control mode. This reduces the burden on the user to perform the operation of switching the robot arm 50 and instrument 2 from autonomous mode to remote control mode when the robot arm 50 and instrument 2 are automatically switched from autonomous mode to remote control mode. In addition, when a notification prompting the user to switch the robot arm 50 and instrument 2 from autonomous mode to remote control mode is issued, the user is informed that they need to perform the operation of switching the robot arm 50 and instrument 2 from autonomous mode to remote control mode.
[0131] The surgical support system 500 includes an inference unit 613 that takes observational information of the environment surrounding the robot arm 50 and instrument 2, as well as operation commands for the robot arm 50 and instrument 2, as inputs, and outputs inferred operation commands, which are the inferred operation commands. In autonomous mode, the robot arm 50 and instrument 2 operate based on the inferred operation commands output from the inference unit 613. This allows the robot arm 50 and instrument 2 to be easily and accurately operated in autonomous mode using the inference unit 613.
[0132] The risk determination control device 700 performs at least one of the following as determinations of multiple types of risk elements: determining whether the input image input to the inference unit 613 as observation information is different from the input image used when the inference unit 613 was learning; and determining whether the output operation output by the inference unit 613 as an inference operation command is different from the output operation used when the inference unit 613 was learning. In this way, when determining whether the input image input to the inference unit 613 as observation information is different from the input image used when the inference unit 613 was learning, it is possible to determine whether an unknown input image has been input to the inference unit 613. Here, the input of an unknown input image to the inference unit 613 is related to the occurrence of risks such as abnormalities or signs of abnormalities caused by the operation of the robot arm 50 and instrument 2 in autonomous mode. Therefore, based on the input image, it is possible to appropriately determine whether a risk such as an abnormality or a precursor to an abnormality has occurred due to the operation of the autonomous robot arm 50 and instrument 2. Furthermore, as a determination of multiple types of risk elements, when determining whether the output operation output by the inference unit 613 as an inference operation command deviates from the output operation when the inference unit 613 was learning, it is possible to determine whether the inference unit 613 has output an unknown output operation. Here, the fact that the inference unit 613 has output an unknown output operation is related to the occurrence of a risk such as an abnormality or a precursor to an abnormality due to the operation of the autonomous robot arm 50 and instrument 2. Therefore, based on the output operation, it is possible to appropriately determine whether a risk such as an abnormality or a precursor to an abnormality has occurred due to the operation of the autonomous robot arm 50 and instrument 2.
[0133] The risk determination control device 700 determines whether the output operation output by the inference unit 613 belongs to one of a plurality of clusters created in advance based on the pattern of output operations output by the inference unit 613. By doing so, it determines whether the output operation output by the inference unit 613 as an inference operation command deviates from the output operation that the inference unit 613 performed during learning. This effectively utilizes the fact that known output operations belong to one of the plurality of clusters, while unknown output operations do not belong to any of the plurality of clusters, making it easy and accurate to determine whether the inference unit 613 has output an unknown output operation.
[0134] The risk determination control device 700 performs at least one of the following as determinations of multiple types of risk elements: determination regarding the predicted remaining time for the work of the robot arm 50 and instrument 2, and determination regarding the accuracy of the work of the robot arm 50 and instrument 2. In this way, when determining the predicted remaining time for the work of the robot arm 50 and instrument 2 as one of the multiple types of risk elements, the predicted remaining time for the work of the robot arm 50 and instrument 2 is related to the occurrence of risks such as abnormalities or signs of abnormalities caused by the operation of the robot arm 50 and instrument 2 in autonomous mode. Therefore, based on the predicted remaining time for the work of the robot arm 50 and instrument 2, it is possible to appropriately determine whether risks such as abnormalities or signs of abnormalities caused by the operation of the robot arm 50 and instrument 2 in autonomous mode have occurred. For example, if the predicted remaining time for the robot arm 50 and instrument 2 does not change for a predetermined period of time, if the predicted remaining time for the robot arm 50 and instrument 2 increases rapidly, if the predicted remaining time for the robot arm 50 and instrument 2 is excessively large, or if the predicted remaining time for the robot arm 50 and instrument 2 is excessively small, it can be inferred that a risk such as an abnormality or a precursor to an abnormality has occurred due to the operation of the robot arm 50 and instrument 2 in autonomous mode. Furthermore, when determining the accuracy of the work performed by the robot arm 50 and instrument 2 as part of the assessment of multiple types of risk factors, the accuracy of the work performed by the robot arm 50 and instrument 2 is related to the occurrence of risks such as abnormalities or signs of abnormalities caused by the operation of the robot arm 50 and instrument 2 in autonomous mode. Therefore, based on the accuracy of the work performed by the robot arm 50 and instrument 2, it is possible to appropriately determine whether risks such as abnormalities or signs of abnormalities caused by the operation of the robot arm 50 and instrument 2 in autonomous mode have occurred.For example, if the accuracy of the work performed by the robot arm 50 and instrument 2 is excessively low, or if the accuracy of the work performed by the robot arm 50 and instrument 2 decreases rapidly, it can be inferred that a risk such as an abnormality or a precursor to an abnormality has occurred due to the operation of the robot arm 50 and instrument 2 in autonomous mode.
[0135] The risk determination control device 700 uses at least one trained model to determine multiple types of risk factors. This makes it possible to easily and accurately determine multiple types of risk factors using at least one trained model.
[0136] The surgical support system 500 includes a surgical support robot 100, which includes a plurality of robotic arms 50 to which surgical instruments 1 are attached at their tips, and a remote control device 200, which includes an operating unit 110 that receives operations on the surgical instruments 1. This allows the surgical support system 500 to appropriately respond to risks such as abnormalities or signs of abnormalities caused by the operation of the robotic arms 50 and instruments 2 in autonomous mode.
[0137] The multiple robot arms 50 include a robot arm 50c to which an endoscope 3, which serves as a surgical instrument 1, is attached as a tip, and robot arms 50a, 50b, and 50d to which instruments 2, which are surgical instruments 1 other than the endoscope 3, are attached as tips. The endoscope 3 on robot arm 50c and the instruments 2 on robot arms 50a, 50b, and 50d enable smooth surgery on patient P.
[0138] The autonomous mode operation includes suturing to suture the surgical site of patient P. This allows for appropriate responses to risks such as abnormalities or signs of abnormalities caused by the operation of the autonomous robot arm 50 and instrument 2 during the suturing of patient P's surgical site.
[0139] [Variations] It should be noted that the embodiments disclosed herein are illustrative and not restrictive in all respects. The scope of this disclosure is indicated by the claims rather than the description of the embodiments above, and further includes all modifications or variations within the meaning and scope equivalent to the claims.
[0140] In the above embodiment, the first control device 310 is shown to perform risk response actions such as issuing a warning, stopping the robot arm 50 and instrument 2, or switching from autonomous mode to remote control mode, but the disclosure is not limited thereto. In this disclosure, the first control device 310 may perform risk response actions other than issuing a warning, stopping the robot arm 50 and instrument 2, or switching from autonomous mode to remote control mode. In addition to issuing a warning, stopping the robot arm 50 and instrument 2, or switching from autonomous mode to remote control mode, the first control device 310 may perform risk response actions other than those listed above. Furthermore, the first control device 310 may perform multiple types of risk response actions other than issuing a warning, stopping the robot arm 50 and instrument 2, or switching from autonomous mode to remote control mode.
[0141] In the above embodiment, an example was shown in which the first control device 310 and the risk determination control device 700 are arranged separately, but the disclosure is not limited thereto. For example, the first control device 310 may also function as the risk determination control device 700. This simplifies the configuration of the surgical support system 500. Alternatively, the first control device 310 and the risk determination control device 700 may be configured to share the responsibility of determining multiple types of risk elements. This distributes the control burden between the first control device 310 and the risk determination control device 700. Furthermore, one risk determination control device 700 may be configured to determine all types of risk elements, or multiple risk determination control devices 700 may be configured to share the responsibility of determining multiple types of risk elements.
[0142] In the above embodiment, an example was shown in which both the left and right robot arms 50 and the instruments 2 attached thereto operate autonomously in autonomous mode, but the disclosure is not limited thereto. For example, in autonomous mode, one of the left and right robot arms 50 and the instrument 2 attached thereto may operate autonomously. In this case, a switching unit for switching between autonomous mode and remote control mode is provided for each of the left and right robot arms 50. The operator only needs to operate one of the robot arms 50 and the instrument 2 attached thereto, thus reducing the burden on the operator.
[0143] In the above embodiment, the inference unit 613 is shown to take observation information and operation commands from multiple past time points as input and output the next inference operation command, but the disclosure is not limited to this. For example, the inference unit 613 may take observation information and operation commands from one past time point as input and output the next inference operation command. This reduces the control burden on the inference control device 600.
[0144] In the above embodiment, an example was shown in which an image G captured by the endoscope 3 is input to the inference unit 613 as observation information, but the disclosure is not limited to this. For example, a camera may be placed on the ceiling or wall of the operating room, and the image G captured by this camera may be input to the inference unit 613 as observation information. This makes it possible to generate inference operation commands that take into account the environment around the patient P and the robot arm 50. Alternatively, a force sensor may be placed as an environmental information acquisition unit instead of the endoscope 3. This makes it possible to detect interference between the instrument 2 and the surgical area of the patient P using the force sensor, and thus generate inference operation commands that suppress interference.
[0145] In the above embodiment, an example was shown in which the orientation of the multiple link parts of the operating unit 110 is changed to follow the orientation of the robot arm 50 and the instrument 2 in autonomous mode, but the disclosure is not limited thereto. For example, in autonomous mode, the orientation of the multiple link parts of the operating unit 110 may remain stationary without being changed to follow the orientation of the robot arm 50 and the instrument 2. This reduces the control burden on the first control device 310.
[0146] In the above embodiment, an example was shown in which the robot arm 50 and the instrument 2 attached thereto operate in autonomous mode, but the disclosure is not limited thereto. For example, the robot arm 50 may operate in autonomous mode, and the instrument 2 may operate in remote control mode. This allows the operator to concentrate on operating the instrument 2.
[0147] In the above embodiment, an example of applying the present disclosure to a surgical support system 500 comprising a surgical support robot 100 and a remote control device 200 was shown, but the present disclosure is not limited thereto. For example, as shown in the first modified example in Figure 35, the present disclosure may be applied to a manipulation system 800. The manipulation system 800 comprises a social robot 810, a remote control device 820, and a control device 830. The social robot 810 is equipped with an interface for interaction and is capable of communicating with people. The social robot 810 also includes a pair of articulated robot arms 811 and a trolley 812. The pair of articulated robot arms 811 can be switched between a remote control mode, which operates based on the operation of the remote control device 820, and an autonomous mode, which operates autonomously. The remote control device 820 accepts operation from the operator. When the robot arms 811 are operating in autonomous mode, the control device 830 determines multiple types of risk factors and takes different risk responses based on the determination results of the multiple types of risk factors. This allows the manipulation system 800, including the social robot 810, to appropriately respond to risks such as abnormalities or signs of abnormalities caused by the operation of the robot arm 811 in autonomous mode. The robot arm 811 is an example of a manipulation device. The remote control device 820 is an example of an operating device.
[0148] Furthermore, for example, as shown in the second modified example in Figure 36, the present disclosure may be applied to a manipulation system 900. The manipulation system 900 comprises an industrial robot 910 placed in a factory or the like, a remote control device 920, and a control device 930. The industrial robot 910 includes, for example, a vertical articulated robot 911 and a horizontal articulated robot 912. The industrial robot 910 can switch between a remote control mode, which operates based on the operation of the remote control device 920, and an autonomous mode, which operates autonomously. The remote control device 920 accepts operations from an operator. When the industrial robot 910 is operating in autonomous mode, the control device 930 determines multiple types of risk elements and takes different risk responses based on the determination results of the multiple types of risk elements. This makes it possible to appropriately respond to risks such as abnormalities or signs of abnormalities caused by the operation of the industrial robot 910 in autonomous mode in the manipulation system 900 including the industrial robot 910. Note that the industrial robot 910 is an example of a manipulation device. Furthermore, the remote control device 920 is an example of an operating device.
[0149] The above embodiment shows an example in which multiple robot arms 50 are arranged, but the disclosure is not limited thereto. For example, the disclosure may be applied to a surgical support system in which only one robot arm 50 is arranged. In this case, either the robot arm 50 or the instrument 2 can be switched between autonomous mode and remote control mode. This makes it possible to smoothly switch between autonomous mode and remote control mode even in a surgical support system in which only one robot arm 50 is arranged.
[0150] In the above embodiment, an example was shown in which the operation command included a suturing operation command for suturing the surgical site of patient P, but this disclosure is not limited thereto. For example, the operation command may be a command other than a suturing operation command. For example, the operation command may be an excision command for excising the surgical site of patient P. That is, the operation in autonomous mode may include an excision operation for excising the surgical site of patient P. Furthermore, the operation command may include multiple types of surgical operation commands, such as suturing operation commands and excision commands. That is, the operation in autonomous mode may include multiple types of surgical operations, such as suturing operations and excision operations. In this way, this disclosure makes it possible to appropriately respond to risks such as abnormalities or signs of abnormalities caused by the operation of the autonomous robot arm 50 and instrument 2, even in surgical operations other than suturing operations.
[0151] In the above embodiment, an example was shown in which the switching unit for switching between autonomous mode and remote control mode is a coagulation pedal 126L and another foot pedal 120, but the present disclosure is not limited thereto. For example, the switching unit of the present disclosure may be a switching unit for switching between autonomous mode and remote control mode by detecting the movement of the operator's gaze. In this case, for example, a sensor for detecting the movement of the operator's gaze is arranged on the monitor 140. Alternatively, the switching unit of the present disclosure may be a switching unit for switching between autonomous mode and the remote control mode based on the detection of a part of the operator's body. In this case, a sensor for detecting the movement of a part of the operator's body is arranged. For example, the sensor detects the movement of the operator's arm or head. By switching between autonomous mode and remote control mode with such a switching unit, the operator does not need to take their hands off the control unit to switch between autonomous mode and remote control mode, so that the operator can switch between autonomous mode and remote control mode while continuing to perform suturing work. Alternatively, the switching unit of the present disclosure may be a button separately arranged on the control unit 110. This allows the buttons to be operated manually, making it easy to switch between autonomous mode and the remote control mode.
[0152] In the above embodiment, an example was shown in which, in autonomous mode, a manual operation command is replaced with an inference operation command before the limiting unit of the first control device 310 limits the speed of the manual operation command. However, the disclosure is not limited thereto. For example, in autonomous mode, a manual operation command may be replaced with an inference operation command before the limiting unit of the first control device 310 limits the acceleration of the manual operation command. This makes it possible to limit the acceleration of the operation of the surgical support system in the same way in both autonomous mode and remote control mode.
[0153] In the above embodiment, an example was shown in which the autonomous mode is executed while the operator is pressing the coagulation pedal 126L and the remote control mode is executed while the operator is releasing the coagulation pedal 126L, but the disclosure is not limited thereto. For example, conversely to the above operation, the remote control mode may be executed while the coagulation pedal 126L is pressed, and the autonomous mode may be executed while the coagulation pedal 126L is released. Alternatively, the autonomous mode and the remote control mode may be switched each time the coagulation pedal 126L is pressed. In this way, even if the operation for switching between the autonomous mode and the remote control mode is different, the autonomous mode and the remote control mode can be easily switched with the simple operation of pressing the coagulation pedal 126L. Furthermore, depending on the learning proficiency of the inference unit 613, initially the autonomous mode is executed while the operator is pressing the solidification pedal 126L, and the remote control mode is executed while the operator is releasing the solidification pedal 126L. Once the proficiency exceeds a predetermined standard, the remote control mode may be executed while the solidification pedal 126L is pressed, and the autonomous mode may be executed while the solidification pedal 126L is released.
[0154] In the above embodiment, an example was shown in which the coagulation pedal 126L switches between autonomous mode and remote control mode, and the coagulation pedal 126R switches between starting and stopping the storage operation of training data. However, this disclosure is not limited to this. For example, the above switching may be performed by detecting the movement of the operator's gaze or by detecting a part of the operator's body other than their hands. This allows the above switching to be performed while the operator continues to operate the control unit 110.
[0155] In the above embodiment, an example was shown in which the first control device 310 and the inference control device 600 are arranged separately, but the disclosure is not limited thereto. For example, the first control device 310 may also serve as the inference control device 600. This simplifies the configuration of the surgical support system 500.
[0156] Furthermore, although the above embodiment shows an example in which four robot arms 50 are provided, this disclosure is not limited to this. In this disclosure, the number of robot arms 50 may be other than four.
[0157] Furthermore, although the above embodiment shows an example in which the arm portion 51 and the positioner 30 are composed of a 7-axis articulated robot, this disclosure is not limited to this. For example, the arm portion 51 and the positioner 30 may be composed of an articulated robot with an axis configuration other than a 7-axis articulated robot. An axis configuration other than a 7-axis articulated robot would be, for example, 6 axes or 8 axes.
[0158] Furthermore, while the above embodiment shows an example in which the surgical assistance robot 100 includes a medical cart 10, a positioner 30, an arm base 40, and a robot arm 50, this disclosure is not limited to this. For example, the medical cart 10, the positioner 30, and the arm base 40 are not necessarily required, and the surgical assistance robot 100 may include only the robot arm 50.
[0159] The functions of the elements disclosed herein may be implemented using one or more circuits or processing circuits, including general-purpose processors, special-purpose processors, integrated circuits, ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and / or conventional circuits. The functions of the elements disclosed herein may be implemented using one or more circuits or processing circuits, including combinations of general-purpose processors, special-purpose processors, integrated circuits, ASICs, FPGAs, and conventional circuits. One or more circuits or processing circuits may be programmed using one or more programs stored together or individually in one or more memories, or otherwise configured to perform the disclosed functions. A processor is considered a processing circuit or circuit because it includes transistors and other circuits. A processor may be a programmed processor that executes programs stored in memory. In this disclosure, a circuit, unit, or means is hardware that performs the enumerated functions individually or in combination with each other, or hardware programmed to perform the enumerated functions individually or in combination with each other. Hardware may be any hardware disclosed herein that is programmed or configured to perform the enumerated functions. A computer program, including computer instructions, is stored in memory. Computer instructions provide logic and routines that enable the hardware to perform the methods disclosed herein. Hardware includes, for example, processing circuits or circuits. A computer program may be implemented in known formats on computer-readable storage media, computer program products, memory devices, recording media such as CD-ROMs or DVDs, and / or in the memory of FPGAs or ASICs.
[0160] [Embodiments] The exemplary embodiments described above will be understood by those skilled in the art to be specific examples of the following embodiments.
[0161] (Aspect 1) A manipulation system comprising: an operating device that accepts operations from an operator; a manipulation device that can switch between a remote operation mode that operates based on the operation of the operating device and an autonomous mode that operates autonomously; and a control device that, when the manipulation device is operating in the autonomous mode, determines multiple types of risk elements and performs different risk responses based on the determination results of the multiple types of risk elements.
[0162] (Aspect 2) The manipulation system according to aspect 1, wherein the control device performs at least one of the following actions as a risk response based on the determination result of the plurality of types of risk elements: issuing a warning, stopping the manipulation device, and switching from the autonomous mode to the remote control mode.
[0163] (Aspect 3) The manipulation system according to aspect 2, wherein the manipulation device includes a plurality of manipulation devices, the plurality of manipulation devices are switchable between the autonomous mode and the remote control mode, and the control device switches at least one of the plurality of manipulation devices from the autonomous mode to the remote control mode as a risk response based on the determination result of the plurality of types of risk elements.
[0164] (Aspect 4) The manipulation system according to aspect 2 or aspect 3, wherein a portion of the manipulation device is switchable between the autonomous mode and the remote control mode, and the control device switches a portion of the manipulation device from the autonomous mode to the remote control mode as a risk response based on the determination results of the plurality of types of risk elements.
[0165] (Aspect 5) The manipulation system according to any one of aspects 1 to 4, wherein the control device, based on the determination results of the plurality of types of risk elements, switches the manipulation device from the autonomous mode to the remote control mode as a risk response, either automatically switches the manipulation device from the autonomous mode to the remote control mode or provides a notification prompting the manipulation device to switch from the autonomous mode to the remote control mode.
[0166] (Aspect 6) A manipulation system according to any one of aspects 1 to 5, comprising a trained model that takes observational information of the environment surrounding the manipulation device and operation commands of the manipulation device as inputs and outputs inferred operation commands which are inferred operation commands, wherein the manipulation device operates in autonomous mode based on the inferred operation commands output from the trained model.
[0167] (Aspect 7) The manipulation system according to aspect 6, wherein the control device performs at least one of the following as determination of the plurality of types of risk elements: determination of whether the input image input to the trained model as observation information is an input image that deviates from the input image at the time the trained model was trained; and determination of whether the output operation output by the trained model as an inference operation command is an output operation that deviates from the output operation at the time the trained model was trained.
[0168] (Aspect 8) The manipulation system according to aspect 7, wherein the control device determines whether the output operation output by the trained model belongs to one of a plurality of clusters created in advance based on the pattern of output operations output by the trained model, thereby determining whether the output operation output by the trained model as an inference operation command is an output operation that deviates from the output operation when the trained model was trained.
[0169] (Aspect 9) The manipulation system according to any one of aspects 1 to 8, wherein the control device performs at least one of the following as a determination of the plurality of types of risk elements: a determination of the predicted remaining time for the work of the manipulation device and a determination of the accuracy of the work of the manipulation device.
[0170] (Aspect 10) The control device determines the plurality of risk elements using at least one trained model, according to the manipulation system according to any one of aspects 1 to 9.
[0171] (Aspect 11) A manipulation system according to any one of aspects 1 to 10, comprising: a surgical apparatus including a robotic arm as a manipulation device to which a surgical instrument is attached at its tip; and an operating device including an operating section for receiving operations on the surgical instrument.
[0172] (Aspect 12) The manipulation system according to aspect 11, wherein the robot arm includes a first robot arm to which an endoscope as a surgical instrument is attached as a tip, and a second robot arm to which a surgical instrument other than the endoscope is attached as a tip.
[0173] (Aspect 13) The manipulation system according to aspect 12, wherein the operation of the autonomous mode includes a suturing operation for suturing the surgical site of a patient.
[0174] (Aspect 14) A control method for a manipulation system comprising an operating device that accepts operations from an operator, and a manipulation device that can switch between a remote operation mode that operates based on the operation of the operating device and an autonomous mode that operates autonomously, the control method for a manipulation system comprising: determining a plurality of types of risk elements when the manipulation device is operating in the autonomous mode, and performing different risk responses based on the determination results of the plurality of types of risk elements.
[0175] (Aspect 15) A program for executing a control method for a manipulation system comprising an operating device that accepts operations from an operator, a manipulation device that can switch between a remote operation mode that operates based on the operation of the operating device, and an autonomous mode that operates autonomously, the program for executing a control method for a manipulation system comprising: determining a plurality of types of risk elements when the manipulation device is operating in the autonomous mode, and performing different risk responses based on the determination results of the plurality of types of risk elements.
[0176] 1 Surgical instruments 2 Instruments (manipulation devices) 3 Endoscope 50 Robot arm (manipulation device) 50a Robot arm (second robot arm) 50b Robot arm (second robot arm) 50c Robot arm (first robot arm) 50d Robot arm (second robot arm) 100 Surgical support robot (surgical device) 110 Control unit 200 Remote control device (control device) 310 First control device (control device) 311a Program 500 Surgical support system (manipulation system) 700 Risk assessment control device (control device) 721 Program 800 Manipulation system 811 Robot arm (manipulation device) 820 Remote control device (control device) 830 Control device 900 Manipulation system 910 Industrial robot (manipulation device) 920 Remote control device (control device) 930 Control device P patient
Claims
1. A manipulation system comprising: an operating device that accepts input from an operator; a manipulation device that can switch between a remote operation mode that operates based on the operation of the operating device and an autonomous mode that operates autonomously; and a control device that, when the manipulation device is operating in the autonomous mode, determines multiple types of risk factors and performs different risk responses based on the determination results of the multiple types of risk factors.
2. The manipulation system according to claim 1, wherein the control device performs at least one of the following actions as a risk response based on the determination result of the plurality of types of risk elements: issuing a warning, stopping the manipulation device, and switching from the autonomous mode to the remote control mode.
3. The manipulation system according to claim 2, wherein the manipulation device includes a plurality of manipulation devices, the plurality of manipulation devices are switchable between the autonomous mode and the remote control mode, and the control device switches at least one of the plurality of manipulation devices from the autonomous mode to the remote control mode as a risk response based on the determination result of the plurality of types of risk elements.
4. The manipulation system according to claim 2, wherein a portion of the manipulation device is switchable between the autonomous mode and the remote control mode, and the control device switches a portion of the manipulation device from the autonomous mode to the remote control mode as a risk response based on the determination results of the plurality of types of risk elements.
5. The manipulation system according to claim 1, wherein, based on the determination results of the plurality of types of risk elements, the control device either automatically switches the manipulation device from the autonomous mode to the remote control mode as a risk response, or provides a notification prompting the control device to switch from the autonomous mode to the remote control mode.
6. The manipulation system according to claim 1, comprising a trained model that takes observational information of the environment surrounding the manipulation device and operation commands of the manipulation device as inputs and outputs inferred operation commands which are inferred operation commands, wherein the manipulation device operates in autonomous mode based on the inferred operation commands output from the trained model.
7. The manipulation system according to claim 6, wherein the control device performs at least one of the following as determination of the plurality of types of risk elements: determination of whether the input image input to the trained model as observation information is an input image that deviates from the input image used when the trained model was trained; and determination of whether the output operation output by the trained model as an inference operation command is an output operation that deviates from the output operation used when the trained model was trained.
8. The manipulation system according to claim 7, wherein the control device determines whether the output operation output by the trained model belongs to one of a plurality of clusters created in advance based on the pattern of output operations output by the trained model, thereby determining whether the output operation output by the trained model as an inference operation command is an output operation that deviates from the output operation performed by the trained model during training.
9. The manipulation system according to claim 1, wherein the control device performs at least one of the following as a determination of the plurality of types of risk elements: a determination of the predicted remaining time for the work of the manipulation device, and a determination of the accuracy of the work of the manipulation device.
10. The control device determines the plurality of risk elements using at least one trained model, as described in claim 1.
11. The manipulation system according to claim 1, comprising: a surgical apparatus including a robotic arm as a manipulation device to which a surgical instrument is attached at its tip; and an operating device including an operating section for receiving operations on the surgical instrument.
12. The manipulation system according to claim 11, wherein the robotic arm includes a first robotic arm to which an endoscope as a surgical instrument is attached as a tip, and a second robotic arm to which a surgical instrument other than the endoscope is attached as a tip.
13. The manipulation system according to claim 12, wherein the operation of the autonomous mode includes suturing operations for suturing the surgical site of a patient.
14. A control method for a manipulation system comprising an operating device that accepts operations from an operator, a manipulation device that can switch between a remote operation mode that operates based on the operation of the operating device, and an autonomous mode that operates autonomously, the method comprising: determining a plurality of types of risk elements when the manipulation device is operating in the autonomous mode; and performing different risk responses based on the determination results of the plurality of types of risk elements.
15. A program for executing a control method for a manipulation system comprising an operating device that accepts operator input, a manipulation device that can switch between a remote operation mode that operates based on the operation of the operating device, and an autonomous mode that operates autonomously, the program comprising: determining multiple types of risk elements when the manipulation device is operating in the autonomous mode; and performing different risk responses based on the determination results of the multiple types of risk elements.
