Manipulation system, control method for manipulation system, and program

The manipulation system addresses operability issues by using a control device to manage transitions between autonomous and remote modes, ensuring stable command values and improved user experience.

WO2026240047A1PCT designated stage Publication Date: 2026-11-19KAWASAKI JUKOGYO KK
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2026/017712
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-16
Filing Date
2026-05-11
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Existing manipulation systems experience sudden changes in command values and deteriorating operability due to mismatches between operation units and manipulation devices when switching from autonomous to remote operation modes.

Method used

A manipulation system with a control device that acquires the state of the operating unit and determines whether the manipulation device can be switched from autonomous to remote control mode, thereby preventing sudden changes in command values and improving operability.

Benefits of technology

The system effectively suppresses the deterioration of operability by ensuring smooth transitions between autonomous and remote operation modes, minimizing sudden changes in command values.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2026017712_19112026_PF_FP_ABST
    Figure JP2026017712_19112026_PF_FP_ABST
Patent Text Reader

Abstract

This manipulation system (500) comprises: manipulation devices (50, 2) capable of switching between a remote operation mode, in which the manipulation devices run on the basis of operations of an operation unit (110) of an operation device (200), and an autonomous mode, in which the manipulation devices run autonomously; and a control device (310) that acquires the state of the operation unit (110) and performs control for determining, on the basis of the acquired state of the operation unit (110), whether the state of the operation unit (110) is a state in which the manipulation devices (50, 2) can be switched from the autonomous mode to the remote operation mode.
Need to check novelty before this filing date? Find Prior Art

Description

Manipulation system, control method and program for manipulation system

[0001] The present disclosure relates to a manipulation system, a control method and a program for the manipulation system.

[0002] Conventionally, a manipulation system including an operation device and a manipulation device is known. Japanese Patent No. 7334499 discloses a surgery support system as a manipulation system, which includes an operation device and a surgical manipulator as the manipulation device. In this surgery support system, the surgical manipulator is operated by displacing the position and orientation of the operation device. Further, in this surgery support system, the surgical manipulator can be switched between an autonomous drive mode and a manual operation mode.

[0003] Japanese Patent No. 7334499

[0004] In the surgery support system described in Japanese Patent No. 7334499, the surgical manipulator can be switched between the autonomous drive mode and the manual operation mode. Here, when switching the surgical manipulator from the autonomous drive mode to the manual operation mode, if there is a mismatch between the operation device and the surgical manipulator, a sudden change in command value may occur, leading to deterioration of operability. Further, when switching a manipulation device other than the surgical manipulator from the autonomous mode to the remote operation mode, similarly, if there is a mismatch between the operation unit and the manipulation device, a sudden change in command value may occur, leading to deterioration of operability. Therefore, when switching the manipulation device from the autonomous mode to the remote operation mode, it is desired to suppress the occurrence of a sudden change in command value and the resulting deterioration of operability caused by the mismatch between the operation unit and the manipulation device.

[0005] The present disclosure is to provide a manipulation system, a control method and a program for the manipulation system, which are capable of suppressing the occurrence of a sudden change in command value and the resulting deterioration of operability caused by the mismatch between the operation unit and the manipulation device.

[0006] A manipulation system according to the first aspect of this disclosure includes: an operating device including an operating unit that accepts operations from an operator; a manipulation device that can switch between a remote operation mode that operates based on operations from the operating unit of the operating device and an autonomous mode that operates autonomously; and a control device that acquires the state of the operating unit and, based on the acquired state of the operating unit, performs control to determine whether the state of the operating unit allows the manipulation device to be switched from autonomous mode to remote operation mode.

[0007] The first aspect of the present disclosure of the manipulation system includes a control device that acquires the state of an operating unit and, based on the acquired state of the operating unit, determines whether the state of the operating unit allows the manipulation device to be switched from autonomous mode to remote control mode. As a result, when the state of the operating unit allows the manipulation device to be switched from autonomous mode to remote control mode, the manipulation device can be switched from autonomous mode to remote control mode. Consequently, when switching the manipulation device from autonomous mode to remote control mode, it is possible to suppress the sudden change in command values ​​and deterioration of operability caused by a mismatch between the operating unit and the manipulation device.

[0008] A control method for a manipulation system according to the second aspect of this disclosure comprises an operating device including an operating unit that accepts operations from an operator, and a manipulation device that can switch between a remote operation mode that operates based on operations from the operating unit of the operating device and an autonomous mode that operates autonomously, the method comprising: acquiring the state of the operating unit, and determining, based on the acquired state of the operating unit, whether the state of the operating unit allows the manipulation device to be switched from autonomous mode to remote operation mode.

[0009] The control method for a manipulation system according to the second aspect of this disclosure comprises acquiring the state of an operating unit and determining, based on the acquired state of the operating unit, whether or not the state of the operating unit allows the manipulation device to be switched from autonomous mode to remote control mode. As a result, if the state of the operating unit allows the manipulation device to be switched from autonomous mode to remote control mode, the manipulation device can be switched from autonomous mode to remote control mode. Consequently, a control method for a manipulation system is provided that can suppress the deterioration of operability caused by a sudden change in command values ​​due to a mismatch between the operating unit and the manipulation device when switching the manipulation device from autonomous mode to remote control mode.

[0010] A program according to the third aspect of this disclosure is a program that executes a control method for a manipulation system comprising an operating device including an operating unit that accepts operator input, and a manipulation device that can switch between a remote operation mode that operates based on the operation of the operating unit of the operating device and an autonomous mode that operates autonomously, the program which executes a control method for a manipulation system comprising: acquiring the state of the operating unit, and determining, based on the acquired state of the operating unit, whether or not the state of the operating unit allows the manipulation device to be switched from autonomous mode to remote operation mode.

[0011] A program according to the third aspect of this disclosure executes a control method for a manipulation system, comprising: acquiring the state of an operating unit; and determining, based on the acquired state of the operating unit, whether the state of the operating unit allows the manipulation device to be switched from autonomous mode to remote control mode. As a result, if the state of the operating unit allows the manipulation device to be switched from autonomous mode to remote control mode, the manipulation device can be switched from autonomous mode to remote control mode. Consequently, a program is provided that can suppress the deterioration of operability caused by a sudden change in command values ​​due to a mismatch between the operating unit and the manipulation device when switching the manipulation device from autonomous mode to remote control mode.

[0012] According to this disclosure, it is possible to suppress the deterioration of operability caused by sudden changes in command values ​​due to a mismatch between the operating unit and the manipulation device.

[0013] This is a diagram showing the configuration of the surgical support system according to the first embodiment. This is a diagram showing the display unit of the medical trolley according to the first embodiment. This is a diagram showing the configuration of the medical trolley according to the first embodiment. This is a diagram showing the configuration of the robot arm according to the first embodiment. This is a diagram showing the instruments. This is a perspective view showing the configuration of the arm operation unit according to the first embodiment. This is a diagram for explaining the translational movement of the robot arm. This is a diagram for explaining the rotational movement of the robot arm. This is a diagram showing the endoscope. This is a diagram showing the pivot position setting device. This is a diagram showing the operation unit according to the first embodiment. This is a diagram showing the right-hand wrist unit according to the first embodiment. This is a diagram showing the left-hand wrist unit according to the first embodiment. This is a perspective view showing the foot pedal according to the first embodiment. This is a control block diagram of the surgical support system according to the first embodiment. This is a control block diagram of the robot arm according to the first embodiment. This is a control block diagram of the positioner and medical trolley according to the first embodiment. This is a control block diagram of the operation unit according to the first embodiment. This is a diagram for explaining the control of the first control device according to the first embodiment. This is a diagram for explaining the configuration of the inference control device according to the first embodiment. This is a diagram showing the patient's surgical unit and instruments displayed on the monitor and display unit. This is a flowchart for explaining the autonomous mode of the surgical support system according to one embodiment. This diagram illustrates how to determine whether the operating unit is in a state where it can be switched from autonomous mode to remote control mode. This diagram illustrates how to make the operating unit follow the operation of the manipulation device. This flowchart illustrates the control method of the surgical support system according to the first embodiment. This diagram shows the configuration of the manipulation system according to the second embodiment. This diagram illustrates the display of the display device according to the second embodiment. This flowchart illustrates the control method of the manipulation system according to the second embodiment. This diagram shows an industrial robot as a modified example.

[0014] [First Embodiment] (Configuration of Surgical Support System) The configuration of the surgical support system 500 according to the first embodiment will be described. The surgical support system 500 includes a surgical support robot 100, a remote control device 200, a vision unit 300, an image processing unit 400, and an inference control device 600. The surgical support robot 100 is an example of a surgical device. The remote control device 200 is an example of an operating device. The surgical support system 500 is an example of a manipulation system.

[0015] In this specification, as shown in Figure 4, the longitudinal direction of the surgical instrument 1 is defined as the Z direction. The tip side of the surgical instrument 1 is defined as the Z1 side, and the base side of the surgical instrument 1 is defined as the Z2 side. The direction perpendicular to the Z direction is defined as the X direction. The direction perpendicular to both the Z and X directions is defined as the Y direction.

[0016] As shown in Figure 1, the surgical robot 100 is positioned inside the operating room. The remote control device 200 is positioned at a distance from the surgical robot 100. The remote control device 200 and the surgical robot 100 may communicate via a wired connection or wirelessly. The remote control device 200 receives commands for the surgical instrument 1. Specifically, an operator, such as a physician, inputs commands to the remote control device 200 to cause the surgical robot 100 to perform a desired action. The remote control device 200 transmits the input commands to the surgical robot 100. The surgical robot 100 operates based on the received commands. The surgical robot 100 is positioned inside the operating room, which is a sterile field.

[0017] (Configuration of the surgical support robot) As shown in Figure 1, the surgical support robot 100 comprises a medical trolley 10, a trolley positioner operating unit 20, a positioner 30, an arm base 40, a plurality of robot arms 50, and an arm operating unit 60 provided on each robot arm 50. The plurality of robot arms 50 may be arranged on an operating table on which the patient P is placed.

[0018] As shown in Figure 3, the trolley positioner operation unit 20 is supported by a trolley positioner operation support unit 21 at the rear of the medical trolley 10, and the medical trolley 10 or positioner 30 is moved when the trolley positioner operation unit 20 is operated. The trolley positioner operation unit 20 includes an input device 22 and an operation handle 23. The input device 22 accepts operations for moving and changing the posture of the positioner 30, arm base 40, and multiple robot arms 50, mainly for preparing for surgery before the procedure. The medical trolley 10 includes an operation handle 23.

[0019] As shown in Figure 3, the input device 22 includes a display unit 22a, a joystick 22b, an enable switch 22c, an error reset button 22d, and a speaker 22e. The display unit 22a is, for example, a liquid crystal panel. As shown in Figure 2, the display unit 22a displays numbers corresponding to multiple robot arms 50. The display unit 22a also displays the type of surgical instrument 1 attached to each of the multiple robot arms 50. The display unit 22a displays a check mark CM indicating that the pivot position PP, which will be described later, has been set.

[0020] As shown in Figure 3, the joystick 22b is positioned near the display unit 22a of the input device 22. By selecting an operation mode displayed on the display unit 22a and operating the joystick 22b, the positioner 30 is moved in three dimensions.

[0021] The enable switch 22c is located near the joystick 22b. The enable switch 22c allows or disallows the movement of the positioner 30. When the enable switch 22c is pressed and the movement of the positioner 30 is permitted, the joystick 22b is operated, causing the positioner 30 to move.

[0022] The error reset button 22d clears errors in the surgical support system 500. The errors are, for example, deviation anomaly errors. A pair of speakers 22e are provided. The pair of speakers 22e are located near the location where the positioner 30 of the medical trolley 10 is positioned.

[0023] Furthermore, the operating handle 23 is located near the display unit 22a. The operating handle 23 has a throttle 23a that controls the movement of the medical cart 10 when grasped and rotated by an operator such as a nurse or technician. Specifically, the operating handle 23 is located below the input device 22. When the throttle 23a is rotated from the front to the back, the medical cart 10 moves forward. When the throttle 23a is rotated from the back to the front, the medical cart 10 moves backward. The speed of the medical cart 10 is also changed according to the amount of rotation of the throttle 23a. The operating handle 23 is configured to be rotatable left and right, as indicated by the R direction, and the medical cart 10 rotates along with the rotation of the operating handle 23.

[0024] Furthermore, an enable switch 23b is located on the operating handle 23 to allow or disallow the movement of the medical trolley 10. When the enable switch 23b is pressed and the movement of the medical trolley 10 is permitted, the throttle 23a on the operating handle 23 is operated, causing the medical trolley 10 to move.

[0025] As shown in Figure 1, the positioner 30 consists of a 7-axis articulated robot. The positioner 30 is positioned on a medical trolley 10. The positioner 30 is a 7-axis articulated robot that adjusts the position of the arm base 40. The positioner 30 moves the position of the arm base 40 in three dimensions. However, the positioner 30 is not limited to a 7-axis articulated robot; for example, it may be an articulated robot with 6 axes or less, or 8 axes or more.

[0026] The positioner 30 includes a base portion 31 and a plurality of link portions 32 connected to the base portion 31. The plurality of link portions 32 are connected to each other by joints 33.

[0027] The arm base 40 is attached to the tip of the positioner 30. The base of each of the multiple robot arms 50 is attached to the arm base 40. The multiple robot arms 50 can be folded and stored. The arm base 40 and the multiple robot arms 50 are used covered with sterile drapes. The robot arms 50 also support surgical instruments 1.

[0028] The arm base 40 is equipped with a status indicator 41 and an arm status indicator 42, as shown in Figure 15. The status indicator 41 displays the status of the surgical support system 500. The arm status indicator 42 displays the status of the robot arm 50.

[0029] Multiple robot arms 50 are arranged. Specifically, four robot arms 50a, 50b, 50c, and 50d are arranged. Robot arms 50a, 50b, 50c, and 50d have similar configurations to each other. Robot arms 50a, 50b, 50c, and 50d are operated independently based on motion commands. Note that robot arm 50 is an example of a multi-joint robot arm. Also, robot arms 50a, 50b, and 50d are examples of second robot arms. Also, robot arm 50c is an example of a first robot arm.

[0030] As shown in Figure 4, the robot arm 50 includes an arm section 51, a first link section 52, a second link section 53, and a translational movement mechanism section 54. The robot arm 50 has joints JT1, JT2, JT3, JT4, JT5, JT6, JT7, and JT8. Joints JT1, JT2, JT3, JT4, JT5, JT6, and JT7 each have axes A1, A2, A3, A4, A5, A6, and A7 as rotation axes. JT8 has an axis A8 as a linear motion axis. The arm section 51 includes a base section 51a and a link section 51b.

[0031] The arm section 51 consists of a 7-axis articulated robot arm. The first link section 52 is located at the tip of the arm section 51. The arm operation section 60, which will be described later, is attached to the second link section 53. The translational movement mechanism section 54 is located between the first link section 52 and the second link section 53. A holder 55 for holding the surgical instrument 1 is located on the second link section 53. The translational movement mechanism section 54 translates the holder 55, to which the surgical instrument 1 is attached, between a first position and a second position. The first position is the position of the end of the range of movement of the holder 55 by the translational movement mechanism section 54 along the A8 axis, on the Z2 side. The second position is the position of the end of the range of movement of the holder 55 by the translational movement mechanism section 54 along the A8 axis, on the Z1 side.

[0032] A surgical instrument 1 is attached to the tip of each of the multiple robot arms 50. The surgical instrument 1 includes, for example, a replaceable instrument 2, an endoscope 3 shown in Figure 9 for capturing images of the surgical site, and a pivot position setting device 4 shown in Figure 10 for setting the pivot position PP. The instrument 2 includes a driven unit 2a, an end effector 2b, a wrist joint 2c shown in Figure 5, and a shaft 2d. The end effector 2b is connected to the tip of the shaft 2d via the wrist joint 2c. Note that the configuration including the robot arms 50 and instruments 2 is an example of a manipulation device. The wrist joint 2c is also an example of a joint.

[0033] As shown in Figure 1, an endoscope 3 is attached to the tip of one of the multiple robot arms 50, for example, robot arm 50c, and instruments 2, which are surgical instruments 1 other than the endoscope 3, are attached to the tips of the remaining robot arms 50a, 50b, and 50d. Robot arms 50b and 50d are positioned adjacent to robot arm 50c. Robot arm 50a is positioned adjacent to robot arm 50b. Preferably, the endoscope 3 is attached to one of the two centrally positioned robot arms 50b and 50c, of the four robot arms 50 that are positioned adjacent to each other. The endoscope 3 observes the environment around robot arms 50a, 50b, and 50d and outputs the observation information. Specifically, the endoscope 3 images the surgical site of patient P as the environment around robot arms 50a, 50b, and 50d, and outputs the captured images to the monitor 140 of the remote control device 200 and the display unit 220 located on the cart 210.

[0034] (Instrument Configuration) As shown in Figure 5, an end effector 2b having jaw members 2g and 2h is attached to the tip of the instrument 2. Scissors, grapplers, needle holders, micro dissectors, stable applicators, tackers, suction cleaning tools, snare wires, and clip applicators can be used as the end effector 2b.

[0035] Instrument 2 includes a first support member 2e and a second support member 2f. The first support member 2e is attached to the shaft 2d. The second support member 2f is rotatably supported by the first support member 2e around the A10 axis and supports the end effector 2b so as to be rotatable around the A11 axis which intersects the A10 axis. The shaft 2d rotates around the A9 axis. The wrist joint 2c is positioned between the second support member 2f and the first support member 2e with the A10 axis as its axis of rotation.

[0036] (Configuration of the arm operating section) As shown in Figure 6, the arm operating section 60 is attached to the robot arm 50 and operates the robot arm 50. Specifically, the arm operating section 60 is attached to the second link section 53.

[0037] The arm operation unit 60 includes an enable switch 61, a joystick 62, a linear switch 63, a mode switching button 64, a mode indicator 65, a pivot button 66, and an adjustment button 67.

[0038] The enable switch 61, when pressed, allows or disallows the movement of the robot arm 50 by the joystick 62 and linear switch 63. When the arm control unit 60 is held by an operator such as a nurse or assistant, pressing the enable switch 61 allows the robot arm 50 to move the surgical instrument 1.

[0039] The joystick 62 is a control device for operating the movement of the surgical instrument 1 by the robot arm 50. The joystick 62 controls the direction and speed of movement of the robot arm 50. The robot arm 50 moves according to the direction and angle at which the joystick 62 is tilted.

[0040] The linear switch 63 is a switch for moving the surgical instrument 1 in the Z direction, which is the longitudinal direction of the surgical instrument 1. The linear switch 63 includes a linear switch 63a that moves the surgical instrument 1 in the direction of insertion into the patient P, and a linear switch 63b that moves the surgical instrument 1 away from the patient P. Both the linear switch 63a and the linear switch 63b are push-button switches.

[0041] The mode switching button 64 is a push button switch for switching between a mode for translating the surgical instrument 1 and a mode for rotationally moving the surgical instrument 1. As shown in FIG. 7, in the mode for translating the robot arm 50, the robot arm 50 is moved such that the distal end 1a of the surgical instrument 1 moves on the X-Y plane. As shown in FIG. 8, in the mode for rotationally moving the robot arm 50, when the pivot position PP is not stored in the storage unit 351 shown in FIG. 15, the instrument 2 as the surgical instrument 1 rotationally moves about the center of the end effector 2b on the A11 axis or the distal end of the end effector 2b as a fulcrum, and when the pivot position PP is stored in the storage unit 351, the robot arm 50 is moved such that the surgical instrument 1 rotationally moves about the pivot position PP as a fulcrum. Note that the surgical instrument 1 is rotationally moved in a state where the shaft 1c of the surgical instrument 1 is inserted into the trocar T. The mode switching button 64 is disposed on the surface of the arm operation unit 60 on the Z direction side.

[0042] The mode indicator 65 displays the switched mode. Lighting of the mode indicator 65 indicates the rotational movement mode, and extinguishing of the mode indicator 65 indicates the translational movement mode. Further, the mode indicator 65 also serves as a pivot position indicator that displays that the pivot position PP has been set. The mode indicator 65 is disposed on the surface of the arm operation unit 60 on the Z direction side.

[0043] The pivot button 66 is a push button switch for setting the pivot position PP that serves as a fulcrum for movement of the surgical instrument 1 attached to the robot arm 50. When the pivot button 66 is pressed, the pivot position PP is stored in the storage unit 351.

[0044] The adjustment button 67 is a button for optimizing the position of the robot arm 50. After the pivot position PP for the robot arm 50 to which the endoscope 3 is attached is set, pressing the adjustment button 67 optimizes the positions of the other robot arms 50 and the arm base 40. The adjustment button 67 is a button different from the enable switch 61.

[0045] (Remote Control Device) As shown in Figure 1, the remote control device 200 is located, for example, inside or outside the operating room. The remote control device 200 includes an operating unit 110, a foot pedal 120, a touch panel 130, a monitor 140, a support arm 150, a support bar 160, and an error reset button 161. The operating unit 110 constitutes an operating handle for an operator, such as a doctor, to input commands. The operating unit 110 receives the operator's input to the robot arm 50.

[0046] As shown in Figure 11, the operating unit 110 is a handle for operating the surgical instrument 1. The operating unit 110 also receives input from the surgical instrument 1. The operating unit 110 operates the robot arm 50 and the instrument 2 by changing its position and orientation. When the position and orientation of the operating unit 110 are changed, the position and orientation of the robot arm 50 and the instrument 2 are changed in accordance with the change in the position and orientation of the operating unit 711. The operating unit 110 includes an operating unit 110L located on the left side as viewed from the operator, such as a doctor, and operated by the operator's left hand, and an operating unit 110R located on the right side and operated by the operator's right hand. The operating unit 110 includes an arm section 111 and a wrist section 112. The operating unit 110R includes an arm section 111R and a wrist section 112R. The operating unit 110L includes an arm section 111L and a wrist section 112L.

[0047] The arm portion 111 has joints JT21, JT22, and JT23 as shown in Figure 11, and JT24, JT25, JT26, and JT27 as shown in Figures 12 and 13. The rotation axes of joints JT21, JT22, JT23, JT24, JT25, JT26, and JT27 are designated as axes A21, A22, A23, A24, A25, A26, and A27, respectively.

[0048] As shown in FIG. 11, the arm portion 111R includes a link portion 111a, a link portion 111b, and a link portion 111c. The upper end side of the link portion 111a is attached to the remote operation device 200 so as to be rotatable around the A21 axis along the vertical direction. The upper end side of the link portion 111b is attached to the lower end side of the link portion 111a so as to be rotatable around the A22 axis along the horizontal direction. One end side of the link portion 111c is attached to the lower end side of the link portion 111b so as to be rotatable around the A23 axis along the horizontal direction. A wrist portion 112 is attached to the other end side of the link portion 111c so as to be rotatable around the A24 axis. The link portion 111a is connected to the remote operation device 200 via a joint JT21. The link portion 111a and the link portion 111b are connected via a joint JT22. The link portion 111b and the link portion 111c are connected via a joint JT23. The arm portion 111 supports the wrist portion 112. Note that the arm portion 111L also has the same configuration as the arm portion 111R.

[0049] The wrist portion 112 includes a wrist portion 112R operated by the operator's right hand shown in FIG. 12, and a wrist portion 112L operated by the operator's left hand shown in FIG. 13. FIG. 12 shows the reference posture of the operation unit 110R, and FIG. 13 shows the reference posture of the operation unit 110L. The configuration of the wrist portion 112R and the configuration of the wrist portion 112L are the same.

[0050] The wrist portion 112 includes a link portion 112a, a link portion 112b, a link portion 112c, and a grip support member 112d operated by an operator such as a doctor. The base end of the link portion 112a is connected to the distal end of the arm portion 111, and rotates around the A24 axis. The base end of the link portion 112b is connected to the distal end of the link portion 112a, and rotates around the A25 axis relative to the link portion 112a. The base end of the link portion 112c is connected to the distal end of the link portion 112b, and the grip support member 112d is connected to the distal end thereof, so that the link portion 112c rotates around the A26 axis relative to the link portion 112b. The grip support member 112d rotates around the A27 axis relative to the link portion 112c. Each of the link portion 112a, the link portion 112b, and the link portion 112c has an L-shape.

[0051] The wrist section 112 includes a pair of grip members 112e that are opened and closed by the operator. The grip members 112e consist of elongated plate-shaped lever members, and the proximal ends of the pair of grip members 112e are rotatably connected to the proximal end of the grip support member 112d. The grip members 112e have cylindrical finger insertion portions 112f. The operator operates the wrist section 112 by inserting their fingers into the pair of finger insertion portions 112f. The base ends of the pair of grip members 112e are connected to the grip support member 112d, and the opening angle between the jaw members 2g and 2h is changed by increasing or decreasing the angle between the pair of grip members 112e. A magnet is placed on one of the grip members 112e, and a Hall sensor is placed on the grip support member 112d. When the operator opens and closes the grip members 112e, the magnet and Hall sensor function as angle detection sensors, and the Hall sensor outputs the opening angle. Alternatively, a Hall sensor may be placed on the grip member 112e, and a magnet may be placed on the grip support member 112d.

[0052] As shown in Figure 1, the monitor 140 is a scope-type display device for displaying images captured by the endoscope 3. The monitor 140 also has a notification unit 141, which emits an error sound. The support arm 150 supports the monitor 140 so that its height is at the same height as the operator's face, such as a doctor. The touch panel 130 is located on the support bar 160. The surgical support robot 100 can be operated by the remote control device 200 by detecting the operator's head using a sensor located near the monitor 140. The operator operates the control unit 110 and foot pedal 120 while viewing the surgical area on the monitor 140. This inputs a command to the remote control device 200. The command input to the remote control device 200 is transmitted to the surgical support robot 100.

[0053] (Foot pedals) As shown in Figure 14, the foot pedals 120 are provided in multiple units to perform functions related to the surgical instrument 1. The multiple foot pedals 120 are arranged on the base unit 121. The foot pedals 120 include a switching pedal 122, a clutch pedal 123, a camera pedal 124, an incision pedal 125, a coagulation pedal 126, and a foot detection unit 127. The switching pedal 122, clutch pedal 123, camera pedal 124, incision pedal 125, and coagulation pedal 126 are operated by the operator's feet. The incision pedal 125 includes an incision pedal 125R for the right robot arm 50 and an incision pedal 125L for the left robot arm 50. The coagulation pedal 126 includes a coagulation pedal 126R for the right robot arm 50 and a coagulation pedal 126L for the left robot arm 50.

[0054] The switching pedal 122 switches the robot arm 50 to be operated by the control unit 110. The clutch pedal 123 performs a clutch operation that temporarily disconnects the control connection between the robot arm 50 and the control unit 110. While the clutch pedal 123 is pressed down by the operator, the operation of the control unit 110 is not transmitted to the robot arm 50. Also, while the camera pedal 124 is pressed down by the operator, the control unit 110 can operate the robot arm 50 to which the endoscope 3 is attached. While the incision pedal 125 or coagulation pedal 126 is pressed down by the operator, the electrosurgical device is activated.

[0055] (Vision Unit and Image Processing Unit) As shown in Figure 1, the vision unit 300 and the image processing unit 400 are mounted on the cart 210. The image processing unit 400 processes the images captured by the endoscope 3. A display unit 220 is located on the cart 210. The display unit 220 displays the images captured by the endoscope 3.

[0056] (Control System Configuration) As shown in Figure 15, the surgical support system 500 comprises a first control device 310, an arm control unit 320, a positioner control unit 330, an operation control unit 340, and a second control device 350. The surgical support system 500 also comprises a storage unit 311 connected to the first control device 310 and a storage unit 351 connected to the second control device 350. Note that the first control device 310 is an example of a control device.

[0057] The first control device 310 is positioned inside the medical trolley 10 to communicate with the arm control unit 320 and the positioner control unit 330, and controls the entire surgical support system 500. Specifically, the first control device 310 communicates with the arm control unit 320, the positioner control unit 330, and the operation control unit 340, and controls these units. The first control device 310 is connected to the arm control unit 320, the positioner control unit 330, and the operation control unit 340 by a LAN or the like. The first control device 310 is located inside the medical trolley 10.

[0058] An arm control unit 320 is located for each of the multiple robot arms 50. In other words, multiple arm control units 320 corresponding to the number of robot arms 50 are located inside the medical trolley 10.

[0059] As shown in Figure 15, the input device 22 is connected to the first control device 310 by a LAN or the like. The status indicator 41, arm status indicator 42, operating handle 23, throttle 23a, and joystick 22b are connected to the positioner control unit 330 by serial communication via a communication network that allows them to share information with each other, through wiring 360. In Figure 15, it is shown that all of the status indicators, including the status indicator 41 and arm status indicator 42, are connected to a single wiring 360, but in reality, there is a separate wiring 360 for each of the status indicator 41, arm status indicator 42, operating handle 23, throttle 23a, and joystick 22b.

[0060] As shown in Figure 16, the arm section 51 is equipped with multiple servo motors SM1, encoders EN1, and a reduction gear, each corresponding to a joint JT1, JT2, JT3, JT4, JT5, JT6, and JT7. The encoder EN1 detects the rotation angle of the servo motor SM1. The reduction gear reduces the rotation of the servo motor SM1 to increase the torque. Inside the medical trolley 10, a servo control unit SC1 for controlling the servo motor SM1 is located adjacent to the arm control unit 320. The encoder EN1 for detecting the rotation angle of the servo motor SM1 is electrically connected to the servo control unit SC1.

[0061] The second link section 53 is equipped with a servo motor SM2, an encoder EN2, and a reduction gear for rotating a driven member located on the driven unit 2a of the surgical instrument 1. The encoder EN2 detects the rotation angle of the servo motor SM2. The reduction gear reduces the rotation of the servo motor SM2 to increase the torque. The medical trolley 10 is also equipped with a servo control unit SC2 for controlling the servo motor SM2 that drives the surgical instrument 1. The encoder EN2 for detecting the rotation angle of the servo motor SM2 is electrically connected to the servo control unit SC2. Note that multiple servo motors SM2, encoders EN2, and servo control units SC2 are provided.

[0062] The translational movement mechanism 54 is equipped with a servo motor SM3 for translating the surgical instrument 1, an encoder EN3, and a reduction gear. The encoder EN3 detects the rotation angle of the servo motor SM3. The reduction gear reduces the rotation of the servo motor SM3 to increase the torque. The medical trolley 10 also has a servo control unit SC3 for controlling the servo motor SM3 that translates the surgical instrument 1. The encoder EN3 for detecting the rotation angle of the servo motor SM3 is electrically connected to the servo control unit SC3. The encoders EN1, EN2, and EN3 are, for example, magnetic, mechanical, or electromagnetic induction type. The high resolution of the encoders EN1, EN2, and EN3 themselves improves the positional accuracy of the tip of the robot arm 50 and the tip of the surgical instrument 1. Furthermore, by reducing the rotational speed transmitted from the servo motors SM1, SM2, and SM3 using a reduction gear, the accuracy of the rotational angle of the joints of the robot arm 50, which is the output shaft, is further improved.

[0063] The first control device 310 generates command values ​​that command the positions (rotation angles of each motor) of the servo motors SM1, SM2, and SM3 based on the operation received by the remote control device 200, and drives the servo motors SM1, SM2, and SM3 based on the command values. The first control device 310 then detects an error indicating a deviation abnormality if the difference between the command values ​​and the positions of the servo motors SM1, SM2, and SM3 detected by the sensors exceeds an acceptable range.

[0064] As shown in Figure 17, the positioner 30 is equipped with multiple servo motors SM4, an encoder EN4, and a reduction gear, corresponding to the multiple joints 33 of the positioner 30. The encoder EN4 is configured to detect the rotation angle of the servo motors SM4. The reduction gear is configured to reduce the rotation of the servo motors SM4 and increase the torque.

[0065] The medical trolley 10 is equipped with wheels, including front wheels that serve as drive wheels and rear wheels that are steered by an operating handle 23. The rear wheels are positioned closer to the operating handle 23 than the front wheels. The medical trolley 10 is also equipped with servo motors SM5, encoders EN5, reduction gears, and brakes BRK, each of which drives one of the multiple front wheels of the medical trolley 10. The reduction gear is configured to reduce the rotation of the servo motors SM5 and increase the torque. A potentiometer P1, as shown in Figure 3, is positioned on the operating handle 23, and the servo motors SM5 of the front wheels are driven based on the rotation angle detected by the potentiometer P1 in response to the twisting of the throttle 23a. The rear wheels of the medical trolley 10 are of the twin-wheel type, and the rear wheels are steered based on the left and right rotation of the operating handle 23. Furthermore, a potentiometer P2, as shown in Figure 3, is positioned on the rotation axis of the operating handle 23, and a servo motor SM6, encoder EN6, and reduction gear are positioned on the rear wheels of the medical trolley 10. The reduction gear is configured to reduce the rotation of the servo motor SM6 and increase the torque. The servo motor SM6 is driven based on the rotation angle detected by the potentiometer P2 in response to the left and right rotation of the operating handle 23. In other words, the steering of the rear wheels by the left and right rotation of the operating handle 23 is power-assisted by the servo motor SM6.

[0066] The medical trolley 10 moves in the forward and backward direction when its front wheels are driven. In addition, when the operating handle 23 is rotated, the rear wheels are steered, causing the medical trolley 10 to rotate in the left and right direction.

[0067] As shown in Figure 17, the medical trolley 10 is equipped with a servo control unit SC4 for controlling the servo motor SM4 that moves the positioner 30. An encoder EN4 for detecting the rotation angle of the servo motor SM4 is electrically connected to the servo control unit SC4. The medical trolley 10 is also equipped with a servo control unit SC5 for controlling the servo motor SM5 that drives the front wheels of the medical trolley 10. An encoder EN5 for detecting the rotation angle of the servo motor SM5 is electrically connected to the servo control unit SC5. The medical trolley 10 is also equipped with a servo control unit SC6 for controlling the servo motor SM6 that power assists the steering of the rear wheels of the medical trolley 10. An encoder EN6 for detecting the rotation angle of the servo motor SM6 is electrically connected to the servo control unit SC6.

[0068] As shown in Figures 16 and 17, each of the joints JT1, JT2, JT3, JT4, JT5, JT6, and JT7 of the arm section 51 and the joint 33 of the positioner 30 are equipped with brakes. Brakes are also equipped on the front wheels of the medical trolley 10, the arm base 40, and the translational movement mechanism 54. Control signals are transmitted unidirectionally from the arm control unit 320 to each of the brakes mounted on the joints JT1, JT2, JT3, JT4, JT5, JT6, and JT7 of the arm section 51 and the translational movement mechanism 54. The control signals are signals to turn the brakes on and off. The signal to turn on the brakes includes a signal to maintain the state in which the brakes are engaged. The same applies to the control signals from the positioner control unit 330 to each of the brakes mounted on the joint 33 of the positioner 30 and the arm base 40. At startup, all brakes (BRK) on the arm base 40, arm section 51, and translational movement mechanism 54 are released, and the servo motor SM is driven to counteract gravity, thereby maintaining the posture of the robot arm 50 and the arm base 40. When an error occurs in the surgical support system 500, the brakes (BRK) on the arm base 40, arm section 51, and translational movement mechanism 54 are turned on. When the error in the surgical support system 500 is cleared, the brakes (BRK) on the arm base 40, arm section 51, and translational movement mechanism 54 are turned off. The brakes (BRK) on the arm base 40, arm section 51, and translational movement mechanism 54 are turned on when the surgical support system 500 is shut down. In addition, the front wheels of the medical trolley 10 always have the brakes (BRK) turned on, and the brakes (BRK) are released only while the enable switch 23b is pressed. Furthermore, each joint 33 of the positioner 30 has the brake BRK always on, and the brake BRK is released only while the enable switch 22c is pressed.

[0069] As shown in Figure 18, servo motors SM7a, SM7b, SM7c, SM7d, SM7e, SM7f, and SM7g are arranged on the joints JT21, JT22, JT23, JT24, JT25, JT26, and JT27 of the operating section 110, respectively. Servo motor SM7a rotates the link section 111a around the A21 axis. Servo motor SM7b rotates the link section 111b around the A22 axis. Servo motor SM7c rotates the link section 111c around the A23 axis. Servo motor SM7d rotates the link section 112a around the A24 axis. Servo motor SM7e rotates the link section 112b around the A25 axis. Servo motor SM7f rotates the link section 112c around the A26 axis. Furthermore, the servo motor SM7g rotates the grip support member 112d around the A27 axis. Servo control units SC7a, SC7b, SC7c, SC7d, SC7e, SC7f, and SC7g are also provided to control each servo motor. Each servo control unit is electrically connected to encoders EN7a, EN7b, EN7c, EN7d, EN7e, EN7f, and EN7g for detecting the rotation angle of each servo motor. Each servo motor, each servo control unit, and each encoder are provided in the operation unit 110L and the operation unit 110R, respectively. The servo motors SM7a, SM7b, SM7c, SM7d, SM7e, SM7f, and SM7g move the respective link sections of the operation unit 110. Note that servo motors SM7a, SM7b, SM7c, SM7d, SM7e, SM7f, and SM7g are examples of actuators.

[0070] The first control device 310, via the operation control unit 340, controls each servo motor to generate a torque that counteracts the gravitational torque generated on the rotation axis of each servo motor, according to the posture of the operation unit 110. This makes it possible for the operator to operate the operation unit 110 with relatively little force.

[0071] Furthermore, when the operator rotates the grip support member 112d around the A27 axis of the operating section 110 shown in Figures 12 and 13, the shaft 2d of the instrument 2 rotates around the A9 axis shown in Figure 5. Also, when the operator rotates the joints JT24, JT25, and JT26 of the operating section 110 shown in Figures 12 and 13, the end effector 2b bends around the A10 axis or A11 axis shown in Figure 5.

[0072] The operation control unit 340 is located in the main body of the remote control device 200. The operation control unit 340 controls the operation unit 110. As shown in Figure 15, the operation control unit 340 is positioned to correspond to the left-hand operation unit 110L and the right-hand operation unit 110R, respectively.

[0073] As shown in Figure 15, the vision unit 300 and the image processing unit 400 are connected to the first control device 310 via a LAN or the like. The display unit 220 is connected to the vision unit 300.

[0074] (Control operation of the first control device) The control of the first control device 310 when the operation unit 110 receives an operation from the operator will be described below. The following control is performed by a functional block of the first control device 310 that operates based on the program 311a stored in the storage unit 311.

[0075] As shown in Figure 19, when the operator operates the operating unit 110, the axis values ​​of the operating unit 110 are input to the first control device 310. The first control device 310 calculates the position and orientation of the operating unit 110 and the opening angle of the pair of grip members 112e from the axis values ​​of the operating unit 110. In Figure 19, the operating unit 110 is denoted as HC. The position and orientation refer to the coordinate values ​​in the TCP coordinate system, and the opening angle of the pair of grip members 112e refers to the opening angle of the jaw member 2g and jaw member 2h. TCP is an abbreviation for Tool Center Point. TCP is set, for example, at the tip of the instrument 2. The first control device 310 also calculates the difference between the current position and orientation of the operating unit 110 and the previous position and orientation of the operating unit 110, and calculates the amount of operation of the operating unit 110 based on the difference. Furthermore, the position and orientation of the endoscope 3 are input to the first control device 310.

[0076] The first control device 310 converts the operating amount of the control unit 110 into the coordinate system of the endoscope 3 based on the input position and orientation of the endoscope 3. Then, the first control device 310 calculates the operating amount of the instrument 2 based on the operating amount of the control unit 110 converted into the coordinate system of the endoscope 3. The first control device 310 also calculates the sum of the current operating amount of the instrument 2 and the previous operating amount of the instrument 2, and calculates the position and orientation of the instrument 2 based on the sum. Note that "this time" means the current control cycle, and "previous time" means the previous control cycle.

[0077] Furthermore, the first control device 310 performs a process to limit the axis speed of the robot arm 50 based on the calculated position and orientation of the instrument 2. The maximum value of the axis speed for each axis is set in advance, and if the axis speed of the robot arm 50 exceeds the maximum value, the first control device 310 limits the axis speed so that the axis speed becomes the maximum value. The functional block of the first control device 310 that limits the axis speed is called the limiting unit.

[0078] The first control device 310 performs inverse kinematics calculations for the limited axial velocities. The first control device 310 calculates command values ​​for each axis of the robot arm 50 based on the inverse kinematics calculations. The first control device 310 then performs a process to apply acceleration limits to the calculated axial command values. Note that acceleration includes acceleration in the direction of increasing velocity and acceleration in the direction of decreasing velocity. The maximum value of the acceleration for each axis is set in advance, and if the acceleration of the robot arm 50 exceeds the maximum value, the first control device 310 limits the acceleration so that the acceleration becomes the maximum value. The limited axial velocities are then output to the servo control unit SC1 that drives the robot arm 50 and the servo control unit SC2 that drives the instrument 2. Furthermore, the signals indicating the position, orientation, and opening angles of the jaw members 2g and 2h included in the instrument 2, generated based on the operator's operation of the control unit 110 of the remote control device 200, will be referred to as manual operation commands below. The signals indicating the position, orientation, and opening angles of the jaw members 2g and 2h included in the instrument 2, inferred by the inference unit 613 of the inference control device 600 (described later), will be referred to below as inference operation commands.

[0079] (Inference Control Device) As shown in Figure 1, the inference control device 600 is located separately from the surgical support robot 100, the remote control device 200, the vision unit 300, and the image processing unit 400. The inference control device 600 is, for example, a PC (personal computer). As shown in Figure 15, the inference control device 600 is connected to the first control device 310 by a LAN or the like. As shown in Figure 20, the inference control device 600 includes a processing unit 610 such as a CPU (Central Processing Unit). The inference control device 600 also includes a storage unit 620. The storage unit 620 includes, for example, ROM (Read Only Memory), RAM (Random Access Memory), and HDD (hard disk drive). Furthermore, the inference control device 600 includes an input unit 630. The input unit 630 is, for example, a keyboard. The inference control device 600 may also be placed inside a surgical support robot 100, a remote control device 200, a vision unit 300, or an image processing unit 400 as a device having a processing unit 610 and a storage unit 620.

[0080] In this embodiment, the processing unit 610 includes a data collector 611, a learner 612, and an inference unit 613. The data collector 611, learner 612, and inference unit 613 are functional blocks executed by the CPU of the processing unit 610 based on a program 621 stored in the storage unit 620.

[0081] The data acquisition unit 611 receives images G captured by the endoscope 3 attached to the robot arm 50c. As shown in Figure 21, images G include the surgical site of the patient P and the tip of the instrument 2. The data acquisition unit 611 collects and stores the images G captured by the endoscope 3 in chronological order. The data acquisition unit 611 also collects and stores manual operation commands and inference operation commands processed by the first control device 310 in chronological order. If the sampling periods of the images G captured by the endoscope 3 and the manual operation commands and inference operation commands are different, the data acquisition unit 611 adjusts for the time difference between the images G captured by the endoscope 3 and the manual operation commands and inference operation commands. The data acquired by the data acquisition unit 611 is stored, for example, in the data acquisition unit 611 itself or in the storage unit 620.

[0082] The learner 612 generates an inference 613, which is a trained model that has been trained by machine learning, using observational information of the environment surrounding multiple robot arms 50, each operated independently based on motion commands, and motion commands as input. Specifically, the learner 612 generates an inference 613, which is a trained model, by machine learning, using images G captured by the endoscope 3 collected by the data collector 611 and manual motion commands as training data. Machine learning can be, for example, deep learning using a neural network model, imitation learning that mimics human methods, or deep imitation learning that learns human methods using a neural network model.

[0083] In the first embodiment, the manual operation commands learned by the learning device 612 include suturing operation commands for suturing the surgical site of patient P. Specifically, the suturing operation commands are manual operation commands generated based on the operator's operation of the control unit 110 in order for the operator to suture the surgical site of patient P. The image G learned by the learning device 612 is an image G captured by the endoscope 3 when suturing is performed on the surgical site of patient P. As shown in Figure 21, this image G shows the surgical site of patient P, the jaw members 2g and 2h at the tip of the instrument 2, the suture needle 641 held by the jaw members 2g and 2h, and the thread 642 threaded through the suture needle 641. In other words, the learning device 612 learns the suturing operation of the surgical site of patient P and generates an inference device 613 for inferring the suturing operation of the surgical site of patient P.

[0084] In the first embodiment, the data acquisition and storage operation in which the data acquisition unit 611 acquires and stores images G captured by the endoscope 3 and manual operation commands as training data for generating the inference unit 613 is initiated and stopped, for example, by the coagulation pedal 126R. The coagulation pedal 126R is located separately from the coagulation pedal 126L, which switches between autonomous mode and remote operation mode, as described later. The coagulation pedal 126R is pressed by the operator's foot, which switches the start and stop of the data acquisition and storage operation of the data acquisition unit 611. Pressing by the foot means stepping on it with the foot.

[0085] Furthermore, when the coagulation pedal 126R is pressed, an alert sound is emitted. For example, the alert sound is emitted from the notification unit 141 of the monitor 140 shown in Figure 1 or from the speaker 22e of the input device 22.

[0086] (Performance Level) In addition to the images G captured by the endoscope 3 and manual operation commands, the learning device 612 can also learn the performance level of the robot arm 50 and instrument 2. The performance level includes, for example, who the operator is, the execution time of the suturing operation on the patient P's surgical site, and the score of the quality of the suturing. The performance level is input, for example, using the input unit 630 of the inference control device 600.

[0087] In the first embodiment, the inference unit 613 takes observational information of the environment surrounding the robot arm 50 and instrument 2 as input and operation commands, and outputs inferred operation commands, which are the inferred operation commands. As shown in Figure 19, the inference control device 600 receives, for example, an image G captured by the endoscope 3, the position and orientation of the tip of the endoscope 3, the position and orientation of the left and right instrument 2, and the opening angles of the jaw members 2g and 2h included in each of the left and right instrument 2. The left and right instrument 2 refer to the two robot arms 50 selected by the operator as the target of operation from among the robot arms 50a, 50b, and 50d to which the instrument 2 is attached. The operator switches the robot arm 50 to be operated by the operation unit 110 by operating the switching pedal 122. The inference control device 600 also receives information on whether it is in autonomous mode or remote control mode. This information may be stored in the storage unit 620, storage unit 311, storage unit 351, or on the cloud. The information indicating whether the system is in autonomous mode or remote control mode refers to whether the coagulation pedal 126L is being pressed by the operator, as will be described later. The inference control device 600 also receives information on the start or stop of the collection and storage operation, which collects and stores images G captured by the endoscope 3 and manual operation commands as training data for generating the inference unit 613. This information may also be stored in the storage unit 620, storage unit 311, storage unit 351, or on the cloud. This information may also be stored for each robot arm 50.

[0088] In the first embodiment, the inference unit 613 takes observation information and operation commands from multiple past points in time stored in the data collector 611 or the storage unit 620 as input and outputs inference operation commands for the next and subsequent times. The next time may be immediately after the current time, or a little while after the current time. The next inference operation command may be an inference operation command for one step in the next time, or an inference operation command for several steps after the next time. For example, the inference unit 613 receives images G captured by the endoscope 3 for several steps of the control cycle of the surgical support robot 100 as observation information. The inference unit 613 also receives manual operation commands, inference operation commands, or operation commands that are a mixture of manual operation commands and inference operation commands for several steps of the control cycle of the surgical support robot 100 as input. The images G, manual operation commands, inference operation commands, etc., are input to the inference control device 600 via the input port 640. The inference unit 613 outputs inference operation commands based on this input information.

[0089] In the first embodiment, the robot arm 50 and instrument 2 are configured to be switchable between a remote control mode, which operates based on the operation of the control unit 110 of the remote control device 200, and an autonomous mode, which operates autonomously. Specifically, at least one set of multiple robot arms 50 and instrument 2 can be switched between an autonomous mode, which operates based on inference operation commands output from the inference unit 613, and a remote control mode, which operates based on manual operation commands, which are operation commands from the remote control device 200. In the first embodiment, all sets of robot arms 50a, 50b, and 50d, and the instrument 2 attached thereto, can be switched between autonomous mode and remote control mode. It is also possible to set some of the sets of multiple robot arms 50 and instrument 2 to autonomous mode and others to remote control mode. Furthermore, the robot arm 50c to which the endoscope 3 is attached may also be switchable between autonomous mode and remote control mode. The operation in autonomous mode includes suturing operations for suturing the surgical site of patient P.

[0090] In addition, additional information related to surgery may be included as input information to the inference unit 613 in autonomous mode. This additional information may include, for example, information specifying a surgical procedure or a specific step in a particular surgical procedure, or information specifying a particular technique such as suturing, or information on steps that further subdivide that technique. The input information may be, for example, voice instructions, input via an interface other than voice instructions such as a touch panel, input via a hard switch, or input via the operator's gestures. Furthermore, if, for example, a suturing technique is input as additional information related to surgery, the inference unit 613 can further subdivide the suturing technique into lower-level steps for inference. Lower-level steps refer to individual actions such as, for example, holding the suture needle 641, passing the suture needle 641 through the suture site of patient P, and tying the thread 642. It is also possible to directly input information on these lower-level steps via voice, an interface, or a hard switch.

[0091] Furthermore, the duration of the image G input to the inference unit 613 and the duration of the operation command may be set. For example, by operating the touch panel 130 of the remote control device 200, the duration of the image G input to the inference unit 613 and the duration of the operation command may be set to 5 steps, 10 steps, 15 steps, etc., of the control cycle.

[0092] Furthermore, in addition to the image G and the operation command, the performance level of the instrument 2 may be input to the inference unit 613, and the inference unit 613 may be trained according to the performance level. Then, the performance level of the inference unit 613 to be used for inference may be selected. In other words, multiple inference units 613 may be prepared according to their performance levels. Then, by operating the touch panel 130 of the remote control device 200, the performance level of the inference unit 613 is selected, and inference is performed by the selected inference unit 613.

[0093] In the first embodiment, in autonomous mode, the operation command and the image G acquired by the endoscope 3 are input to the inference unit 613, and the inference unit 613 outputs an inference operation command, and the robot arm 50 and instrument 2 operate based on the inference operation command. The inference operation command also includes, for example, the position and orientation of the left and right instrument 2, and the opening angles of the jaw members 2g and 2h included in each of the left and right instrument 2. Here, as shown in Figure 19, the first control device 310 performs a process to limit the axial speed of the robot arm 50 and instrument 2 as described above. The inference operation command is replaced with a manual operation command before the process of applying axial speed and before the calculation of the position and orientation of the instrument 2. That is, in autonomous mode, the manual operation command is replaced with an inference operation command before the manual operation command is limited by the first control device 310.

[0094] As described above, the switching pedal 122 switches between the robot arms 50 operated by the left and right control units 110. For example, the switching pedal 122 selects two robot arms 50 to be operated by the left and right control units 110 from among the robot arms 50a, 50b, and 50d to which the instrument 2 is attached. Here, when the mode is switched, if both the left robot arm 50 and instrument 2 set and the right robot arm 50 and instrument 2 set are switched modes, the autonomous mode and remote control mode are switched by the solidification pedal 126L. The solidification pedal 126L is operated by the operator's foot. The autonomous mode is executed while the operator is pressing down on the solidification pedal 126L with their foot. The remote control mode is executed while the operator is releasing the pressure on the solidification pedal 126L. Pressing down with the foot means that the operator is pressing down on the solidification pedal 126L. Releasing the pressure means that the operator has stopped pressing the coagulation pedal 126L. On the other hand, when switching modes for each left-hand pair, two switching units are required: one for switching the right-hand mode and another for switching the left-hand mode. For example, in addition to the coagulation pedal 126L, another foot pedal 120 may be assigned as a switching unit.

[0095] Even in remote control mode, the inference unit 613 continues to infer inference operation commands using manual operation commands in remote control mode as input. The autonomous mode is started when the operator presses down on the solidification pedal 126L. In autonomous mode, the robot arm 50 and instrument 2 operate based on the inference operation commands output from the inference unit 613. The remote control mode is started when the operator releases the pressure on the solidification pedal 126L. In remote control mode, manual operation commands are generated when the operator operates the control unit 110, and the robot arm 50 and instrument 2 operate based on the manual operation commands. Here, even after switching from autonomous mode to remote control mode, the inference unit 613 continues to infer inference operation commands. Specifically, at the initial point after switching from autonomous mode to remote control mode, the inference operation commands and manual operation commands are included in several steps of the past control cycle, so the inference unit 613 infers inference operation commands using past observation information, past inference operation commands, and past manual operation commands as input. After some time has passed since switching from autonomous mode to remote control mode, the past few steps of the control cycle will only contain past manual operation commands. Therefore, the inference unit 613 uses past observation information and past manual operation commands as input to infer an inference operation command.

[0096] Next, the control process in the autonomous mode of the surgical support system 500 will be described. In the following control method of the surgical support system 500, the operations performed by the inference control device 600 are performed based on the program 621 stored in the memory unit 620, and the operations performed by the first control device 310 are performed based on the program 311a stored in the memory unit 311.

[0097] As shown in Figure 22, when autonomous mode is started, in step S41, the operation commands for a control cycle of approximately 1 second and the image G captured by the endoscope 3 are collected by the data collector 611. The image G from the endoscope 3 is based on the Full HD standard, and the resolution is 1920 x 1080 pixels. Next, in step S42, the inference unit 613 infers an inferred operation command based on the collected operation commands and image G. At the initial point after switching from remote control mode to autonomous mode, the past few steps of the control cycle contain both inferred operation commands and manual operation commands, so the inference unit 613 uses past observation information, past inferred operation commands, and past manual operation commands as input to infer an inferred operation command. After some time has passed since switching from remote control mode to autonomous mode, the past few steps of the control cycle contain only past inferred operation commands, so the inference unit 613 uses past observation information and past inferred operation commands as input to infer an inferred operation command. Furthermore, the image G of the endoscope 3 used by the inference unit 613 for inferring the inference operation command has been reduced in size to a resolution of 256 x 128 pixels. The inference operation command that the inference unit 613 infers is an inference operation command for approximately one second following the operation command for approximately one second of the control cycle collected by the data collector 611. The time required for the inference unit 613 to infer the inference operation command is approximately 0.1 seconds, which is shorter than the period of the inference operation command that the inference unit 613 infers. In step S42, the data collector 611 collects operation commands and the image G captured by the endoscope 3 in parallel while the inference unit 613 is performing the inference. In step S43, the inference control device 600 outputs the inferred inference operation command to the first control device 310. As a result, the robot arm 50 and instrument 2 operate autonomously, and the posture of the operating unit 110 is also changed to follow the changes in the posture of the robot arm 50 and instrument 2.

[0098] (Determination of the state of the operating unit) In the first embodiment, as shown in Figure 23, the first control device 310 acquires the state of the operating unit 110 and, based on the acquired state of the operating unit 110, performs control to determine whether the state of the operating unit 110 is such that the robot arm 50 and instrument 2 can be switched from autonomous mode to remote control mode. Furthermore, if the first control device 310 determines that the state of the operating unit 110 is such that the robot arm 50 and instrument 2 can be switched from autonomous mode to remote control mode, it performs the switching of the robot arm 50 and instrument 2 from autonomous mode to remote control mode. If the first control device 310 determines that the state of the operating unit 110 is such that the robot arm 50 and instrument 2 cannot be switched from autonomous mode to remote control mode, it performs control to notify that the state of the operating unit 110 is such that the robot arm 50 and instrument 2 cannot be switched from autonomous mode to remote control mode. Note that switching the robot arm 50 and instrument 2 from autonomous mode to remote control mode means replacing the inferred action commands with manual action commands. Also, in Figure 23, only instrument 2 is shown among the robot arm 50 and instrument 2.

[0099] If the first control device 310 determines that the state of the operation unit 110 is not in a state where the robot arm 50 and instrument 2 can be switched from autonomous mode to remote control mode, it performs control to notify a warning by display and sound at least one of the following: for example, the first control device 310 performs control to display a warning screen on at least one of the display unit 22a, touch panel 130, monitor 140 and display unit 220. Alternatively, for example, the first control device 310 performs control to notify a warning sound on at least one of the speaker 22e and notification unit 141. After notifying a warning, the state of the operation unit 110 may be changed manually or automatically to a state where it can be switched from autonomous mode to remote control mode. Furthermore, when the state of the operation unit 110 is changed to a state where it can be switched from autonomous mode to remote control mode, the robot arm 50 and instrument 2 may be switched from autonomous mode to remote control mode.

[0100] Furthermore, in the first embodiment, the first control device 310 performs control to determine whether the state of the operating unit 110 is such that the robot arm 50 and instrument 2 can be switched from autonomous mode to remote control mode, based on at least one of the position and orientation of the operating unit 110. The first control device 310 may make the determination based only on the orientation of the operating unit 110, based only on the position of the operating unit 110, or based on both the position and orientation of the operating unit 110. The first control device 310 performs control to acquire at least one of the position and orientation of the operating unit 110 based on the detected values ​​of the encoders EN7a, EN7b, EN7c, EN7d, EN7e, EN7f, and EN7g.

[0101] When determining whether a switch can be made based on the position of the operating unit 110, the first control device 310 compares the position of the operating unit 110 with the positions of the robot arm 50 and instrument 2 as a manipulation device, and determines whether the position of the operating unit 110 matches the positions of the robot arm 50 and instrument 2. In this case, it is permissible for the position of the operating unit 110 to be slightly different from the positions of the robot arm 50 and instrument 2. That is, if the position of the operating unit 110 is within the permissible range, the first control device 310 may determine that the position of the operating unit 110 matches the positions of the robot arm 50 and instrument 2, and if the position of the operating unit 110 is outside the permissible range, it may determine that the position of the operating unit 110 does not match the positions of the robot arm 50 and instrument 2. The permissible range can be determined as a range based on the positions of the robot arm 50 and instrument 2, and predetermined permissible values ​​related to the positions. The predetermined tolerances for position may be determined based on factors such as the encoder resolution, the play in the reducer or gears, the amount of deformation in the links, or ergonomic considerations.

[0102] Furthermore, when determining whether a switch can be made based on the posture of the operating unit 110, the first control device 310 compares the posture of the operating unit 110 with the postures of the robot arm 50 and instrument 2 as manipulation devices, and determines whether the posture of the operating unit 110 matches the postures of the robot arm 50 and instrument 2. In this case, it is permissible for the posture of the operating unit 110 to be slightly different from the postures of the robot arm 50 and instrument 2. That is, if the posture of the operating unit 110 is within the permissible range, the first control device 310 may determine that the posture of the operating unit 110 matches the postures of the robot arm 50 and instrument 2, and if the posture of the operating unit 110 is outside the permissible range, it may determine that the posture of the operating unit 110 does not match the postures of the robot arm 50 and instrument 2. The permissible range can be determined as a range based on the postures of the robot arm 50 and instrument 2, and predetermined permissible values ​​related to posture. The predetermined tolerances for posture may be determined based on factors such as the encoder resolution, the play in the reducer and gears, the amount of deformation in the links, or ergonomic considerations.

[0103] Furthermore, the first control device 310 performs control to determine whether the state of the operating unit 110 is such that the robot arm 50 and instrument 2 can be switched from autonomous mode to remote control mode, based on the opening angle of the pair of grip members 112e. The first control device 310 performs control to acquire the opening angle of the pair of grip members 112e of the operating unit 110 based on the detection value of a sensor that detects the opening angle of the pair of grip members 112e, such as a Hall sensor placed on the grip support member 112d. The first control device 310 may also make a determination based on the position of the operating unit 110, the posture of the operating unit 110 and the opening angle of the pair of grip members 112e, or the posture of the operating unit 110 and the opening angle of the pair of grip members 112e. Furthermore, when making a determination based on the posture of the operating unit 110 and the opening angle of the pair of grip members 112e, it is assumed that there is a clutch function by the clutch pedal 123, for example.

[0104] When determining whether or not to switch based on the opening angle of a pair of grip members 112e, the first control device 310 compares the opening angle of the pair of grip members 112e with the opening angles of the jaw members 2g and 2h included in the instrument 2, and determines whether or not the opening angle of the pair of grip members 112e matches the opening angles of the jaw members 2g and 2h. In this case, it is permissible for the opening angle of the pair of grip members 112e to be slightly different from the opening angles of the jaw members 2g and 2h. That is, if the opening angle of the pair of grip members 112e is within the permissible range, the first control device 310 may determine that the opening angle of the pair of grip members 112e matches the opening angles of the jaw members 2g and 2h, and if the opening angle of the pair of grip members 112e is outside the permissible range, it may determine that the opening angle of the pair of grip members 112e does not match the opening angles of the jaw members 2g and 2h. The tolerance range can be determined as a range based on the opening angles of jaw members 2g and 2h, and predetermined tolerance values ​​for the opening angles. The predetermined tolerance values ​​for the opening angles may be determined from the encoder resolution, the backlash of the reducer and gears, the amount of deformation of the links, or from an ergonomic standpoint.

[0105] Furthermore, when the first control device 310 switches the robot arm 50 and instrument 2 from autonomous mode to remote control mode, it may perform control to reset at least one of the position and orientation of the operation unit 110 to zero. By performing this reset, it is possible to suppress sudden changes in command values ​​that could worsen operability. For example, the first control device 310 may perform control to reset whichever of the position and orientation of the operation unit 110 was not used to determine whether or not to switch modes, to zero. Specifically, if the determination of whether or not to switch modes is made based only on the orientation of the operation unit 110, the first control device 310 will perform control to reset the position of the operation unit 110 to zero. If the determination of whether or not to switch modes is made based only on the position of the operation unit 110, the first control device 310 will perform control to reset the orientation of the operation unit 110 to zero. Also, if necessary, the first control device 310 may perform control to reset both the position and orientation of the operation unit 110 to zero.

[0106] Furthermore, in the first embodiment, servo motors SM7a, SM7b, SM7c, SM7d, SM7e, SM7f, and SM7g are arranged in the operating unit 110. As shown in Figure 24, when the robot arm 50 and instrument 2 are operating in autonomous mode, the first control device 310 controls the servo motors SM7a, SM7b, SM7c, SM7d, SM7e, SM7f, and SM7g by the operation control unit 340 so that at least one of the position and orientation of the operating unit 110 follows the movement of the robot arm 50 and instrument 2. The first control device 310 may make only the position of the operating unit 110 follow the movements of the robot arm 50 and instrument 2, or only the posture of the operating unit 110 follow the movements of the robot arm 50 and instrument 2, or both the position and posture of the operating unit 110 follow the movements of the robot arm 50 and instrument 2. Even when at least one of the position and posture of the operating unit 110 follows the movements of the robot arm 50 and instrument 2, when switching the robot arm 50 and instrument 2 from autonomous mode to remote control mode, there is a possibility that the state of the operating unit 110 may not be in a state where the robot arm 50 and instrument 2 can be switched from autonomous mode to remote control mode due to some factor. Therefore, the first control device 310 determines whether the state of the operating unit 110 is in a state where the robot arm 50 and instrument 2 can be switched from autonomous mode to remote control mode. Furthermore, in Figure 24, only the instrument 2 is shown among the robot arm 50 and the instrument 2.

[0107] As described above, the operating unit 110 includes link sections 111a, 111b, 111c, 112a, 112b, 112c, and a grip support member 112d. In the following description, the grip support member 112d is also referred to as a link section. The operating unit 110 also includes servo motors SM7a to SM7g that drive the above-mentioned plurality of link sections. In autonomous mode, the servo motors SM7a to SM7g change the position and orientation of at least one of the plurality of link sections to follow at least one of the position and orientation of the robot arm 50 and the instrument 2. That is, the plurality of link sections of the operating unit 110 rotate automatically to correspond to at least one of the position and orientation of the robot arm 50 and the instrument 2 in autonomous mode, and the position and orientation of at least one of the plurality of link sections is changed.

[0108] Furthermore, in the first embodiment, the first control device 310 may, while the robot arm 50 and instrument 2 are operating in autonomous mode, perform control to switch the robot arm 50 and instrument 2 from autonomous mode to remote control mode based on detection of operator operation on the operating unit 110 that follows the movement of the robot arm 50 and instrument 2. In this case, the first control device 310 performs control to detect operator operation to switch the robot arm 50 and instrument 2 from autonomous mode to remote control mode based on the load on the operating unit 110 or the amount of operation on the operating unit 110.

[0109] For example, the first control device 310 detects that an operator has performed an action to switch the robot arm 50 and instrument 2 from autonomous mode to remote control mode if the load applied by the operator to the control unit 110 exceeds a predetermined load. In this case, for example, the first control device 310 performs an action to acquire the load applied by the operator to the control unit 110 based on the detection value of the force sensor provided on the control unit 110, or the current values ​​of the servo motors SM7a, SM7b, SM7c, SM7d, SM7e, SM7f, and SM7g. Also, for example, the first control device 310 detects that an operator has performed an action to switch the robot arm 50 and instrument 2 from autonomous mode to remote control mode if the amount of operation of the control unit 110 generated by the operator moving the control unit 110 exceeds a predetermined amount. In this case, for example, the first control device 310 performs control to acquire the amount of operation of the operation unit 110 generated by the operator moving the operation unit 110, based on the detected values ​​of the encoders EN7a, EN7b, EN7c, EN7d, EN7e, EN7f, and EN7g. Alternatively, a contact sensor or proximity sensor may be placed on the operation unit 110, and the operator's operation may be detected by the sensor placed on the operation unit 110. Furthermore, the operator's operation may be detected by combining multiple detection methods. In this case, it is possible to improve the detection accuracy.

[0110] (Control Method for Surgical Support System) Next, the control method for the surgical support system 500 will be described. In the following control method for the surgical support system 500, the operations performed by the first control device 310 are performed based on the program 311a stored in the storage unit 311. Note that the control method for the surgical support system 500 is an example of a control method for a manipulation system.

[0111] As shown in Figure 25, in step S1, the first control device 310 performs control to determine whether or not the operator has performed an operation to switch the robot arm 50 and instrument 2 from autonomous mode to remote control mode. If it is determined that the operator has not performed an operation to switch the robot arm 50 and instrument 2 from autonomous mode to remote control mode, the process in step S1 is repeated. If it is determined that the operator has performed an operation to switch the robot arm 50 and instrument 2 from autonomous mode to remote control mode, the process proceeds to step S2.

[0112] Then, in step S2, the first control device 310 performs control to acquire the state of the operation unit 110. Specifically, the first control device 310 performs control to acquire at least one of the position and orientation of the operation unit 110 as the state of the operation unit 110. Then, in step S3, the first control device 310 performs control to determine, based on the state of the operation unit 110, whether the state of the operation unit 110 is such that the robot arm 50 and instrument 2 can be switched from autonomous mode to remote control mode. If it is determined that the state of the operation unit 110 is such that the robot arm 50 and instrument 2 can be switched from autonomous mode to remote control mode, the process proceeds to step S4.

[0113] Then, in step S4, the first control device 310 performs control to switch the robot arm 50 and instrument 2 from autonomous mode to remote control mode. If the operating unit 110 determines that the state is not such that the robot arm 50 and instrument 2 can be switched from autonomous mode to remote control mode, the process proceeds to step S5. Then, in step S5, the first control device 310 performs control to notify a warning.

[0114] [Effects of the First Embodiment] The surgical support system 500 includes a first control device 310 that acquires the state of the operation unit 110 and performs control to determine whether the state of the operation unit 110 is such that the robot arm 50 and instrument 2 can be switched from autonomous mode to remote control mode, based on the acquired state of the operation unit 110. As a result, when the state of the operation unit 110 is such that the robot arm 50 and instrument 2 can be switched from autonomous mode to remote control mode, the robot arm 50 and instrument 2 can be switched from autonomous mode to remote control mode.

[0115] If the first control device 310 determines that the state of the operation unit 110 is such that the robot arm 50 and instrument 2 can be switched from autonomous mode to remote control mode, it executes the switch of the robot arm 50 and instrument 2 from autonomous mode to remote control mode. If the first control device 310 determines that the state of the operation unit 110 is such that the robot arm 50 and instrument 2 cannot be switched from autonomous mode to remote control mode, it performs control to notify that the state of the operation unit 110 is such that the robot arm 50 and instrument 2 cannot be switched from autonomous mode to remote control mode. As a result, if the state of the operation unit 110 is such that it is such that the robot arm 50 and instrument 2 can be switched from autonomous mode to remote control mode, it can execute the switch of the robot arm 50 and instrument 2 from autonomous mode to remote control mode. Furthermore, if the operating unit 110 determines that the robot arm 50 and instrument 2 are not in a state where they can be switched from autonomous mode to remote control mode, it can notify the user that the operating unit 110 is not in a state where they can be switched from autonomous mode to remote control mode. This allows the user to be informed that the operating unit 110 is not in a state where they can be switched from autonomous mode to remote control mode, while suppressing sudden changes in command values ​​that could worsen operability.

[0116] The first control device 310 performs control to determine whether the state of the operating unit 110 is such that the robot arm 50 and instrument 2 can be switched from autonomous mode to remote control mode, based on at least one of the position and orientation of the operating unit 110. As the position and orientation of the operating unit 110 are information correlated with the state of the robot arm 50 and instrument 2, the first control device 310 can accurately determine whether the state of the operating unit 110 is such that the robot arm 50 and instrument 2 can be switched from autonomous mode to remote control mode.

[0117] The control unit 110 is equipped with servo motors SM7a, SM7b, SM7c, SM7d, SM7e, SM7f, and SM7g. When the robot arm 50 and instrument 2 are operating in autonomous mode, the first control device 310 controls the servo motors SM7a, SM7b, SM7c, SM7d, SM7e, SM7f, and SM7g so that at least one of the position and orientation of the control unit 110 follows the movements of the robot arm 50 and instrument 2. This allows the state of the robot arm 50 and instrument 2 to be synchronized with the state of the control unit 110 when the robot arm 50 and instrument 2 are operating in autonomous mode. As a result, the state of the control unit 110 can be easily switched from autonomous mode to remote control mode for the robot arm 50 and instrument 2. Furthermore, user training can be conducted where an inexperienced operator grasps the control unit 110 and moves it in accordance with the movements of the robot arm 50 and instrument 2, allowing them to learn the operation of an expert.

[0118] The first control device 310, while the robot arm 50 and instrument 2 are operating in autonomous mode, detects an operator's operation on the control unit 110 that follows the movements of the robot arm 50 and instrument 2, and performs control to switch the robot arm 50 and instrument 2 from autonomous mode to remote control mode. As a result, the operator can easily switch the robot arm 50 and instrument 2 from autonomous mode to remote control mode simply by operating on the control unit 110 that follows the movements of the robot arm 50 and instrument 2 while they are operating in autonomous mode.

[0119] The robot arm 50 and instrument 2 can also be switched between autonomous mode and remote control mode. Unlike the case where only one of the robot arm 50 or instrument 2 can be switched between autonomous mode and remote control mode, in autonomous mode the operator does not need to perform any operations, thus further reducing the burden on the operator.

[0120] The surgical support system 500 includes a surgical support robot 100, which includes a plurality of robot arms 50 to which surgical instruments 1 are attached at their tips, and a remote control device 200, which includes an operating unit 110 that receives operations on the surgical instruments 1. This makes it possible to suppress the deterioration of operability caused by a sudden change in command values ​​due to a mismatch between the operating unit 110 and the robot arms 50 when switching the robot arms 50 from autonomous mode to remote control mode in the surgical support system 500.

[0121] The multiple robot arms 50 include a robot arm 50c to which an endoscope 3, which serves as a surgical instrument 1, is attached as a tip, and robot arms 50a, 50b, and 50d to which instruments 2, which are surgical instruments 1 other than the endoscope 3, are attached as tips. The endoscope 3 on robot arm 50c and the instruments 2 on robot arms 50a, 50b, and 50d enable smooth surgery on patient P.

[0122] The autonomous mode operation includes suturing to suture the surgical site of patient P. This prevents abrupt changes in command values ​​and deterioration of operability that can occur when switching the robot arm from autonomous mode to remote control mode during the suturing of patient P's surgical site, due to a mismatch between the control unit 110 and the robot arm.

[0123] The first control device 310 performs control to determine whether the state of the operating unit 110 is such that the robot arm 50 and instrument 2 can be switched from autonomous mode to remote control mode, based on the opening angle of the pair of grip members 112e. As a result, since the opening angle of the pair of grip members 112e is correlated with the opening angles of the jaw members 2g and 2h, which represent the state of the robot arm 50 and instrument 2, the operating unit 110 can accurately determine whether the state of the operating unit 110 is such that the robot arm 50 and instrument 2 can be switched from autonomous mode to remote control mode.

[0124] [Second Embodiment] Next, a second embodiment of the present disclosure will be described with reference to Figures 26, 27, and 28. The second embodiment will describe an example in which the present disclosure is applied to a manipulation system different from that of the first embodiment.

[0125] (Configuration of the Manipulation System) As shown in Figure 26, the manipulation system 700 comprises a remote control device 710, a robot 720, a control device 730, a storage device 740, and a display device 750. The remote control device 710 is an example of an operating device.

[0126] The remote control device 710 is provided for operating the robot 720 from a remote location located away from the robot 720. The remote control device 710 includes an operating unit 711 that receives input from the operator. The remote control device 710 outputs commands to the control device 730 based on the operator's input received by the operating unit 711. The operating unit 711 is a handheld type of operating unit that can be grasped and carried by the operator. The operating unit 711 operates the robot arm 721, which will be described later, by changing its position and orientation. When the position and orientation of the operating unit 711 are changed, the position and orientation of the robot arm 721 are changed in accordance with the change in the position and orientation of the operating unit 711. The remote control device 710 includes one or more sensors for detecting the position and orientation of the operating unit 711. The sensors are, for example, acceleration sensors and angular velocity sensors. Based on the sensor detection values, it is possible to obtain the position and orientation of the operating unit 711. The operating unit 711 does not have actuators such as servo motors. In other words, the operating unit 711 does not have a drive source to change its own position and orientation.

[0127] Robot 720 is a social robot. Robot 720 is equipped with an interface for dialogue and is capable of communicating with people. Robot 720 comprises a pair of articulated robot arms 721 and a trolley 722. The robot arms 721 are configured to be switchable between a remote control mode, which operates based on the operation of the control unit 711, and an autonomous mode, which operates autonomously. The trolley 722 may also be switchable between a remote control mode and an autonomous mode. The remote control mode and autonomous mode of the robot arms 721 or the trolley 722 are switched by the operation of the control unit 711. Furthermore, the robot arms 721 are an example of a manipulation device.

[0128] The control device 730 controls various parts of the manipulation system 700. The control device 730 controls the robot 720 based on commands output from the remote control device 710. The control device 730 includes a processor such as a CPU (Central Processing Unit) and a memory for storing information. The control device 730 performs various control processes by executing a program 741 stored in the storage device 740.

[0129] The storage device 740 includes non-volatile memory such as a hard disk drive and stores the program 741 that the control device 730 executes.

[0130] The display device 750 includes a display such as a liquid crystal display or an organic EL (Electroluminescence) display, and displays various information related to the manipulation system 700. For example, the display device 750 displays an image of the robot arm 721 of the robot 720 captured by an imaging unit such as a camera. The display device 750 is positioned so that it can be seen by the operator operating the remote control device 710. The display device 750 is positioned closer to the remote control device 710 than to the robot 720. Note that the display device 750 may be integrated with the remote control device 710 rather than being a separate unit.

[0131] In the second embodiment, the control device 730 acquires the state of the operation unit 711 and, based on the acquired state of the operation unit 711, performs control to determine whether the state of the operation unit 711 allows the robot arm 721 to be switched from autonomous mode to remote control mode. In the second embodiment, if the control device 730 determines that the state of the operation unit 711 allows the robot arm 721 to be switched from autonomous mode to remote control mode, it performs the switch of the robot arm 721 from autonomous mode to remote control mode. If the control device 730 determines that the state of the operation unit 711 does not allow the robot arm 721 to be switched from autonomous mode to remote control mode, it performs control to notify the user, as guidance information described later, that the state of the operation unit 711 does not allow the robot arm 721 to be switched from autonomous mode to remote control mode. The determination of whether the state of the operation unit 711 allows the robot arm 721 to be switched from autonomous mode to remote control mode can be performed in the same manner as in the first embodiment.

[0132] In the second embodiment, as shown in Figure 27, when the control device 730 switches the robot arm 721 from autonomous mode to remote control mode, it performs control to notify guidance information for aligning at least one of the position and orientation of the operation unit 711 with the robot arm 721. The control device 730 may notify guidance information for aligning only the position of the operation unit 711 with the robot arm 721, or for aligning only the orientation of the operation unit 711 with the robot arm 721, or for aligning both the position and orientation of the operation unit 711 with the robot arm 721. The control device 730 performs control to notify the guidance information by at least one of display and sound. For example, the control device 730 performs control to display the guidance information on the display device 750 as text information, graphic information, or a combination thereof. For example, a cubic display representing the current state of the control unit 711 or a three-axis vector display may be displayed as graphic information, and work may be performed to match the display. Alternatively, a numerical value indicating the degree of agreement between the current state of the control unit 711 and the state of the robot arm 721 may be displayed as graphic information, and work may be performed to increase the degree of agreement. Furthermore, for example, the control device 730 controls the notification unit provided in the manipulation system 700 to announce guidance information as voice. Figure 27 shows an example in which the display device 750 displays guidance information as text information.

[0133] The operator changes at least one of the position and orientation of the remote control device 710 based on guidance information notified by the control device 730. During the change of at least one of the position and orientation of the remote control device 710, updated guidance information corresponding to the change in at least one of the position and orientation of the remote control device 710 may be notified by the control device 730. When at least one of the position and orientation of the remote control device 710 becomes such that the state of the operating unit 711 can be determined to be in a state where the robot arm 721 can be switched from autonomous mode to remote control mode, the control device 730 performs control to notify the operator, by at least one of display and sound, that the state of the operating unit 711 has become in a state where the robot arm 721 can be switched from autonomous mode to remote control mode. The control device 730 then performs control to switch the robot arm 721 from autonomous mode to remote control mode.

[0134] (Control Processing of the Manipulation System) Next, the control method of the manipulation system 700 will be described. In the following control method of the manipulation system 700, the operations performed by the control device 730 are performed based on the program 741 stored in the storage device 740.

[0135] As shown in Figure 28, in step S11, the control device 730 performs control to determine whether the operator has performed an operation to switch the robot arm 721 from autonomous mode to remote control mode. If it is determined that the operator has not performed an operation to switch the robot arm 721 from autonomous mode to remote control mode, the process in step S11 is repeated. If it is determined that the operator has performed an operation to switch the robot arm 721 from autonomous mode to remote control mode, the process proceeds to step S12.

[0136] Then, in step S12, the control device 730 performs control to acquire the state of the operating unit 711. Specifically, the control device 730 performs control to acquire at least one of the position and orientation of the operating unit 711 as the state of the operating unit 711. Then, in step S13, the control device 730 performs control to determine, based on the state of the operating unit 711, whether the state of the operating unit 711 is such that the robot arm 721 can be switched from autonomous mode to remote control mode.

[0137] If the operating unit 711 determines that the robot arm 721 can be switched from autonomous mode to remote control mode, the process proceeds to step S14. In step S14, the control device 730 performs control to switch the robot arm 721 from autonomous mode to remote control mode. If the operating unit 711 determines that the robot arm 721 cannot be switched from autonomous mode to remote control mode, the process proceeds to step S15. In step S15, the control device 730 performs control to notify guidance information. The process then proceeds to step S13. Steps S13 and S15 are repeated until the operating unit 711 determines that the robot arm 721 can be switched from autonomous mode to remote control mode.

[0138] [Effects of the Second Embodiment] The manipulation system 700 includes a control device 730 that acquires the state of the operating unit 711 and, based on the acquired state of the operating unit 711, determines whether the state of the operating unit 711 is such that the robot arm 721 can be switched from autonomous mode to remote control mode. This makes it possible to suppress the deterioration of operability caused by a sudden change in command values ​​due to a mismatch between the operating unit 711 and the robot arm 721 when switching the robot arm 721 from autonomous mode to remote control mode, similar to the first embodiment.

[0139] The operating unit 711 does not have an actuator, and the control device 730 performs control to notify guidance information to align at least one of the position and orientation of the operating unit 711 with the robot arm 721 when switching the robot arm 721 from autonomous mode to remote control mode. As a result, even though the operating unit 711 does not have an actuator, the operator can easily adjust the state of the operating unit 711 to a state where the robot arm 721 can be switched from autonomous mode to remote control mode based on the guidance information to align at least one of the position and orientation of the operating unit 711 with the robot arm 721.

[0140] Furthermore, the other effects of the second embodiment are the same as those of the first embodiment described above.

[0141] [Modifications] It should be understood that the embodiments disclosed herein are illustrative and not restrictive in all respects. The scope of this disclosure is indicated by the claims rather than the description of the embodiments above, and further includes all modifications or variations within the meaning and scope equivalent to the claims.

[0142] In the first embodiment described above, an example of an arm-type operating unit 110 was shown, and in the second embodiment described above, an example of a handheld operating unit 711 was shown, but the disclosure is not limited thereto. For example, the operating unit may be a wearable type operating unit that is attached to a part of the operator's body, such as the arm. This makes it possible to suppress the deterioration of operability caused by a sudden change in command values ​​due to a mismatch between the operating unit and the manipulation device when switching the manipulation device from autonomous mode to remote control mode, even when using a wearable type operating unit.

[0143] In the first embodiment described above, an example was shown in which, in autonomous mode, at least one of the position and orientation of the operating unit 110 follows the movements of the robot arm 50 and the instrument 2. However, the disclosure is not limited thereto. For example, in autonomous mode, the position and orientation of the operating unit 110 may remain stationary without following the movements of the robot arm 50 and the instrument 2. This reduces the control burden on the first control device 310. In this case, as in the second embodiment described above, guidance information may be notified to the operator. This makes it easy to adjust the state of the operating unit 110 to match the robot arm 50 and the instrument 2, even when the position and orientation of the operating unit 110 do not follow the movements of the robot arm 50 and the instrument 2.

[0144] In the first embodiment described above, an example was shown in which both the left and right robot arms 50 and the instruments 2 attached thereto operate autonomously in autonomous mode, but the disclosure is not limited thereto. For example, in autonomous mode, one of the left and right robot arms 50 and the instrument 2 attached thereto may operate autonomously. In this case, a switching unit for switching between autonomous mode and remote control mode is provided for each of the left and right robot arms 50. The operator only needs to operate one of the robot arms 50 and the instrument 2 attached thereto, thus reducing the burden on the operator.

[0145] In the first embodiment described above, the inference unit 613 was shown to take observation information and operation commands from multiple past time points as input and output the next inference operation command, but the disclosure is not limited thereto. For example, the inference unit 613 may take observation information and operation commands from one past time point as input and output the next inference operation command. This reduces the control burden on the inference control device 600.

[0146] In the first embodiment described above, an example was shown in which an image G captured by the endoscope 3 is input to the inference unit 613 as observation information, but the disclosure is not limited to this. For example, a camera may be placed on the ceiling or wall of the operating room, and the image G captured by this camera may be input to the inference unit 613 as observation information. This makes it possible to generate inference operation commands that take into account the environment around the patient P and the robot arm 50. Alternatively, a force sensor may be placed as an environmental information acquisition unit instead of the endoscope 3. This makes it possible to detect interference between the instrument 2 and the surgical area of ​​the patient P using the force sensor, and thus generate inference operation commands that suppress interference.

[0147] In the first embodiment described above, an example was shown in which the orientation of multiple link parts of the operating unit 110 is changed to follow the orientation of the robot arm 50 and instrument 2 in autonomous mode, but the disclosure is not limited thereto. For example, in autonomous mode, the orientation of multiple link parts of the operating unit 110 may remain stationary without being changed to follow the orientation of the robot arm 50 and instrument 2. This reduces the control burden on the first control device 310.

[0148] In the first embodiment described above, an example was shown in which the robot arm 50 and the instrument 2 attached thereto operate in autonomous mode, but the disclosure is not limited thereto. For example, the robot arm 50 may operate in autonomous mode, and the instrument 2 may operate in remote control mode. This allows the operator to concentrate on operating the instrument 2.

[0149] In the first embodiment described above, an example of applying the present disclosure to a surgical support system 500 comprising a surgical support robot 100 and a remote control device 200 was shown, and in the second embodiment described above, an example of applying the present disclosure to a manipulation system comprising a remote control device 710 and a robot 720 as a social robot was shown, but the present disclosure is not limited thereto. For example, as shown in the modified example in Figure 29, the present disclosure may be applied to a manipulation system 800. The manipulation system 800 comprises an industrial robot 810 placed in a factory or the like, a remote control device 820, and a control device 830. The industrial robot 810 includes, for example, a vertical articulated robot 811 and a horizontal articulated robot 812. The industrial robot 810 can switch between a remote control mode in which it operates based on the operation of the remote control device 820 and an autonomous mode in which it operates autonomously. The remote control device 820 includes an operation unit 820a that receives operation from an operator. The control device 830 acquires the state of the operating unit 820a and, based on the acquired state of the operating unit 820a, performs control to determine whether the state of the operating unit 820a allows the industrial robot 810 to be switched from autonomous mode to remote control mode. This prevents sudden changes in command values ​​and deterioration of operability that can occur due to a mismatch between the operating unit 820a and the industrial robot 810 in the manipulation system 800 including the industrial robot 810. Note that the industrial robot 810 is an example of a manipulation device. The remote control device 820 is an example of an operating device.

[0150] The first embodiment described above shows an example in which multiple robot arms 50 are arranged, but the disclosure is not limited thereto. For example, the disclosure may be applied to a surgical support system in which only one robot arm 50 is arranged. In this case, either the robot arm 50 or the instrument 2 can be switched between autonomous mode and remote control mode. This makes it possible to smoothly switch between autonomous mode and remote control mode even in a surgical support system in which only one robot arm 50 is arranged.

[0151] In the first embodiment described above, an example was shown in which the operation command included a suturing operation command for suturing the surgical site of patient P, but the present disclosure is not limited thereto. For example, the operation command may be a command other than a suturing operation command. For example, the operation command may be an excision command for excising the surgical site of patient P. That is, the operation in autonomous mode may include an excision operation for excising the surgical site of patient P. Furthermore, the operation command may include multiple types of surgical operation commands, such as a suturing operation command and an excision command. That is, the operation in autonomous mode may include multiple types of surgical operations, such as a suturing operation and an excision operation. In this way, in the present disclosure, even in surgical operations other than suturing, when switching the robot arm 50 and instrument 2 from autonomous mode to remote control mode, it is possible to suppress the sudden change in command values ​​and deterioration of operability caused by a mismatch between the operation unit 110 and the robot arm 50 and instrument 2.

[0152] In the first embodiment described above, an example was shown in which the switching unit for switching between autonomous mode and remote control mode is a coagulation pedal 126L and another foot pedal 120, but the present disclosure is not limited thereto. For example, the switching unit of the present disclosure may be a switching unit for switching between autonomous mode and remote control mode by detecting the movement of the operator's gaze. In this case, for example, a sensor for detecting the movement of the operator's gaze is arranged on the monitor 140. Alternatively, the switching unit of the present disclosure may be a switching unit for switching between autonomous mode and the remote control mode based on the detection of a part of the operator's body. In this case, a sensor for detecting the movement of a part of the operator's body is arranged. For example, the sensor detects the movement of the operator's arm or head. By switching between autonomous mode and remote control mode with such a switching unit, the operator does not need to take their hands off the control unit to switch between autonomous mode and remote control mode, so that the operator can switch between autonomous mode and remote control mode while continuing to perform suturing work. Alternatively, the switching unit of the present disclosure may be a button separately arranged on the control unit 110. This allows the buttons to be operated manually, making it easy to switch between autonomous mode and the remote control mode.

[0153] In the first embodiment described above, an example was shown in which, in autonomous mode, a manual operation command is replaced with an inference operation command before the limiting unit of the first control device 310 limits the speed of the manual operation command. However, the disclosure is not limited thereto. For example, in autonomous mode, a manual operation command may be replaced with an inference operation command before the limiting unit of the first control device 310 limits the acceleration of the manual operation command. This makes it possible to limit the acceleration of the operation of the surgical support system in the same way in both autonomous mode and remote control mode.

[0154] In the first embodiment described above, an example was shown in which the autonomous mode is executed while the operator is pressing the coagulation pedal 126L and the remote control mode is executed while the operator is releasing the coagulation pedal 126L. However, the disclosure is not limited thereto. For example, conversely to the above operation, the remote control mode may be executed while the coagulation pedal 126L is pressed, and the autonomous mode may be executed while the coagulation pedal 126L is released. Alternatively, the autonomous mode and the remote control mode may switch each time the coagulation pedal 126L is pressed. In this way, even if the operation for switching between the autonomous mode and the remote control mode is different, the autonomous mode and the remote control mode can be easily switched with the simple operation of pressing the coagulation pedal 126L. Furthermore, depending on the learning proficiency of the inference unit 613, initially the autonomous mode is executed while the operator is pressing the solidification pedal 126L, and the remote control mode is executed while the operator is releasing the solidification pedal 126L. Once the proficiency exceeds a predetermined standard, the remote control mode may be executed while the solidification pedal 126L is pressed, and the autonomous mode may be executed while the solidification pedal 126L is released.

[0155] In the first embodiment described above, an example was shown in which the autonomous mode and remote control mode are switched by the coagulation pedal 126L, and the start and stop of the training data storage operation are switched by the coagulation pedal 126R. However, the disclosure is not limited to this. For example, the above switching may be performed by detecting the movement of the operator's gaze or by detecting a part of the operator's body other than their hands. This allows the above switching to be performed while the operator continues to operate the control unit 110.

[0156] In the first embodiment described above, an example was shown in which the first control device 310 and the inference control device 600 are arranged separately, but the disclosure is not limited thereto. For example, the first control device 310 may also serve as the inference control device 600. This simplifies the configuration of the surgical support system 500.

[0157] Furthermore, although the first embodiment described above shows an example in which four robot arms 50 are provided, the present disclosure is not limited to this. In the present disclosure, the number of robot arms 50 may be other than four.

[0158] Furthermore, although the first embodiment described above shows an example in which the arm portion 51 and the positioner 30 are composed of a 7-axis articulated robot, the disclosure is not limited to this. For example, the arm portion 51 and the positioner 30 may be composed of an articulated robot with an axis configuration other than a 7-axis articulated robot. An axis configuration other than a 7-axis articulated robot would be, for example, 6 axes or 8 axes.

[0159] Furthermore, while the first embodiment described above shows an example in which the surgical assistance robot 100 includes a medical cart 10, a positioner 30, an arm base 40, and a robot arm 50, this disclosure is not limited to this. For example, the medical cart 10, the positioner 30, and the arm base 40 are not necessarily required, and the surgical assistance robot 100 may include only a robot arm 50.

[0160] Furthermore, while the first embodiment described above shows an example in which the jaw members 2g and 2h are opened and closed in accordance with the opening and closing of a pair of grip members 112e, the disclosure is not limited thereto. For example, buttons for opening and closing the jaw members 2g and 2h may be placed on the operating unit 110, and the jaw members 2g and 2h may be opened and closed by operating the buttons on the operating unit 110.

[0161] The functions of the elements disclosed herein may be implemented using one or more circuits or processing circuits, including general-purpose processors, special-purpose processors, integrated circuits, ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and / or conventional circuits. The functions of the elements disclosed herein may be implemented using one or more circuits or processing circuits, including combinations of general-purpose processors, special-purpose processors, integrated circuits, ASICs, FPGAs, and conventional circuits. One or more circuits or processing circuits may be programmed using one or more programs stored together or individually in one or more memories, or otherwise configured to perform the disclosed functions. A processor is considered a processing circuit or circuit because it includes transistors and other circuits. A processor may be a programmed processor that executes programs stored in memory. In this disclosure, a circuit, unit, or means is hardware that performs the enumerated functions individually or in combination with each other, or hardware programmed to perform the enumerated functions individually or in combination with each other. Hardware may be any hardware disclosed herein that is programmed or configured to perform the enumerated functions. A computer program, including computer instructions, is stored in memory. Computer instructions provide logic and routines that enable the hardware to perform the methods disclosed herein. Hardware includes, for example, processing circuits or circuits. A computer program may be implemented in known formats on computer-readable storage media, computer program products, memory devices, recording media such as CD-ROMs or DVDs, and / or in the memory of FPGAs or ASICs.

[0162] [Embodiments] The exemplary embodiments described above will be understood by those skilled in the art to be specific examples of the following embodiments.

[0163] (Aspect 1) A manipulation system comprising: an operating device including an operating unit that receives operations from an operator; a manipulation device capable of switching between a remote operation mode that operates based on the operation of the operating unit of the operating device and an autonomous mode that operates autonomously; and a control device that acquires the state of the operating unit and, based on the acquired state of the operating unit, performs control to determine whether the state of the operating unit is such that the manipulation device can be switched from the autonomous mode to the remote operation mode.

[0164] (Aspect 2) The manipulation system according to aspect 1, wherein the control device, when it determines that the state of the operation unit is such that the manipulation device can be switched from the autonomous mode to the remote control mode, executes the switching of the manipulation device from the autonomous mode to the remote control mode, and when it determines that the state of the operation unit is such that the manipulation device cannot be switched from the autonomous mode to the remote control mode, performs control to notify that the state of the operation unit is such that the manipulation device cannot be switched from the autonomous mode to the remote control mode.

[0165] (Aspect 3) The manipulation system according to aspect 1 or aspect 2, wherein the control device performs control to determine whether the state of the operating unit is such that the manipulation device can be switched from the autonomous mode to the remote control mode, based on at least one of the position and orientation of the operating unit.

[0166] (Aspect 4) The manipulation system according to any one of aspects 1 to 3, wherein an actuator is arranged in the operating section, and the control device controls the actuator such that at least one of the position and orientation of the operating section follows the movement of the manipulation device when the manipulation device is operating in the autonomous mode.

[0167] (Aspect 5) The manipulation system according to aspect 4, wherein the control device controls the manipulation device to switch from the autonomous mode to the remote control mode based on the detection of an operator's operation on the control unit that follows the operation of the manipulation device while the manipulation device is operating in the autonomous mode.

[0168] (Aspect 6) The manipulation system according to any one of aspects 1 to 3, wherein the operating unit does not have an actuator, and the control device performs control to notify guidance information for aligning at least one of the position and orientation of the operating unit with the manipulation device when switching the manipulation device from the autonomous mode to the remote control mode.

[0169] (Aspect 7) The manipulation system according to any one of aspects 1 to 6, wherein the manipulation device includes a multi-joint robot arm and an instrument having joints, and the multi-joint robot arm and the instrument can be switched between the autonomous mode and the remote control mode.

[0170] (Aspect 8) The manipulation system according to any one of aspects 1 to 7, comprising: a surgical apparatus including a robotic arm as a manipulation device to which a surgical instrument is attached at its tip; and an operating device including an operating section for receiving operations on the surgical instrument.

[0171] (Aspect 9) The manipulation system according to aspect 8, wherein the robotic arm includes a first robotic arm to which an endoscope as a surgical instrument is attached as a tip, and a second robotic arm to which a surgical instrument other than the endoscope is attached as a tip.

[0172] (Aspect 10) The manipulation system according to aspect 9, wherein the operation of the autonomous mode includes a suturing operation for suturing the surgical site of a patient.

[0173] (Aspect 11) The manipulation system according to any one of aspects 1 to 10, wherein the operating unit includes a pair of grip members that are opened and closed by an operator, and the control unit performs control to determine whether the state of the operating unit is such that the manipulation device can be switched from the autonomous mode to the remote control mode based on the opening angle of the pair of grip members.

[0174] (Aspect 12) A control method for a manipulation system comprising an operating device including an operating unit that receives operations from an operator, and a manipulation device that can switch between a remote operation mode that operates based on the operation of the operating unit of the operating device and an autonomous mode that operates autonomously, the control method comprising: acquiring the state of the operating unit, and determining, based on the acquired state of the operating unit, whether the state of the operating unit is such that the manipulation device can be switched from the autonomous mode to the remote operation mode.

[0175] (Aspect 13) A program for executing a control method for a manipulation system comprising an operating device including an operating unit that receives operations from an operator, and a manipulation device that can switch between a remote operation mode that operates based on the operation of the operating unit of the operating device and an autonomous mode that operates autonomously, the program comprising: acquiring the state of the operating unit, and determining, based on the acquired state of the operating unit, whether the state of the operating unit is such that the manipulation device can be switched from the autonomous mode to the remote operation mode.

[0176] 1 Surgical instruments 2 Instruments (manipulation devices) 2c Wrist joint (joint) 3 Endoscope 50 Robot arm (manipulation device, articulated robot arm) 50a Robot arm (second robot arm) 50b Robot arm (second robot arm) 50c Robot arm (first robot arm) 50d Robot arm (second robot arm) 100 Surgical support robot (surgical device) 110 Control unit 200 Remote control device (control device) 310 First control device (control device) 311a Program 500 Surgical support system (manipulation system) 700 Manipulation system 710 Remote control device (control device) 711 Control unit 721 Robot arm (manipulation device) 730 Control device 741 Program 800 Manipulation system 810 Industrial robot (manipulation device) 820 Remote control device (operating device) 820a Operating unit 830 Control device P Patient SM7a to SM7g Servo motor (actuator)

Claims

1. A manipulation system comprising: an operating device including an operating unit that receives input from an operator; a manipulation device capable of switching between a remote operation mode that operates based on the operation of the operating unit of the operating device and an autonomous mode that operates autonomously; and a control device that acquires the state of the operating unit and, based on the acquired state of the operating unit, performs control to determine whether the state of the operating unit is such that the manipulation device can be switched from the autonomous mode to the remote operation mode.

2. The manipulation system according to claim 1, wherein the control device, when it determines that the state of the operation unit is such that the manipulation device can be switched from the autonomous mode to the remote control mode, executes a switch of the manipulation device from the autonomous mode to the remote control mode, and when it determines that the state of the operation unit is such that the manipulation device cannot be switched from the autonomous mode to the remote control mode, performs control to notify the operation unit that the state of the operation unit is such that the manipulation device cannot be switched from the autonomous mode to the remote control mode.

3. The manipulation system according to claim 1, wherein the control device performs control to determine whether the state of the operating unit is such that the manipulation device can be switched from the autonomous mode to the remote control mode, based on at least one of the position and orientation of the operating unit.

4. The manipulation system according to claim 1, wherein an actuator is arranged in the operating unit, and the control device controls the actuator such that at least one of the position and orientation of the operating unit follows the movement of the manipulation device when the manipulation device is operating in the autonomous mode.

5. The manipulation system according to claim 4, wherein the control device performs control to switch the manipulation device from the autonomous mode to the remote control mode based on the detection of an operator's operation on the operation unit that follows the operation of the manipulation device while the manipulation device is operating in the autonomous mode.

6. The manipulation system according to claim 1, wherein the operating unit does not have an actuator, and the control device performs control to notify guidance information for aligning at least one of the position and orientation of the operating unit with the manipulation device when switching the manipulation device from the autonomous mode to the remote control mode.

7. The manipulation system according to claim 1, wherein the manipulation device includes a multi-joint robot arm and an instrument having joints, and the multi-joint robot arm and the instrument can be switched between the autonomous mode and the remote control mode.

8. The manipulation system according to claim 1, comprising: a surgical apparatus including a robotic arm as a manipulation device to which a surgical instrument is attached at its tip; and an operating device including an operating section for receiving operations on the surgical instrument.

9. The manipulation system according to claim 8, wherein the robotic arm includes a first robotic arm to which an endoscope as a surgical instrument is attached as a tip, and a second robotic arm to which a surgical instrument other than the endoscope is attached as a tip.

10. The manipulation system according to claim 9, wherein the operation of the autonomous mode includes a suturing operation for suturing the surgical site of a patient.

11. The manipulation system according to claim 1, wherein the operating unit includes a pair of grip members that are opened and closed by an operator, and the control unit performs control to determine whether the state of the operating unit is such that the manipulation device can be switched from the autonomous mode to the remote control mode, based on the opening angle of the pair of grip members.

12. A control method for a manipulation system comprising an operating device including an operating unit that receives operations from an operator, and a manipulation device capable of switching between a remote operation mode that operates based on operations of the operating unit of the operating device, and an autonomous mode that operates autonomously, the method comprising: acquiring the state of the operating unit, and determining, based on the acquired state of the operating unit, whether the state of the operating unit is such that the manipulation device can be switched from the autonomous mode to the remote operation mode.

13. A program for executing a control method for a manipulation system comprising an operating device including an operating unit that accepts operator input, and a manipulation device capable of switching between a remote operation mode that operates based on the operation of the operating unit of the operating device, and an autonomous mode that operates autonomously, the program comprising: acquiring the state of the operating unit, and determining, based on the acquired state of the operating unit, whether the state of the operating unit is such that the manipulation device can be switched from the autonomous mode to the remote operation mode.