Manipulation system, control method for manipulation system, and program

The manipulation system addresses the challenge of improving robot operation performance and model creation by incorporating a state and evaluation acquisition unit to generate an inferred operation command, leading to enhanced training and performance through comprehensive data collection.

WO2026239759A1PCT designated stage Publication Date: 2026-11-19KAWASAKI JUKOGYO KK
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2026/013051
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2026-01-15
Filing Date
2026-03-30
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Existing manipulation systems using trained machine learning models for robot operations face challenges in improving performance and ease of model creation, particularly due to the exclusion of data from failed operations.

Method used

A manipulation system that includes a state acquisition unit, an evaluation acquisition unit, and a learning model generation unit, which collects input information on operation states and environment data to generate an inferred operation command, considering both high-quality and low-quality operations, thereby enhancing the learning model's performance and ease of creation.

Benefits of technology

The system allows for improved operation performance of manipulation devices by generating a learning model that can handle various operations effectively, facilitating easier training and data collection, thus enhancing the robot's operational capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2026013051_19112026_PF_FP_ABST
    Figure JP2026013051_19112026_PF_FP_ABST
Patent Text Reader

Abstract

A surgery assistance system (500) comprises: a remote operation device (200) that outputs an operation command on the basis of an operation by an operator; and a robot arm (50) and an instrument (2) that operate by being operated on the basis of the operation command. The surgery assistance system is also provided with a trainer (612) that, on the basis of input information, the environment around the operated robot arm and instrument, and operation evaluation, generates a trained model that receives at least one of an operation command and observation information on the environment around the robot arm and instrument, and outputs an inferred operation command that is an operation command being inferred.
Need to check novelty before this filing date? Find Prior Art

Description

Manipulation System, Control Method for Manipulation System, and Program

[0001] This disclosure relates to a manipulation system, a control method for a manipulation system, and a program.

[0002] Conventionally, manipulation systems including a manipulation device that operates based on operation commands are known. Japanese Patent Laid-Open No. 2024-101163 discloses a system that operates a robot as a control target using a trained machine learning model. In this system, when acquiring the trained machine learning model, data from cases where the robot's operation failed is not used, and the trained machine learning model is acquired using data from cases where the robot was operated and the operation succeeded.

[0003] Japanese Patent Laid-Open No. 2024-101163

[0004] Here, in systems that operate a robot using a trained machine learning model as described in Japanese Patent Laid-Open No. 2024-101163, it is desirable to improve the performance of the robot's operation by the trained machine learning model, and to enable easy creation of the trained machine learning model.

[0005] The present disclosure provides a manipulation system, a control method for a manipulation system, and a program that can improve the operation performance of a manipulation device by a trained model and enable easy creation of the trained model.

[0006] A manipulation system according to the first aspect of this disclosure includes: a manipulation device that operates based on an operation command; a state acquisition unit that acquires input information including at least one of the following: information on the operation state of an operation device that outputs an operation command based on an operation by an operator; information on the operation state of the manipulation device; and information on the operation state of an instrument corresponding to the manipulation device that has been manually operated; an evaluation acquisition unit that acquires an evaluation of the operation of the manipulation device or instrument that operates in response to the input information; and a learning model generation unit that generates a trained model that takes at least one of the following as input: observation information of the environment surrounding the manipulation device and an operation command, and outputs an inferred operation command, which is an inferred operation command, based on the input information, the environment surrounding the operated manipulation device, and the operation evaluation.

[0007] The manipulation system according to the first aspect of this disclosure includes a learning model generation unit that, based on input information, the environment surrounding the operated manipulation device, and an operation evaluation, takes at least one of the observed information of the environment surrounding the manipulation device and an operation command as input and generates an inferred operation command as output. This allows learning to be performed while taking operation evaluations into account for various operations, including high-quality and low-quality operations, thereby generating a learning model capable of performing higher-quality operations. Furthermore, since more operation data can be collected compared to not using data from failed operations, operation data can be collected easily. This facilitates the training of the learning model. As a result, the performance of the operation of the manipulation device by the learning model is improved, and the learning model can be easily created.

[0008] A control method for a manipulation system according to the second aspect of this disclosure is a control method for a manipulation system including a manipulation device that operates based on an operation command, comprising: acquiring input information including at least one of the following: information on the operation state of an operating device that outputs an operation command based on an operation by an operator; information on the operation state of the manipulation device; and information on the operation state of an instrument corresponding to the manipulation device that has been manually operated; acquiring an operation evaluation of the manipulation device or instrument that operates in response to the input information; and generating a trained model that takes at least one of the following as input: observation information of the environment around the manipulation device and an operation command, and outputs an inferred operation command, based on the input information, the environment around the operated manipulation device, and the operation evaluation.

[0009] The control method for a manipulation system according to the second aspect of this disclosure, as described above, comprises generating a trained model that takes at least one of the observed information of the environment surrounding the manipulation device and an operation command as input, and outputs an inferred operation command, based on input information, the environment surrounding the operated manipulation device, and an operation evaluation. This allows learning to be performed while taking operation evaluations into account for various operations, including high-quality and low-quality operations, so that a trained model capable of performing higher-quality operations can be generated. In addition, since more operation data can be collected compared to not using data from failed operations, operation data can be collected easily. This makes it easier to train the trained model. As a result, it is possible to provide a control method for a manipulation system that improves the performance of the operation of the manipulation device by the trained model and makes it possible to easily create the trained model.

[0010] A program according to the third aspect of this disclosure is a program that performs a control method for a manipulation system including a manipulation device that operates based on an operation command, and the program performs a control method for a manipulation system comprising: acquiring input information including at least one of information on the operation state of an operating device that outputs an operation command based on an operation by an operator, information on the operation state of the manipulation device, and information on the operation state of an instrument corresponding to the manipulation device that has been manually operated; acquiring an operation evaluation of the manipulation device or instrument that operates in response to the input information; and generating a trained model that takes at least one of observation information of the environment around the manipulation device and an operation command as input and outputs an inferred operation command, which is an inferred operation command, based on the input information, the environment around the operated manipulation device, and the operation evaluation.

[0011] The program according to the third aspect of this disclosure, as described above, takes at least one of the observed information of the environment surrounding the manipulated device and the operation command as input, and outputs an inferred operation command, based on the input information, the environment surrounding the operated manipulated device, and the operation evaluation. This allows the program to learn various operations, including high-quality and low-quality operations, while taking operation evaluation into account, thereby enabling the generation of a trained model capable of performing higher-quality operations. Furthermore, it allows for the collection of more operation data compared to not using data from failed operations, making it easier to collect operation data. This facilitates the training of the trained model. As a result, the program can improve the performance of the manipulated device's operation using the trained model and also provide a program that makes it easy to create trained models.

[0012] The manipulation system, control method for the manipulation system, and program of this disclosure improve the performance of the operation of the manipulation device using a trained model, and also make it possible to easily create the trained model.

[0013] This is a diagram showing the configuration of a surgical support system according to one embodiment. This is a diagram showing the display unit of a medical trolley according to one embodiment. This is a diagram showing the configuration of a medical trolley according to one embodiment. This is a diagram showing the configuration of a robot arm according to one embodiment. This is a diagram showing the instruments. This is a perspective view showing the configuration of the arm operation unit according to one embodiment. This is a diagram for explaining the translational movement of the robot arm. This is a diagram for explaining the rotational movement of the robot arm. This is a diagram showing an endoscope. This is a diagram showing a pivot position setting device. This is a diagram showing the operation unit according to one embodiment. This is a diagram showing the right-hand wrist unit according to one embodiment. This is a diagram showing the left-hand wrist unit according to one embodiment. This is a perspective view showing a foot pedal according to one embodiment. This is a control block diagram of a surgical support system according to one embodiment. This is a control block diagram of a robot arm according to one embodiment. This is a control block diagram of a positioner and medical trolley according to one embodiment. This is a control block diagram of the operation unit according to one embodiment. This is a diagram for explaining the control of the first control device according to one embodiment. This is a diagram for explaining the configuration of an inference control device according to one embodiment. This is a diagram for explaining the control block of an inference control device for a surgical support system according to one embodiment. This is a diagram showing the architecture of an AI model used in an inference control device according to one embodiment. This is a diagram showing the patient's surgical unit and instruments displayed on the monitor and display unit. This diagram shows the patient's surgical site, TCP coordinate system, etc., as displayed on the monitor and display unit. This is a flowchart illustrating the process of generating a trained model of a surgical support system according to one embodiment. This is a flowchart illustrating the output process of inference operation commands of a surgical support system according to one embodiment. This is a flowchart illustrating the autonomous mode of a surgical support system according to one embodiment. This diagram shows a social robot as an example. This diagram shows an industrial robot as an example.

[0014] (Configuration of the surgical support system) The configuration of the surgical support system 500 according to this embodiment will now be described. The surgical support system 500 comprises a surgical support robot 100, a remote control device 200, a vision unit 300, an image processing unit 400, and an inference control device 600. The surgical support robot 100 is an example of a surgical device. The remote control device 200 is an example of an operating device. The surgical support system 500 is an example of a manipulation system.

[0015] In this specification, as shown in Figure 4, the longitudinal direction of the surgical instrument 1 is defined as the Z direction. The tip side of the surgical instrument 1 is defined as the Z1 side, and the base side of the surgical instrument 1 is defined as the Z2 side. The direction perpendicular to the Z direction is defined as the X direction. The direction perpendicular to both the Z and X directions is defined as the Y direction.

[0016] As shown in Figure 1, the surgical robot 100 is positioned inside the operating room. The remote control device 200 is positioned at a distance from the surgical robot 100. The remote control device 200 and the surgical robot 100 may communicate via a wired connection or wirelessly. The remote control device 200 receives commands for the surgical instrument 1. Specifically, an operator, such as a physician, inputs commands to the remote control device 200 to cause the surgical robot 100 to perform a desired action. The remote control device 200 transmits the input commands to the surgical robot 100. The surgical robot 100 operates based on the received commands. The surgical robot 100 is positioned inside the operating room, which is a sterile field.

[0017] (Configuration of the surgical support robot) As shown in Figure 1, the surgical support robot 100 comprises a medical trolley 10, a trolley positioner operating unit 20, a positioner 30, an arm base 40, a plurality of robot arms 50, and an arm operating unit 60 provided on each robot arm 50. The plurality of robot arms 50 may be arranged on an operating table on which the patient P is placed.

[0018] As shown in Figure 3, the trolley positioner operation unit 20 is supported by a trolley positioner operation support unit 21 at the rear of the medical trolley 10, and the medical trolley 10 or positioner 30 is moved when the trolley positioner operation unit 20 is operated. The trolley positioner operation unit 20 includes an input device 22 and an operation handle 23. The input device 22 accepts operations for moving and changing the posture of the positioner 30, arm base 40, and multiple robot arms 50, mainly for preparing for surgery before the procedure. The medical trolley 10 includes an operation handle 23.

[0019] As shown in Figure 3, the input device 22 includes a display unit 22a, a joystick 22b, an enable switch 22c, an error reset button 22d, and a speaker 22e. The display unit 22a is, for example, a liquid crystal panel. As shown in Figure 2, the display unit 22a displays numbers corresponding to multiple robot arms 50. The display unit 22a also displays the type of surgical instrument 1 attached to each of the multiple robot arms 50. The display unit 22a displays a check mark CM indicating that the pivot position PP, which will be described later, has been set.

[0020] As shown in Figure 3, the joystick 22b is positioned near the display unit 22a of the input device 22. By selecting an operation mode displayed on the display unit 22a and operating the joystick 22b, the positioner 30 is moved in three dimensions.

[0021] The enable switch 22c is located near the joystick 22b. The enable switch 22c allows or disallows the movement of the positioner 30. When the enable switch 22c is pressed and the movement of the positioner 30 is permitted, the joystick 22b is operated, causing the positioner 30 to move.

[0022] The error reset button 22d clears errors in the surgical support system 500. The errors are, for example, deviation anomaly errors. A pair of speakers 22e are provided. The pair of speakers 22e are located near the location where the positioner 30 of the medical trolley 10 is positioned.

[0023] Furthermore, the operating handle 23 is located near the display unit 22a. The operating handle 23 has a throttle 23a that controls the movement of the medical cart 10 when grasped and rotated by an operator such as a nurse or technician. Specifically, the operating handle 23 is located below the input device 22. When the throttle 23a is rotated from the front to the back, the medical cart 10 moves forward. When the throttle 23a is rotated from the back to the front, the medical cart 10 moves backward. The speed of the medical cart 10 is also changed according to the amount of rotation of the throttle 23a. The operating handle 23 is configured to be rotatable left and right, as indicated by the R direction, and the medical cart 10 rotates along with the rotation of the operating handle 23.

[0024] Furthermore, an enable switch 23b is located on the operating handle 23 to allow or disallow the movement of the medical trolley 10. When the enable switch 23b is pressed and the movement of the medical trolley 10 is permitted, the throttle 23a on the operating handle 23 is operated, causing the medical trolley 10 to move.

[0025] As shown in Figure 1, the positioner 30 consists of a 7-axis articulated robot. The positioner 30 is positioned on a medical trolley 10. The positioner 30 is a 7-axis articulated robot that adjusts the position of the arm base 40. The positioner 30 moves the position of the arm base 40 in three dimensions. However, the positioner 30 is not limited to a 7-axis articulated robot; for example, it may be an articulated robot with 6 axes or less, or 8 axes or more.

[0026] The positioner 30 includes a base portion 31 and a plurality of link portions 32 connected to the base portion 31. The plurality of link portions 32 are connected to each other by joints 33.

[0027] The arm base 40 is attached to the tip of the positioner 30. The base of each of the multiple robot arms 50 is attached to the arm base 40. The multiple robot arms 50 can be folded and stored. The arm base 40 and the multiple robot arms 50 are used covered with sterile drapes. The robot arms 50 also support surgical instruments 1.

[0028] The arm base 40 is equipped with a status indicator 41 and an arm status indicator 42, as shown in Figure 15. The status indicator 41 displays the status of the surgical support system 500. The arm status indicator 42 displays the status of the robot arm 50.

[0029] Multiple robot arms 50 are arranged. Specifically, four robot arms 50a, 50b, 50c, and 50d are arranged. Robot arms 50a, 50b, 50c, and 50d have similar configurations to each other. Robot arms 50a, 50b, 50c, and 50d are operated independently based on motion commands. Note that robot arms 50 are examples of manipulation devices and articulated robot arms. Also, robot arms 50a, 50b, and 50d are examples of second robot arms. Also, robot arm 50c is an example of a first robot arm.

[0030] As shown in Figure 4, the robot arm 50 includes an arm section 51, a first link section 52, a second link section 53, and a translational movement mechanism section 54. The robot arm 50 has joints JT1, JT2, JT3, JT4, JT5, JT6, JT7, and JT8. Joints JT1, JT2, JT3, JT4, JT5, JT6, and JT7 each have axes A1, A2, A3, A4, A5, A6, and A7 as rotation axes. JT8 has an axis A8 as a linear motion axis. The arm section 51 includes a base section 51a and a link section 51b.

[0031] The arm section 51 consists of a 7-axis articulated robot arm. The first link section 52 is located at the tip of the arm section 51. The arm operation section 60, which will be described later, is attached to the second link section 53. The translational movement mechanism section 54 is located between the first link section 52 and the second link section 53. A holder 55 for holding the surgical instrument 1 is located on the second link section 53. The translational movement mechanism section 54 translates the holder 55, to which the surgical instrument 1 is attached, between a first position and a second position. The first position is the position of the end of the range of movement of the holder 55 by the translational movement mechanism section 54 along the A8 axis, on the Z2 side. The second position is the position of the end of the range of movement of the holder 55 by the translational movement mechanism section 54 along the A8 axis, on the Z1 side.

[0032] Each of the multiple robotic arms 50 has a surgical instrument 1 attached to its tip. The surgical instrument 1 includes, for example, a replaceable instrument 2, an endoscope 3 shown in Figure 9 for capturing images of the surgical site, and a pivot position setting device 4 shown in Figure 10 for setting the pivot position PP. The instrument 2 includes a driven unit 2a, an end effector 2b, a wrist joint 2c shown in Figure 5, and a shaft 2d. The end effector 2b is connected to the tip of the shaft 2d via the wrist joint 2c.

[0033] As shown in Figure 1, an endoscope 3 is attached to the tip of one of the multiple robot arms 50, for example, robot arm 50c, and instruments 2, which are surgical instruments 1 other than the endoscope 3, are attached to the tips of the remaining robot arms 50a, 50b, and 50d. Robot arms 50b and 50d are positioned adjacent to robot arm 50c. Robot arm 50a is positioned adjacent to robot arm 50b. Preferably, the endoscope 3 is attached to one of the two centrally positioned robot arms 50b and 50c, of the four robot arms 50 that are positioned adjacent to each other. The endoscope 3 observes the environment around robot arms 50a, 50b, and 50d and outputs the observation information. Specifically, the endoscope 3 images the surgical site of patient P as the environment around robot arms 50a, 50b, and 50d, and outputs the captured images to the monitor 140 of the remote control device 200 and the display unit 220 located on the cart 210. The endoscope 3 is an example of a state acquisition unit, an environmental information acquisition unit, and an imaging unit. The monitor 140 and display unit 220 are examples of display units.

[0034] (Instrument Configuration) As shown in Figure 5, an end effector 2b having jaw members 2g and 2h is attached to the tip of the instrument 2. Scissors, grapplers, needle holders, micro dissectors, stable applicators, tackers, suction cleaning tools, snare wires, and clip applicators can be used as the end effector 2b.

[0035] Instrument 2 includes a first support member 2e and a second support member 2f. The first support member 2e is attached to the shaft 2d. The second support member 2f is rotatably supported by the first support member 2e around the A10 axis and supports the end effector 2b so as to be rotatable around the A11 axis which intersects the A10 axis. The shaft 2d rotates around the A9 axis. The wrist joint 2c is positioned between the second support member 2f and the first support member 2e with the A10 axis as its axis of rotation.

[0036] (Configuration of the arm operating section) As shown in Figure 6, the arm operating section 60 is attached to the robot arm 50 and operates the robot arm 50. Specifically, the arm operating section 60 is attached to the second link section 53.

[0037] (Remote Control Device) As shown in Figure 1, the remote control device 200 is located, for example, inside or outside the operating room. The remote control device 200 includes an operating unit 110, a foot pedal 120, a touch panel 130, a monitor 140, a support arm 150, a support bar 160, and an error reset button 161. The operating unit 110 constitutes an operating handle for an operator, such as a doctor, to input commands. The operating unit 110 receives commands for the robot arm 50.

[0038] As shown in Figure 11, the operating section 110 is a handle for operating the surgical instrument 1. The operating section 110 also receives input from the surgical instrument 1. The operating section 110 includes an operating section 110L located on the left side as viewed from the operator, such as a doctor, and operated by the operator's left hand, and an operating section 110R located on the right side and operated by the operator's right hand. The operating section 110 includes an arm section 111 and a wrist section 112. The operating section 110R includes an arm section 111R and a wrist section 112R. The operating section 110L includes an arm section 111L and a wrist section 112L.

[0039] The arm portion 111 has joints JT21, JT22, and JT23 as shown in Figure 11, and JT24, JT25, JT26, and JT27 as shown in Figures 12 and 13. The rotation axes of joints JT21, JT22, JT23, JT24, JT25, JT26, and JT27 are designated as axes A21, A22, A23, A24, A25, A26, and A27, respectively.

[0040] As shown in Figure 11, the arm portion 111R has a link portion 111a, a link portion 111b, and a link portion 111c. The upper end of link portion 111a is attached to the remote control device 200 so as to be rotatable around axis A21 in the vertical direction. The upper end of link portion 111b is attached to the lower end of link portion 111a so as to be rotatable around axis A22 in the horizontal direction. One end of link portion 111c is attached to the lower end of link portion 111b so as to be rotatable around axis A23 in the horizontal direction. A wrist portion 112 is attached to the other end of link portion 111c so as to be rotatable around axis A24. Link portion 111a is connected to the remote control device 200 by joint JT21. Link portion 111a and link portion 111b are connected by joint JT22. Link portion 111b and link portion 111c are connected by joint JT23. The arm portion 111 supports the wrist portion 112. The arm portion 111L has the same configuration as the arm portion 111R.

[0041] The wrist section 112 includes a wrist section 112R operated by the operator's right hand as shown in Figure 12, and a wrist section 112L operated by the operator's left hand as shown in Figure 13. Figure 12 shows the reference position of the operating section 110R, and Figure 13 shows the reference position of the operating section 110L. The configuration of the wrist section 112R and the wrist section 112L are the same.

[0042] The wrist section 112 includes a link section 112a, a link section 112b, a link section 112c, and a grip support member 112d operated by an operator such as a doctor. The base end of the link section 112a is connected to the tip of the arm section 111 and rotates around the A24 axis. The base end of the link section 112b is connected to the tip of the link section 112a and rotates around the A25 axis. The base end of the link section 112c is connected to the tip of the link section 112b, and the grip support member 112d is connected to the tip of the link section 112b and rotates around the A26 axis relative to the link section 112b. The grip support member 112d rotates around the A27 axis relative to the link section 112c. The link sections 112a, 112b, and 112c each have an L-shape.

[0043] The wrist unit 112 includes a pair of grip members 112e that are opened and closed by an operator. The grip members 112e are each formed of an elongated plate-shaped lever member, and the proximal ends of the pair of grip members 112e are rotatably connected to the proximal end of the grip support member 112d. A cylindrical finger insertion portion 112f is disposed on each grip member 112e. The operator inserts fingers into the pair of finger insertion portions 112f to operate the wrist unit 112. The base ends of the pair of grip members 112e are connected to the grip support member 112d, and by increasing or decreasing the angle between the pair of grip members 112e, the opening angle between the jaw member 2g and the jaw member 2h is changed. A magnet is disposed on one of the grip members 112e, and a Hall sensor is disposed on the grip support member 112d. When the operator opens and closes the grip members 112e, the magnet and the Hall sensor function as an angle detection sensor, and the Hall sensor outputs the opening angle. Note that the Hall sensor may be disposed on the grip member 112e, and the magnet may be disposed on the grip support member 112d.

[0044] As shown in FIG. 1, the monitor 140 is a scope-type display device for displaying an image captured by the endoscope 3. In addition, a notification unit 141 is disposed on the monitor 140. The notification unit 141 notifies an error sound. The support arm 150 supports the monitor 140 so that the height of the monitor 140 matches the height of the face of an operator such as a doctor. The touch panel 130 is disposed on the support bar 160. By detecting the head of the operator with a sensor provided in the vicinity of the monitor 140, the surgical assist robot 100 becomes operable by the remote control device 200. The operator operates the operation unit 110 and the foot pedal 120 while visually recognizing the surgical site via the monitor 140. Thereby, a command is input to the remote control device 200. Then, the robot arm 50 and the instrument 2 attached to the distal end of the arm are operated to move based on the operation command.

[0045] (Foot pedals) As shown in Figure 14, the foot pedals 120 are provided in multiple units to perform functions related to the surgical instrument 1. The multiple foot pedals 120 are arranged on the base unit 121. The foot pedals 120 include a switching pedal 122, a clutch pedal 123, a camera pedal 124, an incision pedal 125, a coagulation pedal 126, and a foot detection unit 127. The switching pedal 122, clutch pedal 123, camera pedal 124, incision pedal 125, and coagulation pedal 126 are operated by the operator's feet. The incision pedal 125 includes an incision pedal 125R for the right robot arm 50 and an incision pedal 125L for the left robot arm 50. The coagulation pedal 126 includes a coagulation pedal 126R for the right robot arm 50 and a coagulation pedal 126L for the left robot arm 50.

[0046] The switching pedal 122 switches the robot arm 50 to be operated by the control unit 110. The clutch pedal 123 performs a clutch operation that temporarily disconnects the control connection between the robot arm 50 and the control unit 110. While the clutch pedal 123 is pressed down by the operator, the operation of the control unit 110 is not transmitted to the robot arm 50. Also, while the camera pedal 124 is pressed down by the operator, the control unit 110 can operate the robot arm 50 to which the endoscope 3 is attached. While the incision pedal 125 or coagulation pedal 126 is pressed down by the operator, the electrosurgical device is activated.

[0047] (Vision Unit and Image Processing Unit) As shown in Figure 1, the vision unit 300 and the image processing unit 400 are mounted on the cart 210. The image processing unit 400 processes the images captured by the endoscope 3. A display unit 220 is located on the cart 210. The display unit 220 displays the images captured by the endoscope 3.

[0048] (Configuration of Control System) As shown in Figure 15, the surgical support system 500 includes a first control device 310, an arm control unit 320, a positioner control unit 330, an operation control unit 340, and a second control device 350. The surgical support system 500 further includes a storage unit 311 connected to the first control device 310, and a storage unit 351 connected to the second control device 350.

[0049] The first control device 310 is arranged inside the medical cart 10 so as to communicate with the arm control unit 320 and the positioner control unit 330, and controls the entire surgical support system 500. Specifically, the first control device 310 communicates with each of the arm control unit 320, the positioner control unit 330, and the operation control unit 340, and controls these control units. The first control device 310, the arm control unit 320, the positioner control unit 330, and the operation control unit 340 are connected via a LAN or the like. The first control device 310 is arranged inside the medical cart 10.

[0050] The arm control unit 320 is provided for each of the plurality of robot arms 50. That is, a plurality of arm control units 320 corresponding to the number of the plurality of robot arms 50 are arranged inside the medical cart 10.

[0051] As shown in Figure 15, the input device 22 is connected to the first control device 310 via a LAN or the like. The status indicator 41, the arm status indicator 42, the operation handle 23, the throttle 23a, the joystick 22b, and the positioner control unit 330 are connected by wiring 360 via serial communication through a communication network that allows mutual sharing of information. Although Figure 15 depicts that all of the status indicator 41, the arm status indicator 42 and the like are connected to a single wiring 360, in practice, the wiring 360 is provided for each of the status indicator 41, the arm status indicator 42, the operation handle 23, the throttle 23a, and the joystick 22b.

[0052] As shown in Figure 16, the arm section 51 is equipped with multiple servo motors SM1, encoders EN1, and a reduction gear, each corresponding to a joint JT1, JT2, JT3, JT4, JT5, JT6, and JT7. The encoder EN1 detects the rotation angle of the servo motor SM1. The reduction gear reduces the rotation of the servo motor SM1 to increase the torque. Inside the medical trolley 10, a servo control unit SC1 for controlling the servo motor SM1 is located adjacent to the arm control unit 320. The encoder EN1 for detecting the rotation angle of the servo motor SM1 is electrically connected to the servo control unit SC1.

[0053] The second link section 53 is equipped with a servo motor SM2, an encoder EN2, and a reduction gear for rotating a driven member located on the driven unit 2a of the surgical instrument 1. The encoder EN2 detects the rotation angle of the servo motor SM2. The reduction gear reduces the rotation of the servo motor SM2 to increase the torque. The medical trolley 10 is also equipped with a servo control unit SC2 for controlling the servo motor SM2 that drives the surgical instrument 1. The encoder EN2 for detecting the rotation angle of the servo motor SM2 is electrically connected to the servo control unit SC2. Note that multiple servo motors SM2, encoders EN2, and servo control units SC2 are provided.

[0054] The translational movement mechanism 54 is equipped with a servo motor SM3 for translating the surgical instrument 1, an encoder EN3, and a reduction gear. The encoder EN3 detects the rotation angle of the servo motor SM3. The reduction gear reduces the rotation of the servo motor SM3 to increase the torque. The medical trolley 10 also has a servo control unit SC3 for controlling the servo motor SM3 that translates the surgical instrument 1. The encoder EN3 for detecting the rotation angle of the servo motor SM3 is electrically connected to the servo control unit SC3. The encoders EN1, EN2, and EN3 are, for example, magnetic, mechanical, or electromagnetic induction type. The high resolution of the encoders EN1, EN2, and EN3 themselves improves the positional accuracy of the tip of the robot arm 50 and the tip of the surgical instrument 1. Furthermore, by reducing the rotational speed transmitted from the servo motors SM1, SM2, and SM3 using a reduction gear, the accuracy of the rotational angle of the joints of the robot arm 50, which is the output shaft, is further improved.

[0055] The first control device 310 generates command values ​​that command the positions (rotation angles of each servo motor) of servo motors SM1, SM2, and SM3 based on the operation received by the remote control device 200, and drives the servo motors SM1, SM2, and SM3 based on the command values. The first control device 310 then detects a deviation anomaly error if the difference between the command values ​​and the positions of servo motors SM1, SM2, and SM3 detected by the sensors exceeds an acceptable range.

[0056] As shown in Figure 17, the positioner 30 is equipped with multiple servo motors SM4, an encoder EN4, and a reduction gear, corresponding to the multiple joints 33 of the positioner 30. The encoder EN4 is configured to detect the rotation angle of the servo motors SM4. The reduction gear is configured to reduce the rotation of the servo motors SM4 and increase the torque.

[0057] As shown in Figure 17, the medical trolley 10 is equipped with a servo control unit SC4 for controlling the servo motor SM4 that moves the positioner 30. An encoder EN4 for detecting the rotation angle of the servo motor SM4 is electrically connected to the servo control unit SC4. The medical trolley 10 is also equipped with a servo control unit SC5 for controlling the servo motor SM5 that drives the front wheels of the medical trolley 10. An encoder EN5 for detecting the rotation angle of the servo motor SM5 is electrically connected to the servo control unit SC5. The medical trolley 10 is also equipped with a servo control unit SC6 for controlling the servo motor SM6 that power assists the steering of the rear wheels of the medical trolley 10. An encoder EN6 for detecting the rotation angle of the servo motor SM6 is electrically connected to the servo control unit SC6.

[0058] As shown in Figures 16 and 17, each of the joints JT1, JT2, JT3, JT4, JT5, JT6, and JT7 of the arm section 51 and the joint 33 of the positioner 30 are equipped with brakes. Brakes are also equipped on the front wheels of the medical trolley 10, the arm base 40, and the translational movement mechanism 54. Control signals are transmitted unidirectionally from the arm control unit 320 to each of the brakes mounted on the joints JT1, JT2, JT3, JT4, JT5, JT6, and JT7 of the arm section 51 and the translational movement mechanism 54. The control signals are signals to turn the brakes on and off. The signal to turn on the brakes includes a signal to maintain the state in which the brakes are engaged. The same applies to the control signals from the positioner control unit 330 to each of the brakes mounted on the joint 33 of the positioner 30 and the arm base 40. At startup, all brakes (BRK) on the arm base 40, arm section 51, and translational movement mechanism 54 are released, and the servo motor SM is driven to counteract gravity, thereby maintaining the posture of the robot arm 50 and the arm base 40. When an error occurs in the surgical support system 500, the brakes (BRK) on the arm base 40, arm section 51, and translational movement mechanism 54 are turned on. When the error in the surgical support system 500 is cleared, the brakes (BRK) on the arm base 40, arm section 51, and translational movement mechanism 54 are turned off. The brakes (BRK) on the arm base 40, arm section 51, and translational movement mechanism 54 are turned on when the surgical support system 500 is shut down. In addition, the front wheels of the medical trolley 10 always have the brakes (BRK) turned on, and the brakes (BRK) are released only while the enable switch 23b is pressed. Furthermore, each joint 33 of the positioner 30 has the brake BRK always on, and the brake BRK is released only while the enable switch 22c is pressed.

[0059] As shown in Figure 18, servo motors SM7a, SM7b, SM7c, SM7d, SM7e, SM7f, and SM7g are arranged on the joints JT21, JT22, JT23, JT24, JT25, JT26, and JT27 of the operating section 110, respectively. Servo motor SM7a rotates the link section 111a around the A21 axis. Servo motor SM7b rotates the link section 111b around the A22 axis. Servo motor SM7c rotates the link section 111c around the A23 axis. Servo motor SM7d rotates the link section 112a around the A24 axis. Servo motor SM7e rotates the link section 112b around the A25 axis. Servo motor SM7f rotates the link section 112c around the A26 axis. Furthermore, the servo motor SM7g rotates the grip support member 112d around the A27 axis. Servo control units SC7a, SC7b, SC7c, SC7d, SC7e, SC7f, and SC7g are also provided to control each servo motor. Each servo control unit is electrically connected to encoders EN7a, EN7b, EN7c, EN7d, EN7e, EN7f, and EN7g for detecting the rotation angle of each servo motor. Each servo motor, each servo control unit, and each encoder are provided in the operation unit 110L and the operation unit 110R, respectively. The servo motors SM7a, SM7b, SM7c, SM7d, SM7e, SM7f, and SM7g move the respective link sections of the operation unit 110.

[0060] The first control device 310, via the operation control unit 340, controls each servo motor to generate a torque that counteracts the gravitational torque generated on the rotation axis of each servo motor, according to the posture of the operation unit 110. This makes it possible for the operator to operate the operation unit 110 with relatively little force.

[0061] Furthermore, when the operator rotates the grip support member 112d around the A27 axis of the operating section 110 shown in Figures 12 and 13, the shaft 2d of the instrument 2 rotates around the A9 axis shown in Figure 5. Also, when the operator rotates the joints JT24, JT25, and JT26 of the operating section 110 shown in Figures 12 and 13, the end effector 2b bends around the A10 axis or A11 axis shown in Figure 5.

[0062] The operation control unit 340 is located in the main body of the remote control device 200. The operation control unit 340 controls the operation unit 110. As shown in Figure 15, the operation control unit 340 is positioned to correspond to the left-hand operation unit 110L and the right-hand operation unit 110R, respectively.

[0063] As shown in Figure 15, the vision unit 300 and the image processing unit 400 are connected to the first control device 310 via a LAN or the like. The display unit 220 is connected to the vision unit 300.

[0064] (Control operation of the first control device) The control of the first control device 310 when the operation unit 110 receives an operation from the operator will be described below. The following control is performed by a functional block of the first control device 310 that operates based on the program 311a stored in the storage unit 311.

[0065] As shown in Figure 1, an instrument 2 is attached to the robot arm 50d as a right-side forceps. An instrument 2 is attached to the robot arm 50b as a left-side forceps. As shown in Figure 19, the first control device 310 has a right-side forceps command generation unit 3100R that outputs a command value for controlling the robot arm 50d based on the operation of the operation unit 110R. The first control device 310 has a left-side forceps command generation unit 3100L that outputs a command value for controlling the robot arm 50b based on the operation of the operation unit 110L.

[0066] As shown in Figure 19, when the operator operates the operating unit 110R, the axis values ​​of the operating unit 110R are input to the right forceps command generation unit 3100R of the first control device 310 (block 3101R). When the operating unit 110R is operated, the joints JT21, JT22, JT23, JT24, JT25, and JT26 of the operating unit 110R rotate accordingly. The operation control unit 340 detects the rotation angle of each joint using encoders EN7a, EN7b, EN7c, EN7d, EN7e, EN7f, and EN7g provided at each joint. The detected rotation angles of each joint are input to the first control device 310 as the axis values ​​of the operating unit 110R. The right forceps command generation unit 3100R calculates the position and orientation of the operating unit 110R from the axis values ​​of the operating unit 110R (block 3102R). Furthermore, the right forceps command generation unit 3100R calculates the opening angle of the pair of grip members 112e of the right forceps (for example, 0° to 40°) based on the opening angle of the pair of grip members 112e of the operating unit 110R (block 3103R). Also, position and orientation refer to coordinate values ​​in the TCP coordinate system, and the opening angle of the pair of grip members 112e refers to the opening angle of jaw members 2g and 2h (for example, 0° to 40°). TCP is an abbreviation for Tool Center Point. TCP is set, for example, at the tip of instrument 2. The right forceps command generation unit 3100R also calculates the difference between the current position and orientation of the operating unit 110R and the previous position and orientation of the operating unit 110R, and calculates the amount of operation of the operating unit 110R based on the difference (block 3104R). Furthermore, the position and orientation of the endoscope 3 are input to the right forceps command generation unit 3100R (block 3121).

[0067] The right forceps command generation unit 3100R converts the operating amount of the control unit 110R into the coordinate system of the endoscope 3 based on the input position and orientation of the endoscope 3 (block 3105R). Then, the right forceps command generation unit 3100R calculates the operating amount of the instrument 2 based on the operating amount of the control unit 110R converted into the coordinate system of the endoscope 3 (block 3106R). The right forceps command generation unit 3100R also calculates the sum of the current operating amount of the instrument 2 and the previous operating amount of the instrument 2, and calculates the position and orientation of the instrument 2 based on the sum (block 3107R). Note that "this time" means the current control cycle, and "previous time" means the previous control cycle. The right forceps command generation unit 3100R also calculates the opening angles of the jaw members 2g and 2h based on the opening angle of the grip member 12e.

[0068] Furthermore, the right forceps command generation unit 3100R performs a process (block 3108R) to ensure that the axial velocities of the robot arm 50d do not exceed predetermined limits, based on the calculated position and orientation of the instrument 2. The right forceps command generation unit 3100R performs inverse kinematics calculations on the position and orientation of the instrument 2 to determine the axial values ​​of the robot arm 50d. If any axial velocity of the robot arm 50d exceeds a predetermined limit, the position and orientation of the instrument 2 are corrected so that all axial velocities are below the predetermined limit. For example, if the axial value with the largest deviation from the limit is, for example, twice the limit, the change in the position and orientation of the instrument 2 from the previous cycle is halved, and then the final inverse kinematics calculation (block 3109R) is performed. The functional block of the first control device 310 that limits the axial velocities is designated as the limiting unit.

[0069] The right forceps command generation unit 3100R performs inverse kinematics calculations for the position and orientation of the instrument 2 (block 3109R). The right forceps command generation unit 3100R calculates command values ​​for each axis of the robot arm 50d based on the inverse kinematics calculations (block 3110R). The command values ​​calculated here do not exceed predetermined limit values ​​for the command values ​​of the robot arm 50d. The right forceps command generation unit 3100R performs a process to apply acceleration limits to the calculated command values ​​for each axis (block 3111R). Note that acceleration includes acceleration in the direction of increasing velocity and acceleration in the direction of decreasing velocity. The maximum value of the acceleration for each axis is set in advance, and if the acceleration of the robot arm 50 exceeds the maximum value, the first control device 310 limits the acceleration so that the acceleration becomes the maximum value. Then, command values ​​that cause the servo motor to rotate while the velocity and acceleration are limited are output to the servo control unit SC1 that drives the robot arm 50 and the servo control unit SC2 of the arm control unit 320 that drives the instrument 2. Furthermore, the signals indicating the position, orientation, and opening angles of the jaw members 2g and 2h included in each of the instrument 2, generated based on the operator's operation of the control unit 110 of the remote control device 200, will hereafter be referred to as manual operation commands. The signals indicating the position, orientation, and opening angles of the jaw members 2g and 2h included in each of the instrument 2, inferred by the inference unit 614 of the inference control device 600 (described later), will hereafter be referred to as inference operation commands.

[0070] Furthermore, as shown in Figure 19, the first control device 310 has a left forceps command generation unit 3100L that outputs command values ​​for controlling the robot arm 50b based on the operation of the operation unit 110L. For input from the operation unit 110L for the left forceps (block 3101L), the left forceps command generation unit 3100L performs the same processing as the right forceps command generation unit 3100R. That is, by switching between manual operation commands and inference operation commands using the right forceps command generation unit 3100R and the left forceps command generation unit 3100L, the remote control mode and autonomous mode of the robot arm 50 corresponding to the right forceps command generation unit 3100R and the left forceps command generation unit 3100L can be switched individually. Also, the remote control mode and autonomous mode can be switched simultaneously for both the right forceps command generation unit 3100R and the left forceps command generation unit 3100L.

[0071] (Inference Control Device) As shown in Figure 1, the inference control device 600 is located separately from the surgical support robot 100, the remote control device 200, the vision unit 300, and the image processing unit 400. The inference control device 600 is, for example, a PC (personal computer). As shown in Figure 15, the inference control device 600 is connected to the first control device 310 by a LAN or the like. As shown in Figure 20, the inference control device 600 includes a processing unit 610 such as a CPU (Central Processing Unit). The processing unit 610 may be composed of a GPU (Graphics Processing Unit). The inference control device 600 also includes a storage unit 620. The memory unit 620 is, for example, ROM (Read Only Memory), RAM (Random Access Memory), HDD (hard disk drive), etc. The inference control device 600 also includes an input unit 630. The input unit 630 is, for example, a keyboard. Here, the inference control device 600 may be located inside a surgical support robot 100, a remote control device 200, a vision unit 300, or an image processing unit 400 as a device having a processing unit 610 and a memory unit 620. The inference control device 600 may also be connected to a surgical support system 500 via an external network such as the cloud. Figure 21 shows the architecture of the AI ​​model used in the inference control device 600 of this embodiment. The AI ​​model includes an Image Encoder that converts input images into features, a Signal Encoder that converts input sensor information, which is time-series data such as action commands, into features, and an Action Decoder that outputs inference action commands from the features obtained by the Image Encoder and Signal Encoder. If the endoscope 3 is 3D compatible, the Image Encoder receives the left viewpoint image (Image 1) and the right viewpoint image (Image 2). The Image Encoder may receive only one image as input.Image Encoders and SignalEncoders are constructed using Convolutional Neural Networks (CNNs) and Transformers. Action Decoders, on the other hand, utilize autoregressive models using Transformers, as well as generative models such as Denoising Diffusion and Flow Matching.

[0072] In this embodiment, the processing unit 610 includes a data collector 611, a learner 612, an evaluator 613, and an inference unit 614. The data collector 611, learner 612, evaluator 613, and inference unit 614 are functional blocks executed by the CPU and GPU accelerator of the processing unit 610 based on a program 621 stored in the storage unit 620. The data collector 611 is an example of a state acquisition unit. The learner 612 is an example of a learning model generation unit. The evaluator 613 is an example of an evaluation acquisition unit. The inference unit 614 is an example of an operation command output unit.

[0073] As shown in Figure 22, the data collector 611 receives images G captured by the endoscope 3 attached to the robot arm 50c as endoscopic data from the first control device 310. As shown in Figure 23, images G show the surgical site of the patient P, the tip of the instrument 2, the suture needle, the suture thread, etc. The data collector 611 collects and stores the images G captured by the endoscope 3 in chronological order. The data collector 611 also receives the position and orientation of the endoscope 3 as endoscopic data from the first control device 310.

[0074] Furthermore, the data collector 611 collects and stores the manual operation commands and inferred operation commands processed by the first control device 310 in chronological order. As described above, the right forceps command generation unit 3100R of the first control device 310 shown in Figure 19 outputs a manual operation command, which is a command value for controlling the robot arm 50d based on the operation of the operation unit 110R. The right forceps command generation unit 3100R also outputs an inferred operation command, which is an inferred operation command output by the inferred operation unit 614, which will be described later and shown in Figure 19. The data collector 611 collects and stores the manual operation commands and inferred operation commands as operation commands output from the first control device 310 in chronological order. The operation commands include the respective positions and orientations of the left and right forceps instruments 2, and the opening angles of the respective jaw members 2g and jaw members 2h of the left and right forceps instruments 2.

[0075] Furthermore, if the sampling periods of the image G captured by the endoscope 3 and the operation command, which is either a manual operation command or an inferred operation command, are different, the data collector 611 adjusts the time difference between the image G captured by the endoscope 3 and the operation command. The data collected by the data collector 611 is stored, for example, in the data collector 611 itself or in the storage unit 620.

[0076] In this embodiment, the learner 612 generates an inference unit 614, which is a trained model obtained by machine learning, using observational information of the environment surrounding multiple robot arms 50, each operated independently based on motion commands, and motion commands as input. Specifically, the learner 612 generates an inference unit 614, which is a trained model obtained by machine learning, using images G captured by the endoscope 3 collected by the data collector 611, motion commands, and the current position and orientation as training data. The machine learning is, for example, deep learning. In addition to the images G input to the learner 612, information about the affected area, jaw members 2g and 2h, suture needle 641, and suture thread 642 extracted from the images G may also be input.

[0077] In this embodiment, the manual operation commands learned by the learner 612 include suturing operation commands for suturing the surgical site of patient P. Specifically, the suturing operation commands are manual operation commands generated based on the operator's operation of the control unit 110 to suture the surgical site of patient P. The image G learned by the learner 612 is an image G captured by the endoscope 3 when suturing is performed on the surgical site of patient P. As shown in Figure 24, this image G shows the surgical site of patient P, the jaw members 2g and 2h at the tip of the instrument 2, the suture needle 641 held by the jaw members 2g and 2h, and the thread 642 threaded through the suture needle 641. Preprocessing may be performed on the image G to extract information such as the movement of the jaw members 2g and 2h at the tip of the instrument 2, the movement of the suture needle, the movement of the suture thread, and the condition of the organ as the affected area, and this information may be input to the learner 612 and the inference unit 614 along with the image G. Furthermore, the sampling rate may be adjusted as a preprocessing step. For example, instead of inputting all images G into the learner 612, images G in batches of several may be extracted from all images G and input into the learner 612. Also, as a preprocessing step, the coordinate system of the images G may be converted to a coordinate system suitable for inference by the inference unit 614. The learner 612 then learns the suturing motion of the surgical site of patient P and generates an inference unit 614 for inferring the suturing motion of the surgical site of patient P.

[0078] Furthermore, the motion commands learned by the learner 612 include the position and orientation of each of the left and right forceps instruments 2, the opening angles of the jaw members 2g and 2h, and the position and orientation of the endoscope 3. In addition, as a preprocessing step, the coordinate system of the motion commands may be converted to a coordinate system suitable for inference by the inference unit 614.

[0079] Furthermore, the current position and orientation learned by the learning unit 612 includes the position and orientation of each of the left and right forceps instruments 2, the opening angles of the jaw members 2g and 2h, and the position and orientation of the endoscope 3. The current position and orientation refers to the position and orientation of the forceps instruments 2. In addition, as a preprocessing step, the coordinate system of the current position and orientation may be converted to a coordinate system suitable for inference by the inference unit 614.

[0080] In this embodiment, the data collector 611 collects and stores images G captured by the endoscope 3, operation commands, and the current position and orientation as learning data for generating the inference unit 614. Instructions to start and stop the data collection and storage operation are input, for example, by the coagulation pedal 126R. The coagulation pedal 126R is located separately from the coagulation pedal 126L, which switches between autonomous mode and remote control mode, as described later. The coagulation pedal 126R is pressed by the operator's foot, which switches the start and stop of the data collection and storage operation of the data collector 611. Pressing by the foot means stepping on it with the foot.

[0081] Furthermore, when the coagulation pedal 126R is pressed, an alert sound is emitted. For example, the alert sound is emitted from the notification unit 141 of the monitor 140 shown in Figure 1 or from the speaker 22e of the input device 22.

[0082] In this embodiment, the evaluator 613 acquires an operational evaluation of the robot arm 50 and instrument 2 that operate in response to the input information. Specifically, the evaluator 613 evaluates the operational state of input information that includes at least one of the following: information on the operation state of the remote control device 200, information on the operational state of the robot arm 50 and instrument 2, and information on the operational state of the device corresponding to instrument 2 when operated manually, and acquires an operational evaluation.

[0083] In this embodiment, the evaluator 613 acquires an operational evaluation based on at least one of the information acquired by the data collector 611 and the evaluator's input operations. For example, time-series images G and operational commands from past surgeries are input from the data collector 611 to the evaluator 613. Information about the surgeon (user name, surgical procedure, etc.) related to the images G and operational commands is also input to the evaluator 613. This data is stored in an external storage device such as a server and is input to the learner 612 as part of the learning data via the data collector 611. The surgeon information input to the data collector 611 can also be directly input by the surgeon instead of from an external storage device. The operational evaluation obtained by the evaluator 613 is input to the learner 612. The operational evaluation can also be called the performance level. For example, the operational evaluation may include who the operator is, the execution time of the suturing operation on the patient P's surgical site, the score of the suturing performance, and the surgical procedure. The performance evaluation of a suturing score is assessed based on factors such as the amount of bleeding at the suture site, the spacing of the sutures, and the degree of deformation at the suture site. However, actions in the same situation may differ depending on the operator's skill level and objective. This uncertainty makes it difficult for deep learning models to train. Furthermore, it is impossible to predict which action will be selected during inference, potentially leading to undesirable choices. Additionally, variations in the characteristics of actions each time result in inconsistent performance. By incorporating these conditions and characteristics into the input for training, diverse data can be learned consistently, leading to more efficient training of the deep learning model and consistent selection of the desired action during inference. By evaluating the performance of a suturing score, the model can select better actions during inference, resulting in better outcomes. Moreover, because it learns from poor performance data, it can recover from minor mistakes during inference. In other words, if only good performance data is used, mistakes such as fumbling the knot or dropping the needle are not included. In this case, learning recovery becomes difficult.

[0084] By obtaining information about the operator and an evaluation of the surgical procedure's actions, the system can select the desired surgical procedure and operator to emulate, thereby obtaining the inference commands the operator performing the surgery would prefer. In other words, by evaluating operator information, the system can select actions that match the user's preferences and characteristics during inference, resulting in user-friendly autonomous assistance. In this case, the user can choose the mode themselves, or the AI ​​can automatically select the closest mode based on their past actions. Furthermore, by adding one's own data to an existing AI during training, efficient additional learning can be performed with a small amount of data, allowing the system to train a unique partner AI. For example, a partner AI is a local AI used only by the user. Also, by evaluating surgical procedure information, the system can perform tasks more appropriately for that procedure by selecting it during inference. In other words, it can use information specifically linked to the task, such as the work content or suturing location, rather than just the surgical procedure itself. For example, in a scene with multiple wounds, using the "suturing skill" might not tell the system which area to suture, but if the location is specified by coordinates on the image or linguistic information, the system can perform consistent actions.

[0085] The learner 612 generates an inference 614 based on input information including at least one of the following: information on the operation status of the remote control device 200, information on the operating status of the robot arm 50 and instrument 2, and information on the operating status of the device corresponding to instrument 2 when operated manually, the environment surrounding the operated robot arm 50 and instrument 2, and the operation evaluation acquired by the evaluator 613.

[0086] Furthermore, the evaluator 613 acquires a motion evaluation based on at least one of the following: processing speed, processing accuracy, and motion trajectory, resulting from the movement of the robot arm 50 and instrument 2. The evaluator 613 also acquires operator information, which is information about the operator who controlled the movement of the robot arm 50 and instrument 2, as part of the motion evaluation. For example, operator information includes the ID information of the operator (a physician), the number of processing sessions, years of experience, and specialty. The processing refers to the processing performed by the movement of the robot arm 50 and instrument 2. For example, the processing is the process of suturing the surgical site of a patient. The processing may also be part or all of the patient's surgical procedure.

[0087] The evaluator 613 gives a higher performance evaluation the faster the processing speed, and a lower performance evaluation the slower the processing speed. Furthermore, the evaluator 613 gives a higher performance evaluation the higher the processing accuracy, and a lower performance evaluation the lower the processing accuracy. Processing accuracy may be quantified and evaluated based on the accuracy of the processing position. Also, the evaluator 613 gives a higher performance evaluation the shorter the movement trajectory, and a lower performance evaluation the longer the movement trajectory. In other words, the evaluator 613 gives a higher performance evaluation the shorter the movement trajectory, achieved by moving efficiently for processing.

[0088] Furthermore, the evaluator 613 acquires a motion evaluation based on the images captured by the endoscope 3. In other words, the evaluator 613 uses images captured by the endoscope 3 to evaluate the movements of the robot arm 50 and instrument 2, and acquires a motion evaluation.

[0089] In this embodiment, the inference unit 614 takes at least one of the observation information and operation commands of the environment surrounding the robot arm 50 and instrument 2 as input, based on the input information, the environment surrounding the operated manipulation device, and the operation evaluation, and outputs an inference operation command. As shown in Figure 19, for example, an image G (block 3115) captured by the endoscope 3 is input to the inference control device 600. As described above, preprocessing may be performed on the image G to extract information such as the movement of the jaw members 2g and 2h at the tip of the instrument 2, the movement of the suture needle, the movement of the suture thread, and the condition of the organ as the affected area, and input this information to the learner 612 and the inference unit 614 along with the image G.

[0090] Furthermore, the inference control device 600 receives input of the position and orientation of the tip of the endoscope 3. In addition, the inference control device 600 receives input from the first control device 310 as operation commands, including the position and orientation of the left and right instruments 2, and the opening angles of the jaw members 2g and 2h included in each of the left and right instruments 2 (3116). The left and right instruments 2 refer to the two robot arms 50 selected by the operator as the target of operation from among the robot arms 50a, 50b, and 50d to which the instruments 2 are attached. The operator can switch the robot arm 50 to be operated by the operation unit 110 by operating the switching pedal 122. In addition, the inference control device 600 receives input of whether it is in autonomous mode or remote control mode. This information may also be stored in the memory unit 620, memory unit 311, memory unit 351, or on the cloud. The information indicating whether the robot is in autonomous mode or remote control mode refers to whether the coagulation pedal 126L is being pressed by the operator, as will be described later. The inference control device 600 also receives information on the start or stop of the collection and storage operation, which collects and stores images G captured by the endoscope 3, operation commands, and the current position and orientation as learning data for generating the inference unit 614. This information may also be stored in the storage unit 620, storage unit 311, storage unit 351, or on the cloud. This information may also be stored for each robot arm 50.

[0091] Furthermore, as shown in Figure 22, the inference operation command output by the inference unit 614 includes the position and orientation of the left and right instruments 2, and the opening angles of the jaw members 2g and 2h included in each of the left and right instruments 2. In addition, the inference operation command output by the inference unit 614 is an operation command for a control cycle further in the future than the current time.

[0092] In this embodiment, the inference unit 614 takes as input observation information and operation commands from multiple past points in time stored in the data collector 611 or storage unit 620, and outputs inference operation commands for the next and subsequent times. The next time may be immediately after the current time, or a little while after the current time. The next inference operation command may be an inference operation command for one step in the next time, or an inference operation command for several steps after the next time. For example, the inference unit 614 of the inference control device 600 receives as observation information images G captured by the endoscope 3 as endoscopic data for several steps of the control cycle of the surgical support robot 100. The inference unit 614 also receives as input manual operation commands, inference operation commands, or operation commands that are a mixture of manual operation commands and inference operation commands for several steps of the control cycle of the surgical support robot 100. Furthermore, the current position and orientation are input to the inference control device 600. The images G, manual operation commands, current position and orientation, etc., are input to the inference control device 600 via the input port 640. The inference unit 614 outputs an inference operation command based on this input information. The robot arm 50 and instrument 2 then operate based on the inference operation command.

[0093] Furthermore, in this embodiment, the inference unit 614 outputs an inference operation command corresponding to the operator's operation based on the input operator information. In other words, the inference unit 614 outputs an inference operation command so that the operation of instrument 2 is similar to the operation of the specified operator. This makes it possible to automatically reproduce the operation of the specified operator.

[0094] Furthermore, in this embodiment, the inference unit 614 outputs an inference operation command so that the operation evaluation result is relatively high. In other words, the inference unit 614 outputs an inference operation command so that the operation evaluation value is equal to or greater than a predetermined value. The inference unit 614 also outputs an inference operation command so that the instrument 2 operates at a high quality. The operation evaluation can be quantified based on one or more operation evaluation items. When the inference unit 614 outputs an inference operation command so that the operation evaluation result is relatively high, it generates an inference operation command so that the numerical value of the operation evaluation is equal to or greater than a predetermined value. Furthermore, when the inference unit 614 quantifies the operation evaluation, it may perform the evaluation by assigning different weights to each of the above-mentioned operation evaluation items.

[0095] In this embodiment, the inference unit 614 receives an operation command and observation information acquired by the endoscope 3 as input and outputs an inference operation command.

[0096] At least one of the sets of multiple robot arms 50 and instruments 2 can be switched between an autonomous mode, which operates based on inference operation commands output from the inference unit 614, and a remote operation mode, which operates based on manual operation commands, which are operation commands from the remote control device 200. In this embodiment, all sets of robot arms 50a, 50b, and 50d, and the instruments 2 attached to them, can be switched between autonomous mode and remote operation mode. Furthermore, the robot arm 50c to which the endoscope 3 is attached may also be switched between autonomous mode and remote operation mode.

[0097] In autonomous mode, the operation command and the image G acquired by the endoscope 3 are input to the inference unit 614, and the inference unit 614 outputs an inference operation command, which the robot arm 50 and instrument 2 operate based on. The inference operation command includes, for example, the position and orientation of the left and right instrument 2, and the opening angles of the jaw members 2g and 2h included in each of the left and right instrument 2. Here, as shown in Figure 19, the first control device 310 performs a process to limit the axial speed of the robot arm 50 and instrument 2 as described above. The inference operation command is replaced with the manual operation command before the process of applying axial speed and before the calculation of the position and orientation of the instrument 2. That is, in autonomous mode, the manual operation command is replaced with the inference operation command before the manual operation command is limited by the first control device 310.

[0098] The autonomous mode and remote control mode can be switched, for example, by the coagulation pedal 126L. The coagulation pedal 126L is operated by the operator's foot. The autonomous mode is performed while the operator is pressing down on the coagulation pedal 126L with their foot. The remote control mode is performed while the operator is releasing the pressure on the coagulation pedal 126L. Pressing down with the foot means that the operator is pressing down on the coagulation pedal 126L. Releasing the pressure means that the operator has stopped pressing down on the coagulation pedal 126L.

[0099] Even in remote control mode, the inference unit 614 continues to infer inference operation commands using manual operation commands in remote control mode as input. The autonomous mode is started when the operator presses down on the solidification pedal 126L. In autonomous mode, the robot arm 50 and instrument 2 operate based on the inference operation commands output from the inference unit 614. The remote control mode is started when the operator releases the pressure on the solidification pedal 126L. In remote control mode, manual operation commands are generated when the operator operates the control unit 110, and the robot arm 50 and instrument 2 operate based on the manual operation commands. Here, even after switching from autonomous mode to remote control mode, the inference unit 614 continues to infer inference operation commands. Specifically, at the initial point after switching from autonomous mode to remote control mode, the inference operation commands and manual operation commands are included in several steps of the past control cycle, so the inference unit 614 infers inference operation commands using past observation information, past inference operation commands, and past manual operation commands as input. After some time has passed since switching from autonomous mode to remote control mode, the past few steps of the control cycle will only contain past manual operation commands. Therefore, the inference unit 614 uses past observation information and past manual operation commands as input to infer an inference operation command.

[0100] Furthermore, in this embodiment, as shown in Figure 24, the monitor 140 of the remote control device 200 and the display unit 220 located on the cart 210 display the current operating position of the instrument 2 and the operating position of the instrument 2 based on the inferred operation commands from the present moment onward. For example, the TCP coordinate system CS is displayed on the monitor 140 and the display unit 220. In Figure 24, the coordinate system CS of the instrument 2 attached to the robot arm 50 operated by the left operation unit 110 and the coordinate system CS of the instrument 2 attached to the robot arm 50 operated by the right operation unit 110 are displayed, respectively. The coordinate system CS is the coordinate system that serves as the operating reference for the robot arm 50 and the instrument 2, and is, for example, the coordinate system of the endoscope 3. In addition, the X, Y, and Z axes of the coordinate system CS are displayed in different colors. Furthermore, the monitor 140 and the display unit 220 display the coordinate system CS of the current TCP and the coordinate system CS of the TCP from the present moment onward. In Figure 24, the coordinate system CS of the current TCP is displayed as a solid line, and the coordinate system CS of the TCP from the present moment onward is displayed as a dotted line. The coordinate system CS of the TCP from the present moment onward is the coordinate system CS of the TCP at a predetermined time ahead of the current TCP coordinate system CS. The coordinate system CS of the TCP from the present moment onward is, for example, the coordinate system CS of the TCP one second later inferred by the inference unit 614. Note that the time ahead of the current TCP coordinate system CS for the coordinate system CS of the TCP from the present moment onward can be changed by setting. Furthermore, the monitor 140 and the display unit 220 display the current opening angle OA of the jaw members 2g and 2h of the instrument 2, and the opening angle OA of the jaw members 2g and 2h of the instrument 2 from the present moment onward. In Figure 24, the current opening angle OA is shown by a solid line, and the opening angle OA from the present moment onward is shown by a dotted line. The opening angle OA from the present moment onward is, for example, the opening angle OA one second later inferred by the inference unit 614. The opening angle OA is also displayed, for example, by a diagram that simulates a pair of jaw members.

[0101] The monitor 140 of the remote control device 200 and the display unit 220 located on the cart 210 superimpose the current operating position of the instrument 2 and the operating position of the instrument 2 based on the inferred operation command onto the image G captured by the endoscope 3. In Figure 24, the image G of the suture position of patient P, captured by the endoscope 3, is displayed on the monitor 140 and the display unit 220. The coordinate system CS and opening angle OA, representing the operating position of the instrument 2, are superimposed on the image G of the suture position of patient P.

[0102] The monitor 140 of the remote control device 200 and the display unit 220 located on the cart 210 display an indicator M that indicates whether the current mode is autonomous mode or remote control mode. For example, as shown in Figure 24, the monitor 140 and the display unit 220 display the characters Mode: Dr as indicator M when the current mode is remote control mode. Also, the monitor 140 and the display unit 220 display the characters Mode: AI as indicator M when the current mode is autonomous mode.

[0103] (Operation of the control unit in autonomous mode) In this embodiment, as described above, the control unit 110 includes link parts 111a, 111b, 111c, 112a, 112b, 112c and a grip support member 112d. In the following description, the grip support member 112d is also referred to as a link part. The control unit 110 also includes servo motors SM7a to SM7g that drive the above-mentioned plurality of link parts. By changing the posture of the plurality of link parts, the posture of the robot arm 50 and instrument 2 is changed. In autonomous mode, the posture of the plurality of link parts is changed by servo motors SM7a to SM7g to follow the posture of the robot arm 50 and instrument 2. That is, the plurality of link parts of the control unit 110 rotate automatically and the posture of the plurality of link parts is changed to correspond to the posture of the robot arm 50 and instrument 2 in autonomous mode. Alternatively, the operating unit 110 may change to a corresponding position so that only the wrist axis of the operating unit 110 follows the posture of the robot arm 50 and the instrument 2.

[0104] Next, the control method for the surgical support system 500 will be described. In the following control method for the surgical support system 500, the operations performed by the inference control device 600 are performed based on the program 621 stored in the memory unit 620, and the operations performed by the first control device 310 are performed based on the program 311a stored in the memory unit 311.

[0105] (Trained Model Generation Process) In the trained model generation process shown in Figure 26, the inference control device 600 generates a trained model for outputting inference operation commands. As shown in Figure 25, in step S1, the trained model generation process involves the data collector 611 acquiring input information that includes at least one of the following: information on the operation status of the remote control device 200, information on the operation status of the robot arm 50 and instrument 2, and information on the operation status of the device corresponding to instrument 2 when operated manually.

[0106] In step S2, the evaluator 613 acquires an evaluation of the operation of the robot arm 50 and instrument 2 in response to the input information. For example, the evaluator 613 acquires an evaluation of the operation based on the image captured by the endoscope 3 when the instrument 2 is in operation. Alternatively, for example, the evaluator 613 acquires an evaluation of the operation after receiving the operator's input.

[0107] In step S3, the learning device 612 acquires the environment surrounding the operated robot arm 50 and instrument 2. Specifically, the learning device 612 acquires the environment surrounding instrument 2 as imaged by the endoscope 3 when instrument 2 is operating.

[0108] In step S4, the learner 612 uses multiple sets of input information, operation evaluations, and the surrounding environment of instrument 2 to learn and generate a trained model for outputting inference operation commands.

[0109] (Output Processing of Inference Operation Commands) In the output processing of inference operation commands shown in Figure 26, the inference unit 614 outputs an inference operation command to operate instrument 2 using the trained model generated by the trained model generation process. In the output processing of inference operation commands, as shown in Figure 26, in step S11, the inference unit 614 accepts input for operation evaluation. Specifically, the inference unit 614 accepts input of information about the operator who performed the operation to be performed, for example. The inference unit 614 also accepts input of the level of operation evaluation, for example.

[0110] In step S12, the inference unit 614 acquires the environment surrounding the robot arm 50 and the instrument 2. Specifically, the inference unit 614 acquires the current environment surrounding the instrument 2 as imaged by the endoscope 3.

[0111] In step S13, the inference unit 614 generates an inference operation command using a trained model based on the received operation evaluation and the environment surrounding the instrument 2.

[0112] In step S14, the inference unit 614 outputs an inference operation command. Specifically, the inference unit 614 outputs the generated inference operation command to the first control device 310. The first control device 310 then performs an inverse conversion based on the inference operation command and outputs command values ​​to each axis of the robot arm 50. As a result, the robot arm 50 and the instrument 2 operate based on the inference operation command.

[0113] (Autonomous Mode Processing) In the autonomous mode processing shown in Figure 27, the inference unit 614 outputs an inference operation command to autonomously operate the instrument 2 based on the operation command. In the autonomous mode processing, as shown in Figure 27, in step S41, the operation command for a control cycle of approximately 1 second and the image G captured by the endoscope 3 are collected by the data collector 611. The image G from the endoscope 3 is based on the Full HD standard, and the resolution is 1920 x 1080 pixels. Next, in step S42, the inference unit 614 infers an inference operation command based on the collected operation command and image G. At the initial point when switching from remote control mode to autonomous mode, the inference operation command and manual operation command are included in several steps of the past control cycle, so the inference unit 614 infers an inference operation command using past observation information, past inference operation commands and past manual operation commands as input. After some time has passed since switching from remote control mode to autonomous mode, the past few steps of the control cycle contain only past inferred operation commands. Therefore, the inferential unit 614 uses past observation information and past inferred operation commands as input to infer an inferred operation command. The image G of the endoscope 3 used by the inferential unit 614 for inferring the inferred operation command has been reduced in size to a resolution of 256 x 128 pixels. The inferred operation command that the inferential unit 614 infers is the inferred operation command for approximately one second after the operation command of the control cycle of approximately one second collected by the data collector 611. The time required for the inferential unit 614 to infer the inferred operation command is approximately 0.1 seconds, which is shorter than the period of the inferred operation command that the inferential unit 614 infers. In step S42, the data collector 611 collects operation commands and the image G captured by the endoscope 3 in parallel with the inferential unit 614 performing the inference. In step S43, the inference control device 600 outputs the inferred inference operation command to the first control device 310. As a result, the robot arm 50 and instrument 2 operate autonomously, and the posture of the operating unit 110 is also changed to follow the changes in the posture of the robot arm 50 and instrument 2.

[0114] [Effects of this embodiment] In the surgical support system 500, the learner 612 takes at least one of the observation information of the environment surrounding the instrument 2 and the operation command as input, and generates a trained model that outputs an inferred operation command, based on the input information, the environment surrounding the operated instrument 2, and the operation evaluation. This allows learning to be performed while taking operation evaluation into account for various operations, including high-quality and low-quality operations, so that a trained model capable of performing higher-quality operations can be generated. In addition, since more operation data can be collected compared to not using data from failed operations, operation data can be collected easily. This makes it easier to train the trained model. As a result, the performance of the operation of the instrument 2 by the trained model is improved, and the trained model can be easily created.

[0115] The surgical support system 500 includes an inference unit 614 that uses a trained model generated by the learner 612 based on input information, the environment surrounding the operated instrument 2, and performance evaluation. The inference unit 614 takes at least one of the observation information of the environment surrounding the instrument 2 and a performance command as input and outputs an inferred performance command. This allows the instrument 2 to be operated automatically based on at least one of the observation information of the environment surrounding the instrument 2 and a performance command.

[0116] The evaluator 613 obtains an operational evaluation based on at least one of the information acquired by the data collector 611 and the evaluator's input operations. This makes it possible to easily and accurately obtain operational evaluations that will be used when generating a trained model.

[0117] The evaluator 613 acquires an operational evaluation based on at least one of the following: processing speed, processing accuracy, and operation trajectory of the instrument 2. This allows for a quantitative evaluation of the operational evaluation based on at least one of the following: processing speed, processing accuracy, and operation trajectory.

[0118] An endoscope 3 is provided to image the environment surrounding instrument 2. The evaluator 613 acquires a performance evaluation based on the images captured by the endoscope 3. This makes it easier to acquire a performance evaluation because the performance evaluation can be performed based on images of the instrument in actual operation.

[0119] The operation evaluation includes operator information, which is information about the operator operating the remote control device 200. Based on the input operator information, the inference unit 614 uses a trained model to output an inference operation command corresponding to the input operator's operation. This makes it possible to make the instrument 2 perform an operation similar to the operation performed by a specific operator.

[0120] The inference unit 614 outputs an inference operation command to perform an operation such that the operation evaluation value is equal to or greater than a predetermined value. This makes it easy to automatically perform high-quality operation of the instrument 2.

[0121] The surgical support system 500 includes an endoscope 3 that observes the environment surrounding the instrument 2 and outputs observational information. The inference unit 614 receives the operation command and the observational information acquired by the endoscope 3 as input to a trained model and outputs an inference operation command, and the instrument 2 operates based on the inference operation command. This makes it easy to automatically perform operations corresponding to the observational information of the environment surrounding the instrument 2.

[0122] The surgical support system 500 includes a monitor 140 and a display unit 220 that display the operating position of instrument 2 based on inferred motion commands. This allows the operator to visually confirm the operating position of instrument 2 based on inferred motion commands from the current point in time onward, enabling even inexperienced operators to perform suturing appropriately by operating in accordance with the operating position of instrument 2 from the current point in time onward. Furthermore, by displaying the operating positions of the robot arm 50 and instrument 2 based on inferred motion commands from the current point in time onward using a model of the surgical site or the like, rather than during actual surgery, inexperienced operators can be trained.

[0123] The surgical support system 500 includes a surgical support robot 100, which includes a plurality of robot arms 50 to which surgical instruments 1 are attached at their tips, and a remote control device 200, which includes an operating unit 110 that receives operations on the surgical instruments 1. This improves the performance of the robot arm 50's movements using a trained model in the surgical support system 500, and also makes it easy to create trained models.

[0124] The multiple robot arms 50 include a robot arm 50c to which an endoscope 3 is attached at its tip, and robot arms 50a, 50b, and 50d to which instruments 2, which are surgical instruments 1 other than an endoscope, are attached at their tips. The endoscope 3 captures observational information of the environment surrounding the instruments 2 attached to the robot arms 50a, 50b, and 50d. As a result, observational information can be acquired by the endoscope 3 which is pre-placed on the surgical support robot 100, eliminating the need to place a separate device for acquiring observational information. Consequently, the complexity of the configuration of the surgical support system 500 can be suppressed.

[0125] The operation commands include suturing operation commands for suturing the surgical site of patient P. This improves the performance of instrument 2's operation using a trained model for suturing the surgical site of patient P, and also makes it easier to create the trained model.

[0126] [Modifications] It should be understood that the embodiments disclosed herein are illustrative and not restrictive in all respects. The scope of this disclosure is indicated by the claims rather than the description of the embodiments above, and further includes all modifications or variations within the meaning and scope equivalent to the claims.

[0127] In the above embodiment, an example was shown in which both the left and right robot arms 50 and the instruments 2 attached thereto operate autonomously in autonomous mode, but the disclosure is not limited thereto. For example, in autonomous mode, one of the left and right robot arms 50 and the instrument 2 attached thereto may operate autonomously. In this case, a switching unit for switching between autonomous mode and remote control mode is provided for each of the left and right robot arms 50. The operator only needs to operate one of the robot arms 50 and the instrument 2 attached thereto, thus reducing the burden on the operator.

[0128] In the above embodiment, an example was shown in which a trained model is generated by learning using information on the operation state of the remote control device 200 when the robot arm 50 and instrument 2 are operated by the remote control device 200, and information on the operation state of the robot arm 50 and instrument 2. However, this disclosure is not limited to this. For example, a trained model may be generated by learning using information on the operation state when an instrument corresponding to the manipulation device is operated manually. For example, an instrument may be operated directly by the operator's hand, and a trained model may be generated using information on the operation state of the operated instrument. This makes it possible to generate a trained model in which the operation of an instrument directly operated by the operator can be reproduced by the manipulation device. For example, a position sensor may be attached to the instrument to record a log of position information, or the operator's movements may be analyzed from camera images. Also, for example, when performing a suturing operation, information such as the trajectory of forceps when a person performs surgery themselves with an endoscope may be used to generate a trained model.

[0129] In the above embodiment, the inference unit 614 is shown to take observation information and operation commands from multiple past time points as input and output the next inference operation command, but the disclosure is not limited to this. For example, the inference unit 614 may take observation information and operation commands from one past time point as input and output the next inference operation command. This reduces the control burden on the inference control device 600.

[0130] In the above embodiment, an example was shown in which an image G captured by the endoscope 3 is input to the inference unit 614 as observation information, but the disclosure is not limited to this. For example, a camera may be placed on the ceiling or wall of the operating room, and the image G captured by this camera may be input to the inference unit 614 as observation information. This makes it possible to generate inference operation commands that take into account the environment around the patient P and the robot arm 50. Alternatively, a force sensor may be placed as an environmental information acquisition unit instead of the endoscope 3. This makes it possible to detect interference between the instrument 2 and the surgical area of ​​the patient P using the force sensor, and thus generate inference operation commands that suppress interference.

[0131] In the above embodiment, an example was shown in which the current operating position of the instrument 2 and the operating position of the instrument 2 based on the inference operation command from the present time onward are displayed on both the monitor 140 and the display unit 220, but the disclosure is not limited to this. For example, these operating positions may be displayed only on the monitor 140. When the operator is operating the device, their head is immersed in the monitor 140, so it may be acceptable for the operating positions to be displayed only on the monitor 140.

[0132] In the above embodiment, an example was shown in which the orientation of the multiple link parts of the operating unit 110 is changed to follow the orientation of the robot arm 50 and the instrument 2 in autonomous mode, but the disclosure is not limited thereto. For example, in autonomous mode, the orientation of the multiple link parts of the operating unit 110 may remain stationary without being changed to follow the orientation of the robot arm 50 and the instrument 2. This reduces the control burden on the first control device 310.

[0133] In the above embodiment, an example was shown in which the robot arm 50 and the instrument 2 attached thereto operate in autonomous mode, but the disclosure is not limited thereto. For example, the robot arm 50 may operate in autonomous mode, and the instrument 2 may operate in remote control mode. This allows the operator to concentrate on operating the instrument 2.

[0134] In the above embodiment, an example of applying the present disclosure to a surgical support system 500 comprising a surgical support robot 100 and a remote control device 200 was shown, but the present disclosure is not limited thereto. For example, as shown in the first modified example in Figure 26, the present disclosure may be applied to a social robot 700. The social robot 700 is equipped with an interface for dialogue and is capable of communicating with people. The social robot 700 also comprises a pair of articulated robot arms 701 and a trolley 702. In other words, the social robot 700 has multiple degrees of freedom. For example, the social robot 700 normally operates in autonomous mode. For example, the switching unit 703 makes it possible to keep the pair of articulated robot arms 701 in autonomous mode and switch the trolley 702 to remote control mode. The remote control mode is executed by an operator operating the control unit 704. In this way, by switching a part of the social robot 700, which has multiple degrees of freedom, between autonomous mode and remote control mode, even if the social robot 700 performs an inappropriate action while operating in autonomous mode, it is possible to switch to remote control mode and correct the action.

[0135] Furthermore, as shown in the second modified example in Figure 27, the present disclosure may also be applied to an industrial robot 710 placed in a factory or the like. The industrial robot 710 includes, for example, a vertical articulated robot 711 and a horizontal articulated robot 712. For example, the industrial robot 710 normally operates in autonomous mode. The switching unit 713 makes it possible to, for example, keep the vertical articulated robot 711 in autonomous mode and switch the horizontal articulated robot 712 to remote control mode. The remote control mode is activated by an operator operating the control unit 714. Here, when there are many robots, many operators are required to operate all robots in remote control mode. Therefore, by switching some of the robots between autonomous mode and remote control mode, even when there are many robots, a small number of operators can select and operate only the robots that require remote control mode.

[0136] The above embodiment shows an example in which multiple robot arms 50 are arranged, but the disclosure is not limited thereto. For example, the disclosure may be applied to a surgical support system in which only one robot arm 50 is arranged. In this case, either the robot arm 50 or the instrument 2 can be switched between autonomous mode and remote control mode. This makes it possible to smoothly switch between autonomous mode and remote control mode even in a surgical support system in which only one robot arm 50 is arranged.

[0137] In the above embodiment, an example was shown in which the operation command included a suturing operation command for suturing the surgical site of patient P, but the present disclosure is not limited thereto. For example, the operation command may be a command other than a suturing operation command. For example, the operation command may be an excision command for excising the surgical site of patient P. Furthermore, the operation command may include multiple surgical operations such as a suturing operation command and an excision command. In this way, the present disclosure enables smooth switching between autonomous mode and remote control mode even for surgical operations other than suturing.

[0138] The above embodiment shows an example of generating a trained model based on information collected by the data collector 611 of the inference control device 600, but the disclosure is not limited thereto. For example, a trained model may be generated based on information stored on a cloud server. The generated trained model may also be used in a virtual simulator rather than an actual surgical support system. Furthermore, the manipulation system may store the results of the operation evaluation in the data collector.

[0139] In the above embodiment, an example was shown where the switching unit of the present disclosure is an incision pedal 125L, but the present disclosure is not limited thereto. For example, the switching unit of the present disclosure may be a switching unit that switches between autonomous mode and remote control mode by detecting the movement of the operator's gaze. In this case, for example, a sensor that detects the movement of the operator's gaze is arranged on the monitor 140. Alternatively, the switching unit of the present disclosure may be a switching unit that switches between autonomous mode and remote control mode based on the detection of a part of the operator's body. In this case, a sensor that detects the movement of a part of the operator's body is arranged. For example, the sensor detects the movement of the operator's arm or head. By switching between autonomous mode and remote control mode with such a switching unit, the operator does not need to take their hands off the control unit to switch between autonomous mode and remote control mode, so that the operator can switch between autonomous mode and remote control mode while continuing to perform suturing work. Alternatively, the switching unit of the present disclosure may be a button separately arranged on the control unit 110. This allows the operator to operate the button by hand, making it easy to switch between autonomous mode and remote control mode.

[0140] In the above embodiment, an example was shown in which, in autonomous mode, a manual operation command is replaced with an inference operation command before the limiting unit of the first control device 310 limits the speed of the manual operation command. However, the disclosure is not limited thereto. For example, in autonomous mode, a manual operation command may be replaced with an inference operation command before the limiting unit of the first control device 310 limits the acceleration of the manual operation command. This makes it possible to limit the acceleration of the operation of the surgical support system in the same way in both autonomous mode and remote control mode.

[0141] In the above embodiment, an example was shown in which, in autonomous mode, the monitoring unit of the first control device 310 determines that the operation of the robot arm 50 and instrument 2 in autonomous mode satisfies predetermined conditions and switches to remote control mode. However, this disclosure is not limited to this. For example, the monitoring unit of the first control device 310 does not need to determine predetermined conditions. This makes it possible to suppress interference with the operation of the robot arm 50 and instrument 2 in autonomous mode.

[0142] In the above embodiment, the predetermined conditions include, but are not limited to, the following: when the robot arm 50 and instrument 2 are not in a state where they can be operated by the remote control device 200; when the robot arm 50 and instrument 2 move outside a preset monitoring range; and when the speed of the robot arm 50 and instrument 2 based on the inference operation command exceeds a predetermined speed threshold. For example, the predetermined conditions may include only one or two of these conditions. This reduces the control burden on the first control device 310.

[0143] In the above embodiment, an example was shown in which the cutting pedal 125L switches between autonomous mode and remote control mode, and the cutting pedal 125R switches between starting the storage operation of training data and other operations. However, this disclosure is not limited to this. For example, the above switching may be performed by detecting the movement of the operator's gaze or by detecting a part of the operator's body other than their hands. This allows the above switching to be performed while the operator continues to operate the control unit 110.

[0144] In the above embodiment, an example was shown in which the current operating position of Instrument 2 and the operating position of Instrument 2 one second later based on the inference operation command from the current point in time are displayed, but the disclosure is not limited to this. In the disclosure, the operating positions of Instrument 2 at multiple points in time from the current point in time may be displayed. For example, the current operating position of Instrument 2, the operating position of Instrument 2 one second later based on the inference operation command from the current point in time, and the operating position of Instrument 2 two seconds later may be displayed. As a result, the operating positions of Instrument 2 at multiple points in time from the current point in time are displayed, so the operator can easily understand how Instrument 2 will operate in the future based on the inference operation command.

[0145] In the above embodiment, an example was shown in which the current operating position of instrument 2 and the operating position of instrument 2 based on inferred operation commands from that point onward are superimposed on the image G captured by the endoscope 3. However, this disclosure is not limited to this. For example, the current operating position of instrument 2 and the operating position of instrument 2 based on inferred operation commands from that point onward may be displayed outside the image G captured by the endoscope 3. This improves the visibility of the operating position.

[0146] In the above embodiment, an example was shown in which the current mode, whether autonomous mode or remote control mode, is displayed in text, but the disclosure is not limited thereto. For example, an alert sound corresponding to the current mode may be provided. This prevents text from being superimposed on the image G captured by the endoscope 3, thereby improving the visibility of the image G captured by the endoscope 3.

[0147] In the above embodiment, an example was shown in which the first control device 310 and the inference control device 600 are arranged separately, but the disclosure is not limited thereto. For example, the first control device 310 may also serve as the inference control device 600. This simplifies the configuration of the surgical support system 500.

[0148] Furthermore, although the above embodiment shows an example in which four robot arms 50 are provided, this disclosure is not limited to this. In this disclosure, the number of robot arms 50 may be other than four.

[0149] Furthermore, although the above embodiment shows an example in which the arm portion 51 and the positioner 30 are composed of a 7-axis articulated robot, this disclosure is not limited to this. For example, the arm portion 51 and the positioner 30 may be composed of an articulated robot with an axis configuration other than a 7-axis articulated robot. An axis configuration other than a 7-axis articulated robot would be, for example, 6 axes or 8 axes.

[0150] Furthermore, while the above embodiment shows an example in which the surgical assistance robot 100 includes a medical cart 10, a positioner 30, an arm base 40, and a robot arm 50, this disclosure is not limited to this. For example, the medical cart 10, the positioner 30, and the arm base 40 are not necessarily required, and the surgical assistance robot 100 may include only the robot arm 50.

[0151] The functions of the elements disclosed herein may be implemented using one or more circuits or processing circuits, including general-purpose processors, special-purpose processors, integrated circuits, ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and / or conventional circuits. The functions of the elements disclosed herein may be implemented using one or more circuits or processing circuits, including combinations of general-purpose processors, special-purpose processors, integrated circuits, ASICs, FPGAs, and conventional circuits. One or more circuits or processing circuits may be programmed using one or more programs stored together or individually in one or more memories, or otherwise configured to perform the disclosed functions. A processor is considered a processing circuit or circuit because it includes transistors and other circuits. A processor may be a programmed processor that executes programs stored in memory. In this disclosure, a circuit, unit, or means is hardware that performs the enumerated functions individually or in combination with each other, or hardware programmed to perform the enumerated functions individually or in combination with each other. Hardware may be any hardware disclosed herein that is programmed or configured to perform the enumerated functions. A computer program, including computer instructions, is stored in memory. Computer instructions provide logic and routines that enable the hardware to perform the methods disclosed herein. Hardware includes, for example, processing circuits or circuits. A computer program may be implemented in a known format on computer-readable storage media, computer program products, memory devices, recording media such as CD-ROMs or DVDs, and / or in the memory of FPGAs or ASICs.

[0152] [Embodiments] The exemplary embodiments described above will be understood by those skilled in the art to be specific examples of the following embodiments.

[0153] (Aspect 1) A manipulation system comprising: a manipulation device that operates based on an operation command; a state acquisition unit that acquires input information including at least one of the following: information on the operation state of an operation device that outputs the operation command based on an operation by an operator; information on the operation state of the manipulation device; and information on the operation state of an instrument corresponding to the manipulation device that has been manually operated; an evaluation acquisition unit that acquires an evaluation of the operation of the manipulation device or the instrument that operates in response to the input information; and a learning model generation unit that generates a trained model that takes observation information of the environment surrounding the manipulation device and at least one of the operation command as input and outputs an inferred operation command which is the inferred operation command, based on the input information, the environment surrounding the operated manipulation device, and the evaluation of the operation.

[0154] (Aspect 2) The manipulation system according to aspect 1, further comprising an operation command output unit that takes observation information of the environment surrounding the manipulation device and an operation command as input and outputs an inference operation command, using the trained model generated by the learning model generation unit based on the input information, the environment surrounding the operated manipulation device, and the operation evaluation.

[0155] (Aspect 3) The manipulation system according to aspect 1 or aspect 2, wherein the evaluation acquisition unit acquires the operation evaluation based on at least one of the information acquired by the state acquisition unit and the input operation of the evaluator.

[0156] (Aspect 4) The manipulation system according to any one of aspects 1 to 3, wherein the evaluation acquisition unit acquires the operation evaluation based on at least one of the processing speed, processing accuracy, and operation trajectory of the operation of the manipulation device.

[0157] (Aspect 5) The manipulation system according to any one of aspects 1 to 4, wherein the state acquisition unit includes an imaging unit that images the environment surrounding the manipulation device, and the evaluation acquisition unit acquires the operation evaluation based on the image captured by the imaging unit.

[0158] (Aspect 6) The manipulation system according to any one of aspects 1 to 5, wherein the operation evaluation includes operator information, which is information of the operator operating the operating device, and the operation command output unit outputs the inference operation command corresponding to the input operator operation using the trained model based on the input operator information.

[0159] (Aspect 7) The manipulation system according to any one of aspects 1 to 6, wherein the operation evaluation includes a numerical evaluation, and the operation command output unit outputs the inference operation command so as to perform an operation in which the value of the operation evaluation is equal to or greater than a predetermined value.

[0160] (Aspect 8) The manipulation system according to any one of aspects 1 to 7, comprising an environmental information acquisition unit that observes the environment surrounding the manipulation device and outputs the observed information, the operation command output unit outputs an inference operation command when the operation command and the observed information acquired by the environmental information acquisition unit are input to the trained model, and the manipulation device operates based on the inference operation command.

[0161] (Aspect 9) The manipulation system according to any one of aspects 1 to 8, wherein the state acquisition unit includes at least one of an imaging unit for imaging the environment surrounding the manipulation device and a force sensor for detecting the force applied to the manipulation device.

[0162] (Aspect 10) The manipulation system according to any one of aspects 1 to 9, further comprising a display unit that displays the operating position of the manipulation device based on the inference operation command.

[0163] (Aspect 11) The manipulation system according to any one of aspects 1 to 10, comprising: a surgical apparatus including a robotic arm as a manipulation device to which a surgical instrument is attached at its tip; and an operating device including an operating section for receiving operations on the surgical instrument.

[0164] (Aspect 12) The manipulation system according to aspect 11, wherein the robot arm includes a first robot arm to which an endoscope is attached at its tip, and a second robot arm to which a surgical instrument other than the endoscope is attached at its tip, and the endoscope captures observational information of the environment surrounding the surgical instrument attached to the first robot arm.

[0165] (Aspect 13) The manipulation system according to aspect 11 or aspect 12, wherein the operation command includes a suturing operation command for suturing the surgical site of a patient.

[0166] (Aspect 14) A method for controlling a manipulation system, comprising: an operating device that outputs an operation command based on an operation by an operator; and a manipulation device that operates when operated based on the operation command, the method comprising: acquiring input information that includes at least one of the following: information on the operating state of the operating device; information on the operating state of the manipulation device; and information on the operating state of a device corresponding to the manipulation device that has been manually operated; acquiring an operational evaluation of the manipulation device or the device that operates in response to the input information; and generating a trained model that takes observational information of the environment surrounding the manipulation device and at least one of the operation command as input and outputs an inferred operation command which is the inferred operation command, based on the input information, the environment surrounding the operated manipulation device, and the operational evaluation.

[0167] (Aspect 15) A program for executing a control method for a manipulation system, comprising: an operating device that outputs an operation command based on an operation by an operator; and a manipulation device that operates when operated based on the operation command, the program for executing a control method for a manipulation system, comprising: acquiring input information including at least one of information on the operation state of the operating device, information on the operating state of the manipulation device, and information on the operating state of a device corresponding to the manipulation device that has been manually operated; acquiring an operational evaluation of the manipulation device or the device that operates in response to the input information; and generating a trained model that takes observational information of the environment surrounding the manipulation device and at least one of the operation command as input and outputs an inferred operation command which is the inferred operation command, based on the input information, the environment surrounding the operated manipulation device, and the operational evaluation.

[0168] 1 Surgical instruments 2 Instruments (manipulation devices) 3 Endoscope (state acquisition unit, environmental information acquisition unit, imaging unit) 50 Robot arm (manipulation device) 50a Robot arm (second robot arm) 50b Robot arm (second robot arm) 50c Robot arm (first robot arm) 50d Robot arm (second robot arm) 100 Surgical support robot (surgical device) 110 Control unit 140 Monitor (display unit) 200 Remote control device (control device) 220 Display unit 311a Program 500 Surgical support system (manipulation system) 611 Data acquisition unit (state acquisition unit) 612 Learning unit (learning model generation unit) 613 Evaluator (evaluation acquisition unit) 614 Inference unit (operation command output unit) 621 Program P Patient

Claims

1. A manipulation system comprising: a manipulation device that operates based on an operation command; a state acquisition unit that acquires input information including at least one of the following: information on the operation state of an operation device that outputs the operation command based on an operation by an operator; information on the operation state of the manipulation device; and information on the operation state of an instrument corresponding to the manipulation device that has been manually operated; an evaluation acquisition unit that acquires an evaluation of the operation of the manipulation device or the instrument that operates in response to the input information; and a learning model generation unit that generates a trained model that takes observation information of the environment surrounding the manipulation device and at least one of the operation command as input and outputs an inferred operation command which is the inferred operation command, based on the input information, the environment surrounding the operated manipulation device, and the evaluation of the operation.

2. The manipulation system according to claim 1, further comprising an operation command output unit that, based on the input information, the environment surrounding the operated manipulation device, and the operation evaluation, uses the trained model generated by the learning model generation unit to input at least one of the observation information of the environment surrounding the manipulation device and the operation command, and outputs the inference operation command.

3. The manipulation system according to claim 1, wherein the evaluation acquisition unit acquires the operation evaluation based on at least one of the information acquired by the state acquisition unit and the input operation of the evaluator.

4. The manipulation system according to claim 1, wherein the evaluation acquisition unit acquires the operation evaluation based on at least one of the processing speed, processing accuracy, and operation trajectory of the operation of the manipulation device.

5. The manipulation system according to claim 1, wherein the state acquisition unit includes an imaging unit that images the environment surrounding the manipulation device, and the evaluation acquisition unit acquires the operation evaluation based on the image captured by the imaging unit.

6. The manipulation system according to claim 2, wherein the operation evaluation includes operator information, which is information of an operator operating the operating device, and the operation command output unit outputs the inference operation command corresponding to the input operator operation using the trained model based on the input operator information.

7. The manipulation system according to claim 2, wherein the operation evaluation includes a numerical evaluation, and the operation command output unit outputs the inference operation command so as to perform an operation in which the value of the operation evaluation is equal to or greater than a predetermined value.

8. The manipulation system according to claim 2, comprising an environmental information acquisition unit that observes the environment surrounding the manipulation device and outputs the observed information, the operation command output unit outputs an inference operation command when the operation command and the observed information acquired by the environmental information acquisition unit are input to the trained model, and the manipulation device operates based on the inference operation command.

9. The manipulation system according to claim 1, wherein the state acquisition unit includes at least one of an imaging unit for imaging the environment surrounding the manipulation device and a force sensor for detecting the force applied to the manipulation device.

10. The manipulation system according to claim 1, further comprising a display unit that displays the operating position of the manipulation device based on the inference operation command.

11. The manipulation system according to claim 1, comprising: a surgical apparatus including a robotic arm as a manipulation device to which a surgical instrument is attached at its tip; and an operating device including an operating section for receiving operations on the surgical instrument.

12. The manipulation system according to claim 11, wherein the robotic arm includes a first robotic arm to which an endoscope is attached at its tip, and a second robotic arm to which a surgical instrument other than the endoscope is attached at its tip, and the endoscope captures observational information of the environment surrounding the surgical instrument attached to the first robotic arm.

13. The manipulation system according to claim 11, wherein the operation command includes a suturing operation command for suturing the surgical site of a patient.

14. A method for controlling a manipulation system including a manipulation device that operates based on an operation command, comprising: acquiring input information including at least one of: information on the operation state of an operating device that outputs the operation command based on an operation by an operator; information on the operation state of the manipulation device; and information on the operation state of an instrument corresponding to the manipulation device that has been manually operated; acquiring an operation evaluation of the manipulation device or the instrument that operates in response to the input information; and generating a trained model that takes observation information of the environment surrounding the manipulation device and at least one of the operation command as input and outputs an inferred operation command which is the inferred operation command, based on the input information, the environment surrounding the operated manipulation device, and the operation evaluation.

15. A program for executing a control method for a manipulation system including a manipulation device that operates based on an operation command, comprising: acquiring input information including at least one of: information on the operation state of an operating device that outputs the operation command based on an operation by an operator; information on the operation state of the manipulation device; and information on the operation state of an instrument corresponding to the manipulation device that has been manually operated; acquiring an operation evaluation of the manipulation device or the instrument that operates in response to the input information; and generating a trained model that takes observation information of the environment surrounding the manipulation device and at least one of the operation command as input and outputs an inferred operation command which is the inferred operation command, based on the input information, the environment surrounding the operated manipulation device, and the operation evaluation.