Surgical robot system having living body movement synchronization function

Abstract

A surgical robot system according to an embodiment of the present invention comprises: an imaging unit that captures image data of a subject; a medical instrument that is inserted into a living body and reaches the subject in a surgical region; a compensation calculation module that receives a first control command as a signal for the medical instrument to initiate movement compensation corresponding to the movement of the subject, calculates the movement compensation of the medical instrument with respect to the subject, and then outputs a second control command received from a synchronization module, thereby compensating and driving the medical instrument at the output timing of the second control command; and the synchronization module which calculates the timing of the second control command by taking into account the timing of the first control command and the movement of the living body. The surgical robot system is characterized in that the timing of the first control command and the timing of the second control command differ from each other, but the medical instrument is compensated and driven so as to correspond to the movement of the subject.

Description

Surgical robot system with bio-motion synchronization function

[0001] The present invention relates to a surgical robot system in which a medical device can be compensated and driven to correspond to the movement of a subject captured in a surgical area, and to a surgical robot system in which the compensated driving of the medical device can be systematically synchronized.

[0002] Minimally invasive surgery is a surgical method in which surgical instruments are inserted and procedures are performed by creating a small incision inside the patient's body. This surgical method offers numerous advantages over open surgery, including faster patient recovery, reduced risk of infection, and superior cosmetic results. With the continuous advancement of medical technology, minimally invasive surgery is evolving into robot-assisted surgical systems capable of remote operation. Reference may be made to the applicant's previously filed Korean Published Patent No. 10-2025-0078369 as an example of a surgical robot system for minimally invasive surgery.

[0003] A surgical robot system controlling medical instruments can be utilized not only for minimally invasive surgery but also for invading through natural openings of the human body, steering to target tissue, and performing intended surgery. An example of such a surgical robot system may be referenced in the applicant's prior art, Korean Published Patent No. 10-2024-0009905.

[0004] Surgical robotic systems are a field of medical technology designed to enhance the accuracy and efficiency of minimally invasive surgery, operating in an environment where medical professionals remotely control medical instruments via an operator console interface. In such systems, medical instruments are typically inserted and steered along complex anatomical paths to the surgical area where the target tissue is located, while the surgical site is monitored during this process using an endoscope or other imaging means.

[0005] In the environment of the aforementioned surgical robot system, dynamic movements occur in the biological subject to surgery due to various biological factors such as respiration and heartbeats. In this regard, the applicant has disclosed, in a prior patent, a motion compensation device that takes into account the dynamic movements of a biological body (Korean Published Patent No. 10-2024-0009376).

[0006] Conventionally, compensatory actuation methods have been proposed to respond to the movement of a patient caused by biological or mechanical factors, thereby correcting the movement of a medical device to correspond with the patient's movement. While various methods for compensatory actuation of medical devices can be proposed, accurately estimating the patient's movement in real time and immediately correcting the device's operation accordingly presents a very high technical difficulty. In practice, due to limitations in real-time computation, data scarcity, and delays in remote control, most methods rely on periodic compensatory actuation methods targeting movements with a fixed period. Compensating for medical device movement using such periodic methods requires additional algorithms and control technologies to align the timing between the patient's movement and the medical device's correction action.

[0007] The present invention aims to provide a surgical robot system capable of recognizing the in vivo movement of a surgical subject and compensating for the movement of a medical device inserted into the body.

[0008] The present invention aims to provide a surgical robot system capable of systematically synchronizing command signals that drive compensation by taking into account the movement of the living body when driving a medical device in response to the movement of a patient.

[0009] The present invention aims to provide a synchronization function capable of calculating the synchronization timing of biological movements using only image data of the surgical area without additional sensors.

[0010] A surgical robot system according to an embodiment of the present invention comprises: a shooting unit for acquiring image data of a subject; a medical device inserted into a living body and reaching the subject in a surgical area; a compensation calculation module that receives a first control command as a signal to initiate movement compensation corresponding to the movement of the subject and forms an output timing at the time of a second control command to compensate and drive the medical device; and a synchronization module that calculates the time of the second control command by reflecting the time of the first control command and the movement of the living body; wherein the time of the first control command and the time of the second control command are different from each other, and the medical device is compensated and driven to correspond to the movement of the subject.

[0011] In one embodiment, the compensation calculation module can calculate the movement compensation of the medical device for the subject and then output the second control command at the time of the second control command received from the synchronization module.

[0012] In one embodiment, the synchronization module can estimate the movement characteristics of the living organism based on image data of the subject and calculate the timing of the second control command by applying an output time difference from the timing of the first control command based on the estimated movement characteristics of the living organism.

[0013] In one embodiment, the synchronization module can calculate a first time difference, which is the time difference to the movement of the living organism corresponding to the time of the first control command, by utilizing a periodic characteristic among the movement characteristics of the living organism.

[0014] In one embodiment, the synchronization module can calculate the first time difference based on data within one cycle from the time of the first control command in the breathing characteristics related to inhalation or exhalation among the movement characteristics of the living organism.

[0015] In one embodiment, the synchronization module may use respiratory parameter information of the living organism as the respiratory characteristic among the movement characteristics of the living organism.

[0016] In one embodiment, the synchronization module can estimate a feature point associated with inhalation or exhalation in the movement characteristics of the biological body and calculate a second time difference, which is the time difference from the first time difference to the feature point.

[0017] In one embodiment, the synchronization module can calculate the output time difference by adding the first time difference and the second time difference.

[0018] In one embodiment, the synchronization module may set the inflection point where respiration changes in the movement of the living organism as the feature point.

[0019] In one embodiment, the synchronization module estimates a feature point associated with inhalation or exhalation in the movement characteristics of the living organism, sets the breathing cycle of the living organism as a first time difference, calculates the time difference from the time of the first control command to the feature point as a second time difference, and calculates the output time difference as the sum of the first time difference and the second time difference.

[0020] In one embodiment, the compensation calculation module may receive the first control command generated at the time when it is determined that the medical device has stopped in the surgical area from the sequence of image data acquired by the imaging unit.

[0021] In one embodiment, the synchronization module can form the timing of the second control command at an inflection region where respiration changes in the movement characteristics of the biological body.

[0022] In one embodiment, the synchronization module calculates the output time difference based on data within one cycle from the time of the first control command in respiratory characteristics related to inhalation or exhalation among the movement characteristics of the living organism, and the output time difference may be calculated to be a range exceeding one cycle from the time of the first control command.

[0023] In one embodiment, the synchronization module can estimate the movement characteristics of the biological body using only continuous image data captured by the imaging unit of the subject, and calculate the timing of the second control command through image analysis.

[0024] In one embodiment, the synchronization module may include an image analysis unit that receives continuous image data of the subject and evaluates the quality or similarity of the subject in the continuous image data.

[0025] In one embodiment, the image analysis unit can estimate breathing characteristics in the movement of the living organism by determining the similarity of a second image data having a different sequence based on a first image data among continuous image data of the subject.

[0026] In one embodiment, the compensation calculation module calculates a movement compensation that moves the medical device so that the subject and the medical device form a fixed distance, outputs the second control command received from the synchronization module to the medical device, and the movement compensation may be a periodic compensation drive corresponding to the movement of the subject according to the respiration of the living organism.

[0027] In one embodiment, the synchronization module may include: an image analysis unit that receives continuous image data capturing the subject and evaluates the quality or similarity of the subject in the continuous image data to estimate a feature point associated with the inhalation or exhalation of the biological body; and a synchronization unit that calculates an output time difference to the time of the movement of the biological body corresponding to the time of the feature point and transmits it to the compensation calculation module.

[0028] In addition, the present invention comprises, in a surgical robot system, a capturing unit for acquiring image data of a subject; a medical device inserted into the body and reaching the subject in a surgical area; a processor that receives a first control command as a signal to initiate movement compensation corresponding to the movement of the subject, calculates the movement compensation of the medical device for the subject, and outputs a second control command as a signal to drive the movement compensation to drive the medical device; and an operator console provided with a control unit that outputs the first control command, wherein the processor forms the output timing of the second control command by applying an output time difference from the time when the first control command generated by the operation of the control unit is received.

[0029] In addition, the present invention is further characterized in that a method for synchronizing the movement of a medical device performed in a processor of a surgical robot system comprises: (a) receiving image data of a subject acquired by a capturing unit; (b) receiving a first control command as a signal for a medical device inserted into the body and reaching the subject in a surgical area to start movement compensation corresponding to the movement of the subject; (c) estimating the movement characteristics of the body based on the image data of the subject and calculating the movement compensation of the medical device for the subject; and (d) calculating the timing of a second control command by applying an output time difference from the timing of the first control command based on the movement characteristics of the body, wherein the output timing is formed at the calculated timing of the second control command and the medical device is driven for compensation.

[0030] According to the present invention, a medical device can be compensated and driven to correspond to the in vivo movement of a subject. Even if the timing of a first control command instructing the compensatory driving of the medical device is formed regardless of the movement state of the body, the present invention can synchronize and output a second control command that executes the compensatory driving of the medical device at a timing corresponding to the movement of the body.

[0031] The present invention has the effect of determining synchronization timing without a separate biosignal sensor by calculating the output time difference with the first control command using only image data during the synchronization process of the second control command.

[0032] The present invention can minimize initial driving delay by estimating signals related to inhalation or exhalation among periodic characteristics of respiration as data within one cycle and calculating synchronization timing.

[0033] FIG. 1 is an embodiment of a surgical robot system according to the present invention.

[0034] FIG. 2 is another embodiment of a surgical robot system according to the present invention.

[0035] FIG. 3 is an explanatory diagram from the perspective of operation of a surgical robot system according to the present invention.

[0036] FIG. 4 is an explanatory diagram of the flexible endoscope part of a surgical robot system according to the present invention.

[0037] FIG. 5 is a configuration diagram of a motion compensation method according to an embodiment of the present invention.

[0038] FIG. 6 is an explanatory diagram of the movement of an overtube or surgical tool part as a medical device according to an embodiment of the present invention.

[0039] FIG. 7 is an explanatory diagram for explaining the motion compensation calculation of a compensation calculation module according to an embodiment of the present invention.

[0040] FIG. 8 shows the relative movement of a subject to a medical device according to an embodiment of the present invention.

[0041] FIG. 9 is an explanatory diagram of the orientation according to an embodiment of the present invention.

[0042] FIG. 10 is an explanatory diagram of an embodiment of the present invention in which the object of movement compensation according to the movement of the object to be treated, or the driving object of movement compensation, is a surgical tool part.

[0043] FIG. 11 is an explanatory diagram of an embodiment of the present invention in which the object of motion compensation according to the movement of the object to be compensated, or the driving object of motion compensation, is an overtube.

[0044] FIG. 12 is an explanatory diagram of the first to fourth data flows according to an embodiment of the present invention.

[0045] FIG. 13 is an explanatory diagram of data collection or calculation of breathing movements by a shooting unit according to an embodiment of the present invention.

[0046] FIG. 14 is an explanatory diagram of an embodiment of the calculation of breathing movements, such as breathing cycles, in a compensation product module according to an embodiment of the present invention.

[0047] FIG. 15 is a configuration diagram of a surgical robot system that performs a biological movement synchronization function according to an embodiment of the present invention.

[0048] FIG. 16 is a configuration diagram of a synchronization module according to an embodiment of the present invention.

[0049] FIG. 17 is a functional diagram of a synchronization module according to an embodiment of the present invention.

[0050] FIG. 18 is a functional diagram of a synchronization unit according to an embodiment of the present invention, where FIG. 18 (a) is a diagram of a feature point estimated as information for image quality evaluation, and FIG. 18 (b) is a diagram of an output time difference calculated using the breathing cycle relationship between the feature point estimated as image quality evaluation information.

[0051] FIG. 19 illustrates a case in which an output time difference is calculated according to an embodiment of the present invention, where FIG. 19 (a) is a case in which a second control command is formed within a range of 2 cycles from the time point of a first control command, FIG. 19 (b) is a case in which a second control command is formed within a range of 3 cycles from the time point of a first control command, and FIG. 19 (c) is a case in which a second control command is formed at an arbitrary point.

[0052] FIG. 20 is an explanatory diagram illustrating the calculation of output disparity according to another embodiment of the present invention, where FIG. 20 (a) shows a feature point learned from image similarity evaluation information, and FIG. 20 (b) is an explanatory diagram illustrating the calculation of output disparity using the learned feature point.

[0053] FIG. 21 is a schematic diagram of a bio-motion synchronization method according to an embodiment of the present invention.

[0054] The various embodiments described in this document are illustrative for the purpose of clearly explaining the technical concept of the invention and disclosure, and are not intended to limit them to specific embodiments. The technical concept of the invention and disclosure includes various modifications, equivalents, alternatives, and embodiments selectively combined from all or part of each embodiment described in this document. Furthermore, the scope of rights of the technical concept of the invention and disclosure is not limited to the embodiments presented below or the specific descriptions thereof.

[0055] Terms used in this document, including technical or scientific terms, may have the meaning generally understood by those skilled in the art to which the invention and disclosure pertain.

[0056] Expressions used in this document, such as "includes," "may include," "is equipped," "may be equipped," "has," and "may have," imply that functions, operations, or components exist as the subject feature and do not exclude the existence of other additional features. In other words, such expressions should be understood as open-ended terms implying the possibility of including other embodiments.

[0057] Singular expressions used in this document may include the meaning of the plural form unless the context otherwise indicates, and this applies likewise to singular expressions described in the claims.

[0058] Expressions used in this document such as “A, B, and C,” “A, B, or C,” “A, B, and / or C,” or “at least one of A, B, and C,” “at least one of A, B, or C,” “at least one of A, B, and / or C,” “at least one selected from A, B, and C,” “at least one selected from A, B, or C,” “at least one selected from A, B, and / or C,” etc., may mean each of the listed items or all possible combinations of the listed items. For example, “at least one selected from A and B” may refer to (1) A, (2) at least one of A, (3) B, (4) at least one of B, (5) at least one of A and at least one of B, (6) at least one of A and B, (7) at least one of B and A, and (8) all of A and B.

[0059] The expression "based on" as used in this document is used to describe one or more factors affecting an act or action of a decision or judgment described in the phrase or sentence containing such expression, and this expression does not exclude additional factors affecting said act or action of a decision or judgment.

[0060] As used in this document, the expression that a certain component (e.g., a first component) is "connected" or "connected" to another component (e.g., a second component) may mean not only that the said certain component is directly connected or connected to the said other component, but also that it is connected or connected through a new other component (e.g., a third component).

[0061] The expression "configured to" as used in this document may have meanings such as "set to," "capable of," "modified to," "made to," or "able to," depending on the context, and is distinguished from the meaning of "consist."

[0062] Various embodiments of the present disclosure will be described below with reference to the accompanying drawings. In the accompanying drawings and the description thereof, identical or substantially equivalent components may be given the same reference numerals. Furthermore, in the description of the various embodiments below, the description of identical or corresponding components may be omitted, but this does not mean that such components are not included in the embodiments.

[0063] A control unit according to an embodiment of the present invention may be implemented in the form of physical hardware such as a controller, a processor, a microprocessor, a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit). The control unit may include a memory, which is a non-transient computer-readable storage medium for storing logic circuits and software instructions, and such instructions are transmitted as control instructions and configured to perform the specific functions of the control unit described herein when executed by a processor.

[0064] In terms of implementation form, the memory and processor according to an embodiment of the present invention may be implemented as separate semiconductor circuits that are physically separated. In another implementation form, the memory and processor may be implemented together within a single integrated semiconductor circuit (SoC: System on Chip). The processor may be implemented as a single processor having a single-core or multi-core structure, or as a distributed processing architecture including a plurality of individual processors.

[0065] In particular, the components referred to as "modules," "units," etc. in the present invention are implemented as devices having a specific physical hardware structure as described above, and refer to specific circuits or programmed electronic devices having actual physical configurations rather than simple functional blocks. For example, a "synchronization module" may be implemented as a specialized semiconductor circuit implementing a neural network architecture, a Field Programmable Gate Array (FPGA), or a combination of a processor and memory programmed to execute a specific algorithm.

[0066] Surgical systems, such as those used in minimally invasive medical procedures, may include large and complex equipment to precisely control and operate relatively small tools or instruments.

[0067] A surgical robot system having a single entry port can be used for various surgeries to use various surgical instruments in a single system. Depending on the embodiment, the surgical robot system may be disclosed as a remote surgical system capable of operating various medical instruments through a single entry port. The surgical robot system may also be disclosed as a surgical robot system capable of operating a single medical instrument.

[0068] The types of driven medical devices may include overtubes and surgical instruments depending on the characteristics of the shaft. Medical devices can be controlled manually, via computer-assisted control, or remotely via an operator console.

[0069] A surgical robot system according to an embodiment of the present invention enables surgery using a single entry port in various parts of a patient's body. Specifically, surgery can be performed by inserting a medical instrument through the patient's oral cavity, intercostal space, thigh, or other natural openings or incisions of the body. An overtube, which is an embodiment of a medical instrument, has a plurality of lumens formed therein, and an endoscope, which is an imaging means, or a general surgical instrument can be inserted through the lumens and inserted into the body together with the overtube. Control from the operator console can independently perform remote control of not only the overtube but also the surgical instrument inserted into the overtube.

[0070] The medical device referred to in this specification may collectively refer to an overtube and other surgical instruments inserted into the overtube, and regarding the steering and control of the overtube, the applicant’s prior Korean patents No. 10-2740479 and No. 10-2684546 may be referenced.

[0071] FIG. 1 is an embodiment of a surgical robot system (1). Referring to FIG. 1, the surgical robot system (1) may include a positioning cart (3), a surgical instrument device (10), and an operator console (5). In an embodiment, the positioning cart (3) may be a mobile platform on which the surgical instrument device (10) is mounted. The positioning cart (3) may perform the function of precisely positioning the surgical instrument device (10) at the surgical site of a patient. The positioning cart (3) may be equipped with wheels at the bottom for access to the operating table. The positioning cart (3) may include a frame composed of a plurality of arms and links. The frame may be configured to enable vertical movement of the surgical instrument device (10), movement in the direction of approach to the patient, and movement away from the patient. The frame of the positioning cart (3) may be configured in various forms considering a kinematic structure that enables access to the operating table and access to the surgical instrument device (10) at the surgical site of the patient. A medical instrument may be mounted on the surgical instrument device (10). The surgical instrument device (10) may be provided in a form mounted on a frame with controlled degrees of freedom, as in the embodiment of FIG. 1, or in a form mounted on a fixed frame, as in the embodiment of FIG. 2. The medical instrument mounted on the surgical instrument device (10) may include an overtube and a surgical instrument, and may be composed of a single surgical instrument or a single endoscope.

[0072] The operator console (5) may refer to medical control equipment capable of verifying surgery-related images and controlling the positioning cart (3) and the surgical instrument device (10). The operator console (5) is a remote control device that controls the entire system and can be installed inside the surgical space or in a separate space. The operator console (5) is an operating unit and may include one or more operating devices. One embodiment of the operating unit may be an input device such as a hand grip, joystick, trackball, data glove, trigger gun, or manual controller. Another embodiment of the operating unit may be a voice recognition device or a touchscreen of a display device. Yet another embodiment of the operating unit may be an operating device such as a clutch or a foot motion controller.

[0073] Although not shown in the drawings, depending on the embodiment, the surgical robot system (1) may further include an auxiliary imaging device, such as a vision cart and a respiration measuring device. The vision cart can provide visual prompts and step-by-step surgical guidance through a touchpad or touchscreen monitor. The respiration measuring device is a device that measures and analyzes the subject's breathing status in real time and can measure respiratory rate, respiratory cycle, etc. Additionally, if necessary, medical imaging equipment for surgical assistance, such as CT and X-ray, may be linked to the system.

[0074] In an embodiment, the system configuration equipment including the positioning cart (3) and the operator console (5) may be equipped with one or more processors. The processors may process information input from various sensors. For example, the sensor information may include the drape state, the state of the positioning cart (3), the state of the surgical instrument device (10), and the state of the medical instrument.

[0075] In an embodiment, the processor may be mounted on the positioning cart (3), on the operator console (5), on the surgical instrument device (10), or on every piece of equipment. The processor may be provided as a separate device, in which case data can be processed via wired or wireless communication with each piece of equipment constituting the surgical robot system (1).

[0076] FIG. 2 is a surgical robot system (1') according to another embodiment, which is an embodiment of a surgical instrument device (10). Referring to FIG. 2, the surgical instrument device (10) may include a medical instrument (100) and a driving unit (150). The operator console (5) may include a display device (53) and an input device (51) as an operating unit. Additionally, the operator console (5) may include a memory (310) and a processor (3).

[0077] The medical device (100) may be implemented in an elongated shaft shape having a steering-capable joint structure, a flexible structure, or a bendable structure. The medical device (100) may be controlled by three or more degrees of freedom, including pitch, yaw, and rolling, through a wire-based drive mechanism, or by bending control that deflects in one axis. As an example, the medical device (100) may be realized in a 3-degree-of-freedom or 4-degree-of-freedom configuration. In a 3-degree-of-freedom configuration, 1 bending (pitch or deflection, up / down direction), rolling, and forward / backward movement are implemented. In a 4-degree-of-freedom configuration, 2 bending (pitch, up / down direction), yaw (left / right direction), rolling, and forward / backward movement are implemented. Among the degrees of freedom, the bending movement may be controlled through a wire mechanism.

[0078] In one embodiment, the medical device (100) is implemented in the form of an overtube to guide a surgical instrument into the body. An imaging unit (220), which will be described later, may be provided in this overtube. The imaging unit (220) may be implemented as an endoscope probe and inserted into the overtube via a separate shaft, or provided as an endoscope camera installed on the overtube itself. In this case, the control of the inserted surgical instrument or imaging unit (220) may be performed independently of the overtube.

[0079] In another embodiment, the medical device (100) may be implemented in the form of a single shaft in which an endoscope and a surgical instrument are integrated. For example, as a surgical instrument for removing kidney stones, an integrated medical device may be utilized in which a basket or a shaft from which a laser is emitted and a camera is provided at the end thereof. FIG. 2 illustrates a surgical instrument for removing kidney stones as an example of the medical device (100) as one of these embodiments.

[0080] As another embodiment, the medical device (100) may be implemented in a combined form of a tissue collection function for a biopsy procedure and an image acquisition function. In this case, the medical device (100) may be equipped with a basket capable of collecting a fine tissue sample and may be provided with a mechanical mechanism and a small camera that provides real-time images of the surrounding tissue.

[0081] The image acquired through the camera provided in the medical device (100) can be configured as image data in the form of an image or a recording image and can be transmitted to and stored in memory (310).

[0082] The drive unit (150) is a component for driving the medical device (100) and may be equipped with one or more motors and encoders. The motor mounted on the drive unit (150) can provide precise driving force to steer the medical device (100) in various directions. Individual drive units (150) may be designed to implement a single movement in a specific axial direction, and multiple drive units (150) may be organically combined to form an integrated motor mechanism. Through the cooperative operation of such a motor mechanism, the medical device (100) can be precisely steered along a complex anatomical path. The drive unit (150) may be provided on the manipulator (13) of the surgical device (10).

[0083] The operator console (5) may include a display device (53), an input device (51), a processor (3), and a memory (310). Here, the display device (53) and the input device (51) may be configured as an operating unit. The devices exemplified in the operator console (5) may refer to the structure of the device presented in the aforementioned Korean Patent Publication No. 10-2024-0009905 or Application No. 10-2024-0170032.

[0084] The display device (53) can display real-time image data captured by an endoscope camera mounted on a medical device (100) in high resolution. In addition to image data, the display device (53) can also display the user's operating environment and various control interfaces in an integrated manner. This display device (53) can support medical staff in intuitively monitoring and controlling the surgical process. In one embodiment, the display device (53) is implemented as a medical display supporting 4K or 8K resolution, so that even fine features of anatomical structures can be clearly displayed. In another embodiment, the display device (53) may be configured to include a touchscreen function so that medical staff can control the system by directly manipulating the screen. Additionally, the display device (53) may be configured as a single display or multiple monitors to simultaneously display various information such as biosignals, patient information, and system status along with the endoscopic image.

[0085] The input device (51) can be defined as a precision operating tool operated by a medical professional using their hands. The input device (51) may be provided in an ergonomic form such as a trackball, a multi-axis joystick, or a precision gimbal, thereby allowing the medical professional to intuitively and accurately control the medical device (100). In one embodiment, the input device (51) may be composed of multiple operating devices separated for the left hand and the right hand, enabling precise operation using both hands.

[0086] In another embodiment, the input device (51) may be implemented to include a haptic feedback function so that the user can feel the resistance of the tissue that the medical device (100) contacts with their hand. This haptic feedback function can help determine the physical characteristics of the tissue that are difficult to grasp with visual information alone. The input device (51) can serve as a key interface for accurately controlling the movement of the medical device (100) by precisely detecting the user's manual control command, converting it into a digital signal, and transmitting it to the processor (3).

[0087] The memory (310) is configured to store record information of the path in which the medical device (100) is steered within the body, and the record information may include image data of the anatomical structure (20) in which the medical device (100) is steered and control data of the medical device (100).

[0088] The memory (310) can construct an image frame sequence by storing recording information in the form of time-series data, each assigned a time index to image data and control data. The stored recording information may have a data structure in which image data and control data having the same time index are matched with each other. This time-synchronized data structure can ensure a correspondence between the anatomical image at a specific point in time and the control command at that moment.

[0089] The processor (3) may be implemented in the form of physical hardware such as a controller, a microprocessor, a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit). The processor (3) may include memory, which is a non-transient computer-readable storage medium for storing logic circuits and software instructions, and such instructions are executed by the processor and sent as control commands, corresponding to the specific functions of the processor (3) described in this specification. In an embodiment to be described later in FIG. 14, the processor (3) includes a compensation calculation module (31) and a synchronization module (32), each of which may be implemented as a logic circuit or by storing instructions in memory.

[0090] In terms of implementation form, the memory (310) and the processor (3) may be implemented as physically separated separate semiconductor circuits. In another implementation form, the memory (310) and the processor (3) may be implemented together within a single integrated semiconductor circuit (SoC: System on Chip). The processor (3) may be implemented as a single processor having a single core or multi-core structure, or as a distributed processing architecture including multiple individual processors.

[0091] In this embodiment, the recorded information may collectively refer to all types of data acquired while the medical device (100) travels along the internal path of the anatomical structure (20). The recorded information can be broadly divided into two main categories: image data and control data.

[0092] As one example of image data, high-resolution image data of anatomical structures (20) and internal pathways captured by an endoscope camera mounted on a medical device (100) may be included. As another example, continuous recording image data of anatomical structures (20) and internal pathways captured by an endoscope camera may be included, which may be captured at a frame rate of 30fps or higher to enable smooth visual reproduction. As yet another example, additional image data of anatomical structures (20) obtained from external medical imaging equipment such as ultrasound, X-ray, CT, MRI, etc. may be included.

[0093] As one embodiment of the control data, precise control signal information of each motor driving the drive unit (150) may be included. This may include parameters such as the motor's rotational speed, direction, and acceleration. As another embodiment, it may be detailed information such as the motor's actual position, rotation angle, and torque recorded in the motor's encoder. Additionally, the control data may include low-level command data such as current values, voltage values, and PWM (Pulse Width Modulation) signals that drive the motor, which may be essential elements for accurately reproducing the movement of the medical device (100). As yet another embodiment, the angle at which the tip of the medical device (100) is bent may be used as the indicated value. The motor may output a motor power signal corresponding to the indicated angle value and may bend the tip of the medical device (100) by interacting with the motor's encoder (angle) information. The motor may receive current from the current controller and receive current corresponding to the indicated angle by receiving feedback on the angle information from the encoder. In one embodiment, the motor may receive current through a current sensing function in the motor driver.

[0094] In the embodiment of the surgical robot system (1') described above, the anatomical structure (20) may be a kidney. The medical device (100) may reach a specific surgical area, such as for the removal of kidney stones, and perform surgery on the subject. In this case, a movement compensation drive and synchronization function of the medical device (100) may be required. Below, an embodiment of the movement compensation drive of the medical device (100) is described, and the compensation drive described below may also be applied to the surgical robot system (1) according to the embodiment of FIG. 1.

[0095] FIG. 3 is an explanatory diagram from the perspective of operation of a surgical robot system (1') according to the present embodiment. FIG. 4 is an explanatory diagram of a flexible endoscope part (170) of a surgical robot system (1') according to the present embodiment. FIG. 4 is an example of a flexible endoscope part (170) including a shooting part (220, FIG. 15) as an example of a medical device (100). Referring to FIG. 3 and FIG. 4, the surgical robot system (1') can be applied to a surgery in which a flexible endoscope part (170) is inserted into the human body.

[0096] In one embodiment, the surgical robot system (1') can be applied to retrograde intrarenal surgery (RIRS) to crush stones around the kidney (21) by inserting an overtube (200) through the urethra (27). The surgical robot system (1') can be applied to crushing or extracting at least one of kidney stones, renal pelvis stones, and ureteral stones.

[0097] Here, the subject (50) is the part that is the purpose of the surgery, and may be a part of an organ, a target for crushing or external extraction such as a stone, or a mucous membrane, inner wall, or structure at a specific point inside the body. The movement of the subject (50) may include human factors such as internal respiration and heartbeat, or mechanical factors. In one embodiment, the primary movement compensation of the surgical robot system (1') may mean compensation for the movement of the subject (50) caused by respiration, and compensation for the movement of the subject (50) caused by other human factors or mechanical factors may be treated as incidental.

[0098] The surgical robot system (1') may include a monitoring unit (110) in which an image captured through a shooting unit (220) is displayed, or an operating unit (130) in which a doctor operates a driving unit (150) through means including a joystick, a handle, a gripper, etc. The monitoring unit (110) and the operating unit (130) may be integrated into a display device (53) as in the embodiment of FIG. 2.

[0099] The driving unit (150) may include at least one of a translational driving unit (151), a bending driving unit (153), and a rotational driving unit (155), and the object of the driving may be at least one of an overtube (200), an imaging unit (220), and a surgical tool unit (240). The imaging unit (220) or the surgical tool unit (240) may be extended along the length inside the overtube (200) in a tubular shape.

[0100] In one embodiment, the motion compensation device (100) may include a master station (MS) or a slave station (SS). The master station (MS) may refer to the aforementioned operator console (5), and the slave station (SS) may refer to the surgical instrument device (10).

[0101] The master station (MS) allows the surgeon performing the operation to monitor the surgical situation and operate at least one of the flexible endoscope unit (170), overtube (200), imaging unit (220), and surgical tool unit (240). The slave station (SS) may be operated by receiving instructions from the surgeon from the master station (MS) or by operating through a separate control unit that can be linked with other display means.

[0102] The master station (MS) or slave station (SS) may be implemented independently or as a single unit, depending on the surgical site or component layout. The monitoring unit (110) or the operation unit (130) may be included in the master station (MS), and the drive unit (150), the flexible endoscope unit (170), or the robot arm (140) may be included in the slave station (SS).

[0103] The flexible endoscope (170) can be mounted on a slave station (SS) including a robot arm (140), and can be adjusted to a height or angle suitable for surgery by the robot arm (140).

[0104] The flexible endoscope (170) may include a flexible tube-type overtube (200) that is inserted into the human body (S110). The overtube (200) may be composed of a plurality of joints having degrees of freedom under certain constraint conditions, and as a result, the end (end tip) of the overtube (200) can freely approach the object (50) with three degrees of freedom.

[0105] In one embodiment, the overtube (200) is inserted into the urethra (27) and can reach the kidney (21) or its vicinity through the bladder (25) and ureter (23). As such, the overtube (200) is flexible so as to extend along irregular, winding, or smooth organs and can be moved in three dimensions by the driving unit (150).

[0106] The imaging unit (220) or surgical tool unit (240) may include a wire extending along a path inside the overtube (200) and an end effector protruding from the tip of the overtube (200). In the case of the imaging unit (220), the end effector may be a shooting means such as a camera capable of collecting image data inside the body. The image data collected by the imaging unit (220) may comprehensively include not only two-dimensional data or three-dimensional data including RGB-D or stereo, but also data mounted on the overtube (200) capable of obtaining body information, such as Lidar, IR, sonar, etc.

[0107] In the case of the surgical tool part (240), the end effector may be a surgical means. The surgical means may include at least one of a crushing means capable of crushing the target object (50), such as a laser; a collecting means, such as a basket, for collecting the target object (50) that needs to be extracted from the body; and a suction means for spraying water onto the target object (50) or suctioning the target object (50).

[0108] The surgical tool unit (240) can apply treatment to the subject (50). Here, 'treatment' may include all actions that can be applied to the subject (50) during surgery, excluding the acquisition of image data by the imaging unit (220), etc. Accordingly, the treatment may include crushing means, collection means, or suction means by the surgical means described above.

[0109] The portion of the flexible endoscope (170) connected to the slave station (SS) or the driving unit (150) may be called the proximal portion, and the tip portion of the flexible endoscope (170) connected to the imaging means or surgical means may be called the distal portion. The proximal portion or the distal portion may be used to refer to the upstream or downstream direction, respectively, along the path extending along the overtube (200). The flexible endoscope (170) may be provided with a handle portion (180) in the proximal portion.

[0110] The handle portion (180) may include an insertion means into which a shooting portion (220) or a surgical tool portion (240) can be inserted, or an operating means capable of operating at least one of the overtube (200), the shooting portion (220), and the surgical tool portion (240). For example, the shooting portion (220) may be provided integrated with the overtube (200), and the surgical tool portion (240) may be inserted or replaced through the handle portion (180).

[0111] The surgical robot system (1') can perform movement compensation including compensation for movement of the subject (50) caused by breathing using image data obtained from the imaging unit (220).

[0112] The surgical tool part (240) equipped with a laser, basket, etc. may be replaced through the insertion means of the handle part (180) as the surgery progresses, or multiple surgical tool parts (240) may be provided at a thickness allowed in the insertion environment.

[0113] The operating means of the handle part (180) is provided separately from the drive part (150) and can serve to assist the drive part (150).

[0114] The access sheath (172) may be provided at the entrance of the urethra and may guide the insertion of the overtube (200) into the human body. Without the access sheath (172), damage due to internal friction is likely to occur during the insertion or withdrawal of the overtube (200). In particular, according to the surgical robot system (1'), the surgical means or surgical tool part (240) may be operated in response to the movement of the subject (50) to compensate for the movement.

[0115] FIG. 5 is a configuration diagram of a motion compensation method according to an embodiment of the present invention.

[0116] Referring to FIG. 5, the motion compensation method may include a preparation step (S100), a data collection step (S200), a compensation step (S30), and a surgery step (S400). The preparation step (S100) may include a step of inserting an overtube into the body (S110) and a step of the tip of the overtube arriving at the location of the target object (S130).

[0117] The data collection step (S200) may include an image data collection step (S210), an external data acquisition step (S220), and a navigation generation (S230) step.

[0118] The compensation step (S30) may include a breathing compensation calculation step (S310), a subsequent synchronization step (S330), and a feedback step (S350). The breathing compensation calculation step (S310) may include a breathing cycle calculation step (S311) and a motion estimation modeling step (S313). The synchronization step (S330) may include a translational motion compensation step (S330) and an additional motion compensation step (S333).

[0119] The surgical step (S400) can be performed while maintaining the same distance from the subject (50) as the medical device (100) synchronizes with the movement of the living body and performs movement compensation driving. The data collection step (S200) and the compensation step (S300) are explained through FIGS. 6 to 14. The synchronization step (S330) is explained through FIGS. 15 to 21.

[0120] FIG. 6 is an explanatory diagram of the movement of an overtube (200) or a surgical tool part (240) in a medical device (100) according to an embodiment of the present invention. FIG. 7 is an explanatory diagram for explaining the movement compensation calculation of a compensation calculation module (31, FIG. 15) according to an embodiment of the present invention.

[0121] Referring to FIG. 6, the surgical tool portion (240) of the overtube (200) can perform translational, bending, or rotational movements, and through at least some of these or a combination thereof, the surgical tool portion (240) can perform all three degrees of freedom of movement. From this, the tip of the overtube (200) can pass through a winding organ and approach the target (50) in a desired direction.

[0122] Although the surgical tool part (240) is illustrated as translating, bending, or rotating (rolling), depending on the structure, the translation, bending, or rotation of the overtube (200) is transmitted to the surgical tool part (240), and consequently, the translation, bending, or rotation of the surgical tool part (240) may be implemented.

[0123] The image or video captured by the shooting unit (220) or the shooting means may be determined according to the direction in which the leading edge of the overtube (200) is facing. Image data including a two-dimensional image of the shooting unit (220) may be implemented as a coordinate system consisting of an axis (z-axis) parallel to the direction in which the shooting unit (220) is facing and a plane (xy-plane) perpendicular to that axis. This coordinate system of the two-dimensional image of the shooting unit (220) may be a relative coordinate system that changes according to the position of the leading edge of the overtube (200).

[0124] In this embodiment, the purpose of the surgical robot system (1') may be to maintain the change in the distance between the surgical tool part (240), such as a laser or basket, which may be provided at the tip of the overtube (200), i.e., the distal part, and the object (50), or between the overtube (200) and the object (50), at a level below a certain limit, or to maintain the distance within a predetermined range.

[0125] In most cases, the purpose is to maintain the distance between the surgical tool part (240) and the object (50) within a predetermined range. However, in cases where the surgical tool part (240) is detached and at least some of the object (50) is sucked using an empty tube (or empty channel) rather than a suction means of the surgical tool part (240), it may be necessary to maintain the distance between the overtube (200), the tip of the overtube (200), and the object (50) within a predetermined range.

[0126] This function can be performed through the compensation calculation module (31) in this embodiment, and the compensation calculation module (31) can ensure that the distance between the surgical tool part (240) and the object (50), or the distance between the overtube (200) and the object (50), is maintained within a predetermined range. The compensation calculation module (31) can determine that there is normal movement compensation if the distance between the surgical tool part (240) and the object (50), or the distance between the overtube (200) and the object (50), is maintained within a predetermined set range, and if it is determined that it deviates from the predetermined set range, the movement compensation can be recalculated.

[0127] Unless otherwise specifically distinguished or mentioned in this specification, maintaining a predetermined range of spacing between the surgical tool part (240) and the object to be treated (50), and maintaining the spacing between the overtube (200) and the object to be treated (50) within a predetermined range are common objectives of the present invention.

[0128] The distance between the surgical tool part (240) and the object (50) can have all three-dimensional components as a vector concept, and the movement of the object (50) has regularity due to its characteristics and can have repetitive movement along a specific direction. At this time, the position of the object (50) attached to the body, such as a stone, can be assumed to follow body movements such as breathing. For example, if the position of the object (50) changes or shows irregular movement with each breath, it may be necessary to identify additional movements to compensate for it.

[0129] The main direction of movement or the direction of movement compensation of the object (50) may vary depending on the position of the object (50) or the position of the overtube (200).

[0130] Therefore, if movement compensation is provided so that the direction of the tip of the overtube (200) matches the specific direction movement of the object (50) as much as possible, it can have the effect of offsetting most of the movement of the object (50).

[0131] The motion compensation function provided by the surgical robot system (1') may be intended to provide convenience to the user, who is a medical professional, during the surgical process. Even if the motion compensation function does not completely cancel out the movement of the subject (50), if it compensates for and cancels out the most dominant direction of movement of the surgical tool unit (240), it can have the effect of canceling out most of the movement of the subject (50) so that the distance between the subject (50) and the surgical tool unit (240), the distance between the subject (50) and the overtube (200), or the distance between the subject (50) and the imaging unit (220) is well maintained within a predetermined range. The compensation calculation module (31) may have an additional function to make the movement compensation direction of the surgical tool unit (240), the overtube (200), or the imaging unit (220) match or parallel to the most dominant direction of movement of the subject (50).

[0132] The direction of translational movement is one example of the main direction of movement of the subject (50), and the process of movement compensation (or main direction of movement) may vary depending on the relative visual condition or position of the overtube (200) located within an organ such as a kidney.

[0133] When described in terms of the relative coordinate system of the imaging unit (220), if the surgical tool unit (240) compensates for movement along the translational motion direction (z-axis direction) in response to the movement of the subject (50) (S331), the largest movement of the subject (50) can be offset. The remaining degrees of freedom along the plane perpendicular to the translational motion direction (xy plane) can be further compensated for by bending motion or rotational motion (S333). That is, if the movement along the z-axis, which is the translational motion direction, is compensated for and offset, only movement along the plane composed of the x-axis and y-axis will be visible.

[0134] Additional movement compensation (S333) may include compensation for movement caused by mechanical factors, such as wire tension, in addition to human factors, such as friction, restraint, and elastic deformation of tissues caused by soft internal tissues or organs, excluding the main direction of movement. The direction of additional movement compensation may vary depending on the position of the object (50) or the position of the overtube (200).

[0135] FIG. 8 illustrates the relative movement of a subject (50) to a medical device (200) according to an embodiment of the present invention. FIG. 9 is an explanatory diagram of an orientation according to an embodiment of the present invention. FIG. 10 is an explanatory diagram in which, in an embodiment of the present invention, the object of movement compensation or the driving object of movement compensation according to the movement of the subject (50) is a surgical tool part (240). FIG. 11 is an explanatory diagram in which, in an embodiment of the present invention, the object of movement compensation or the driving object of movement compensation according to the movement of the subject (50) is an overtube (200).

[0136] FIG. 8 shows that the tip of the over tube (200) has reached the target object (50), such as a calculus (S130). FIG. 10 and FIG. 11 show that movement compensation is performed so that the movement between the surgical tool part (240) or the over tube (200) and the target object (50) is mutually synchronized.

[0137] The over tube (200) can be inserted into the body (S110) and reach the target object (50) (S130) through translation, bending, or rotational movement. The image data may include image data such as photographs or videos that can be monitored by a doctor while performing surgery.

[0138] At this time, the imaging unit (220) can collect image data at a fixed position. Here, fixation may mean stopping with respect to an absolute coordinate system associated with external surgical equipment, such as a master station (MS) and a slave station (SS), independent of the movement of the human body.

[0139] In order to maintain the distance between the surgical tool part (240) and the subject (50) within a predetermined range by synchronizing the surgical tool part (240) with the dynamic movement of the subject (50) over time and compensating for the movement, an absolute coordinate system that can serve as a reference for the movement of the subject (50) is required. Unlike conventional endoscopic surgery, robot-assisted endoscopic surgery has the advantage of ensuring reliable position fixation of the end effector at the end of the over tube (200), which consequently leads to accurate movement compensation.

[0140] When the tip of the surgical tool part (240) or the over tube (200) reaches the target point and is fixed or stopped, as shown in FIG. 8, the distance (L+α) between the surgical tool part (240) and the target (50) also changes as much as the target moves (α).

[0141] In FIG. 10 or FIG. 11, the movement compensation of the present invention is applied so that the change in distance between the surgical tool part (240) and the subject (50) is maintained below a certain value or within a predetermined range (L-β to L+β, where β is the error).

[0142] Since the direction of movement may vary depending on the position of the subject (50) even within the same organ, such as the kidney, the viewing angle of the imaging unit (220) needs to be changed when the subject (50) changes, and the magnitude of the movement that needs to be compensated for, the degree of movement observed, and the direction of movement may change accordingly.

[0143] If the viewing direction of the imaging unit (220) at the initial position (P1) where the over tube (200) approaches the subject (50) to collect data is significantly different from the direction of movement (α) of the subject (50), then an orientation change in the imaging direction of the imaging unit (220) may be required.

[0144] Referring to FIG. 9, if the viewing direction of the imaging unit (220) at the initial position (P1) is slightly misaligned with the movement direction (α) of the subject (50), it can be moved to a modified position (P1') to align the viewing direction with the movement direction (α). In this case, a conversion of the relative coordinate system may occur from the coordinate system (A) (x1 axis, y1 axis, z1 axis) of the initial position (P1) to the coordinate system (A') (x2 axis, y2 axis, z2 axis) of the modified position (P1') by orientation adjustment. The z-axis direction of the modified coordinate system (A') may be parallel to the movement direction of the subject (50). The absolute coordinate system (x-axis, y-axis, z-axis) may be embedded or integrated into external equipment of the surgical robot system (1') or compensation calculation module (31) that does not change according to movements such as breathing, or it may be a coordinate system that is converted to move together with the subject (50).

[0145] The relative coordinate values ​​for the shooting unit (220) or the subject (50) that moves according to breathing, etc., can be accurately calculated by comparing the absolute coordinate system and the relative coordinate system (A or A').

[0146] FIGS. 10 and FIGS. 11 illustrate embodiments of motion compensation of the invention. FIG. 10 may show a surgical tool part (240) that is motion-compensated to correspond to the movement (α) of the object (50). FIG. 11 may show an over tube (200) that is motion-compensated to correspond to the movement (α) of the object (50).

[0147] When the surgical tool part (240) is driven by compensation, the risk of damage to the organ surface, etc. caused by the over tube (200) can be reduced. Additionally, minimizing movement on the organ surface due to the irregular internal structure of the organ can help reduce the accumulation of over tube (200) control errors.

[0148] When the over tube (200) is driven for compensation, a control step for issuing subsequent control commands to end effectors, such as the imaging unit (220) or surgical tool unit (240), can be made clear during the process of proceeding with surgery after movement compensation.

[0149] When the surgical tool unit (240) is driven by compensation, the surgical tool unit (240) continues to perform translational movements, and subsequent commands such as crushing or capturing may be issued in a superposition manner, or additional compensation operations may be performed. On the other hand, when the over tube (200) is driven by compensation, the imaging unit (220) or the surgical tool unit (240) may be fixed in position on the over tube (200), and there is an advantage that if only the over tube (200) is compensated for movement, the remaining imaging unit (220) or the surgical tool unit (240) can automatically obtain a movement compensation effect.

[0150] The imaging unit (220) may have various arrangement structures, such as being fixed in position on the over tube (200), protruding from the tip of the over tube (200) like the surgical tool unit (240) to enable operation, or being implemented to be integrated with the surgical tool unit (240).

[0151] The shooting unit (220) can be the target of motion compensation or the driving target of motion compensation according to the movement of the target object (50).

[0152] Depending on the object of motion compensation or the driving object of motion compensation according to the movement of the subject (50), the installation location of the imaging unit (220) or the arrangement structure between the over tube (200), the imaging unit (220), and the surgical tool unit (240), the image displayed to the doctor through the monitoring unit (110) may appear almost stationary (first image) synchronized with the subject (50), or the subject (50) may appear to be moving (second image).

[0153] In either case of the two types (first image and second image) that are mutually stopped or mutually operated, the distance between the surgical tool unit (240) and the subject (50) remains constant, so one of the two types may be displayed manually or automatically depending on the surgeon's preference or the situation during surgery. Depending on the surgeon's choice, an option may be provided for the two types of images to be switched between each other. Depending on the mutual arrangement structure between the over tube (200), the imaging unit (220), or the surgical tool unit (240), the first image or the second image may be switched between through image data compensation.

[0154] When the movement of the surgical tool unit (240) is compensated (Fig. 10) and the imaging unit (220) is fixed in position on the over tube (200), the image displayed to the doctor through the monitoring unit (110) may appear as if the subject (50) is moving in accordance with the breathing cycle, breathing direction, etc. For the subject (50) to be displayed as fixed on the monitoring unit (100), additional image-based calculations may be required.

[0155] When the movement of the surgical tool unit (240) is compensated (Fig. 10) and the imaging unit (220) is not fixed to the over tube (200) and can operate separately, the imaging unit (220) and the subject (50) can be synchronized through additional synchronization between the imaging unit (220) and the surgical tool unit (240) so that the image displayed to the doctor through the monitoring unit (110) appears as if the subject (50) has stopped.

[0156] When the movement of the surgical tool unit (240) is compensated (Fig. 10) and the imaging unit (220) is implemented to be integrated with the surgical tool unit (240), the image displayed to the doctor through the monitoring unit (110) may appear as if the subject (50) has stopped. When crushing or capturing is performed while the movement of the surgical tool unit (240) is compensated, the subject (50) will remain in a stopped state, but if there is additional compensation or additional movement by the doctor's operation of the surgical tool unit (240), the monitored screen may change.

[0157] When the movement of the over tube (200) is compensated (Fig. 11) and the shooting unit (220) is fixed in position on the over tube (200), the image displayed to the doctor through the monitoring unit (110) may appear as if the subject (50) has stopped.

[0158] When the movement of the over tube (200) is compensated (Fig. 11) and the imaging unit (220) is not fixed to the over tube (200) and can operate separately, additional synchronization between the imaging unit (220) and the over tube (200) may be required so that the image displayed to the doctor through the monitoring unit (110) appears as if the subject (50) has stopped.

[0159] When the movement of the over tube (200) is compensated (Fig. 11) and the imaging unit (220) is implemented to be integrated with the surgical tool unit (240), the image displayed to the doctor through the monitoring unit (110) may appear as if the subject (50) has stopped. When crushing or capturing is performed while the movement of the surgical tool unit (240) is compensated, the subject (50) will remain in a stopped state; however, if there is additional compensation for the surgical tool unit (240) or additional movement caused by the doctor's operation, the monitored screen may change.

[0160] In addition, the over tube (200) and the surgical tool part (240) may not only be individually compensated for movement, but both may also be compensated for movement and operate in conjunction with each other. That is, through the coordinated operation of the over tube (200) and the surgical tool part (240), the distance between the object (50) and the surgical tool part (240), or the distance between the object (50) and the over tube (200), can be maintained within a predetermined range.

[0161] FIG. 12 is an explanatory diagram of the first to fourth data flows according to an embodiment of the present invention. FIG. 13 is an explanatory diagram of data collection or calculation of breathing movements by the imaging unit (220) according to an embodiment of the present invention.

[0162] Referring to FIGS. 7, 12, and 13, an image captured by the imaging unit (220) is represented in the relative coordinate system of the imaging unit (220). For example, the subject (50) may repeatedly approach or move away from the imaging unit (220) according to a breathing cycle consisting of exhalation and inhalation.

[0163] Figure 13 is for breathing compensation as an example of movement compensation.

[0164] During inhalation (B1), the subject (50) may approach the imaging unit (220), and during exhalation (B2), the subject (50) may move away from the imaging unit (220). A pattern of respiration can be identified from the in vivo movement of the images or video of the imaging unit (220) collected according to time series. The waveform pattern for the pressure or volume of respiration may include at least one of an exhalation section or exhalation peak (B31, B32), an inhalation section or inhalation peak (B33, B34), and a functional residual volume section or peak (B35, B36), which is the amount of exhaust that is not completely expelled. The compensation calculation module (31) can correspond the features of these respiration waveform patterns from the images or video of the imaging unit (220) and can calculate respiration compensation for the subject (50) due to respiration, including the respiration cycle, respiration direction (movement direction), etc.

[0165] The compensation calculation module (31) of the present invention can calculate movement information including the distance between the surgical tool part (240) and the subject (50), or the movement time of the subject (50), including the breathing cycle, by using image data such as a two-dimensional type obtained from a shooting part (220) provided at the end of the over tube (200) according to a predetermined time interval.

[0166] Referring to FIG. 7, the compensation calculation module (31) can set a target part (TP) or a feature part (FP) on the captured image of the shooting unit (220). The target part (TP) may be a part that serves as a reference on the screen, such as a target object (50), and the feature part (FP) may measure or represent the change in position of the target part (TP) on the captured image over time.

[0167] The compensation calculation module (31) can calculate the movement of the target object (50) by tracking the target part (TP) or feature part (FP) from the image captured according to a predetermined time interval (e.g., 100ms).

[0168] According to FIG. 12, image data captured by the capturing unit (220) can be transmitted to the data processing unit (310) or the motion estimation modeling unit (500).

[0169] The data processing unit (310) may be provided separately from the motion estimation modeling unit (500) or may be included in the motion estimation modeling unit (500), and may perform the same function as when provided separately.

[0170] The data processing unit (310) can transmit data received from the motion estimation modeling unit (500) or data with a converted data format.

[0171] The motion estimation modeling unit (500) can perform motion compensation in specific scenarios or situations using data from a motion estimation model even when there is no input of image data from the shooting unit (220) or external data from the measurement unit of an external medical device. Additionally, the motion estimation modeling unit (500) can perform motion compensation using data that has been input in advance.

[0172] The compensation calculation module (31) may selectively receive data from the data processing unit (310) or the motion estimation modeling unit (500), or receive data from both.

[0173] Image data captured by the capturing unit (220) may include first data (D1) used to compensate for movement of the subject (50), such as breathing compensation, or second data (D2) that is continuously captured in real time by the capturing unit (220).

[0174] The first data (D1) may include movement information of the object (50) required for the movement compensation calculation of the compensation calculation module (31).

[0175] The first data (D1) may include a movement cycle including a breathing cycle used by the compensation calculation module (31) to calculate (S310) the movement compensation of the over tube (200) or surgical tool part (240), or information such as the target part (TP) and feature part (FP).

[0176] The second data (D2) may be image data that is continuously captured in real time by the imaging unit (220). The second data (D2) may conceptually overlap with at least some of the first data (D1) and may be used as real-time image data collected by the imaging unit (220) to monitor the subsequent surgical process after the calculation of motion information including a motion cycle.

[0177] The second data (D2) can be used alone to calculate the motion compensation.

[0178] The compensation calculation module (31) can obtain the second data (D2) captured in real time by the shooting unit (220).

[0179] The compensation calculation module (31) can operate the over tube (200) or surgical tool unit (240) through the driving unit (150) to compensate for the movement of the subject (50) in real time by calculating the image captured at a predetermined time interval by the shooting unit (220) using the second data (D2) through an image processing technique or machine learning model including depth information or optical flow.

[0180] The compensation calculation module (31) can calculate the direction in which the amount of optical flow is minimized using the second data (D2) observed in real time from the imaging unit (220), and can adjust each joint part of the overtube (200) in real time in the calculated direction.

[0181] When the breathing cycle is calculated using the first data (D1), the distance between the surgical tool part (240) and the subject (50) according to the breathing cycle is calculated, and the translational movement of the overtube (200) or the surgical tool part (240) is compensated (S331), and then the movement compensation is performed in real time using the second data (D2) alone without the need for a series of additional measures to compensate for the remaining movement (S333), the overtube (200) or the surgical tool part (240) can be controlled to compensate for the movement of the subject (50) in real time.

[0182] The data processing unit (310) can receive third data (D3), which is external data received from an external measurement unit (600) rather than the motion compensation device (100) including the shooting unit (220). The compensation calculation module (31) can calculate motion compensation including breathing compensation using data collected or stored in the data processing unit (310) (S310).

[0183] The compensation calculation module (31) may use at least one of the first data (D1) to the fourth data (D4) to compensate for the movement of the object to be compensated (50).

[0184] The compensation calculation module (31) can calculate movement information including breathing cycles, etc. (S311) to calculate movement compensation such as breathing compensation (S310).

[0185] FIG. 14 is an explanatory diagram of an embodiment of the calculation of breathing movements, such as a breathing cycle, in a compensation product module (31) according to an embodiment of the present invention. Referring to FIG. 14, when the tip of the over tube (200) arrives at the position of the object to be treated (S130), the position is fixed and data collection by the imaging unit (220) can begin (S200).

[0186] The camera unit (220) can take a primary image (initial image) while stationary near the object (50), and can obtain image information (e.g., a differential image) that is highly similar to the primary image from images taken thereafter based on the primary image.

[0187] The acquisition of image information (difference image) with high similarity corresponds to finding the sampling period at which the most similar image appears among the repeated movements of the object (50) observed by the shooting unit (220), or to finding the index at which the histogram of the difference image is minimized.

[0188] The compensation calculation module (31) can set the difference between the time of the image with the smallest sum of the acquired difference images and the first captured image as the standard for breathing, and then normalize the measured value within a predetermined range through repeated sampling.

[0189] The motion estimation modeling unit (500) can model the biological motion including kidneys (21), stones (50), etc.

[0190] Movement compensation, such as breathing compensation, is not required to be utilized only in specific locations but must be universally applicable across all situations affected by breathing, and such situations may include movement and precise observation of the affected area. The movement estimation modeling unit (500) can provide the unique movement of the subject (50) using a movement estimation model. To this end, the movement estimation modeling unit (500) may input data into a pre-trained model and use the output value, or may use a pre-entered table or setting value.

[0191] The motion estimation model unit (500) may use at least one of the first data (D1) and second data (D2) obtained from the imaging unit (220), the third data (D3) which is patient information obtainable from an external device such as a ventilator / anesthesia workstation or C-arm, and the fourth data (D4) which is feedback data as input data.

[0192] The motion estimation model unit (500) can calculate at least one of the following based on the input data: information required for motion compensation including three-dimensional position or velocity, a motion reference breathing motion value within the breathing cycle, information on the current motion state of the subject (50), the degree of motion compensation performed, motion information relative to the control input including hysteresis or backlash information based on image data of the shooting unit (220), and the shape of the over tube (200) or the leading edge position information within the subject (50).

[0193] In addition, the motion estimation model can be utilized even when it is difficult to accurately determine the movement of the object (50) within the field of view of the camera unit (220).

[0194] If the first data (D1) or the second data (D2) can be obtained through the imaging unit (220), the value predicted by the motion estimation model can be compared with the existing first data (D1) or second data (D2). If the actual value observed through the imaging unit (220) and the value estimated by the motion estimation model show too large a difference from each other, or if a difference outside a predetermined range is maintained for more than a certain period of time, the parameters implementing the motion estimation model can be updated.

[0195] If there is no video information input through the shooting unit (220) or if the information is unusable, the motion estimation model unit (500) can use the value of the motion estimation model as a compensation value for motion compensation.

[0196] Here, unusable information may occur when the motion information of the subject (50) cannot be confirmed in the shooting unit (220), when feedback on the motion compensation result cannot be fully received, or when it is affected by dangerous situations such as mucosal collision, input of error data, or abnormal situations such as disturbance.

[0197] For example, if the field of view of the imaging unit (220) becomes blurred due to the crushing of the stone (50), or if it is difficult to judge the movement of the subject (50) or the surrounding situation, the compensation calculation unit (400) can receive data from the motion estimation model unit (500) and compensate for the movement as before or according to the prediction of the motion estimation model unit (500) until the field of view becomes clear again.

[0198] When the motion compensation calculation by the compensation calculation unit (400) is completed (S310), the control unit (370) can issue a command to the driving unit (150) to perform synchronization (S330). Synchronization (S330) can first be performed by translational motion compensation by the translational driving unit (151) (S331), and secondarily, if motion compensation is required, compensation for the remaining degrees of freedom of motion excluding translational motion or additional motion compensation due to mechanical factors can be performed (S333).

[0199] Hereinafter, as an embodiment of the synchronization step (S330) for synchronizing the motion compensation described in FIGS. 5 to 14, a biological motion synchronization function that starts the aforementioned motion compensation drive in response to the movement characteristics of the biological body is described.

[0200] FIG. 15 is a configuration diagram of a surgical robot system (1') that performs a biological movement synchronization function according to an embodiment of the present invention. Referring to FIG. 15, the surgical robot system (1') may include a surgical instrument device (10), an operator console (5), and a processor (3).

[0201] The imaging unit (220) can acquire image data of the subject. Here, the subject may be the aforementioned target (50), or a specific tissue or inner wall within the body. The medical device (100) can be inserted into the body to reach the subject in the surgical area. Other configurations of the surgical device (10) may be adapted from the aforementioned embodiments.

[0202] The processor (3) may include a compensation calculation module (31) and a synchronization module (32).

[0203] The compensation calculation module (31) receives a first control command as a signal for the medical device (100) to start movement compensation corresponding to the movement of the subject, calculates the movement compensation of the medical device (100) for the subject, and then outputs a second control command received from the synchronization module (32) to compensate and drive the medical device (100) at the output timing of the second control command. The compensation drive mentioned here may refer to the aforementioned movement compensation drive.

[0204] The compensation calculation module (31) calculates a movement compensation that moves the medical device (100) so that the subject and the medical device (100) form a fixed distance, and can receive the second control command at the time of the second control command calculated by the synchronization module (32). In this case, the movement compensation may be a periodic compensation drive corresponding to the movement of the subject according to the respiration of the living organism.

[0205] The operator console (5) may include an operation unit (130). In this embodiment, the operation unit (130) may output a first control command. The processor (3) may calculate the time of the second control command by applying an output time difference from the time when the operation unit (130) is driven and the first control command is output.

[0206] In this embodiment, the first control command may refer to a systemic signal that directs a compensatory drive to compensate for the aforementioned biological movement. The compensatory drive signal of the medical device (100) corresponding to the movement of the biological body is difficult to implement as an active control signal, and accordingly, the compensatory drive signal may be implemented as a periodic movement drive corresponding to the biological cycle. Therefore, in this case, the timing of directing the periodic movement drive must be appropriately set considering the movement of the biological body.

[0207] The first control command may be a direct operation signal or a systemically generated signal based on a condition. As an example of a direct operation signal, the first control command may be generated by user operation through the operation unit (130). The user may output the first control command by operating the operation unit (130) at a designated timing corresponding to the movement of the living organism while viewing the endoscope screen provided by the imaging unit (220). The output first control command is received by the processor (3), specifically by the compensation calculation module (31). In this embodiment, the first control command may be received by both the compensation calculation module (31) and the synchronization module (32). Alternatively, the synchronization module (32) may receive the first control command through the compensation calculation module (31), and in this case, it may be acceptable to receive only the information at the time when the first control command was received.

[0208] Each user may have a different timing for pressing the button that generates the first control command, and even within the same user, the timing of the generation of the first control command may not be uniform. The input of the active operation signal may be influenced by the user's repeated experience and proficiency. Accordingly, an embodiment of the present invention is proposed to calculate an output time difference regardless of the user's operation time and output a second control command at a time corresponding to the biological movement at a time different from the first control command, so that the medical device (100) is substantially compensated and driven upon receiving the second control command.

[0209] As an example of a systemic generated signal based on a condition, the first control command may be a systemic signal generated when a specific condition is achieved based on image data received by the imaging unit (220). In this example, the specific condition may be when it is determined that the medical device (100) has stopped during the steering process. If the medical device (100) stops for a long time during steering, unnecessary damage may be caused by contact with the inner wall of the body due to movements such as breathing. Accordingly, the first control command may be systemically generated when it is determined that the medical device (100) has stopped for a preset time. The determination of whether the medical device (100) has stopped may be performed by evaluating the quality or similarity of the image in the image analysis unit (321), which will be described later. Therefore, in this case, the compensation calculation module (31) may receive the first control command generated at the time when it is determined that the medical device (100) has stopped in the surgical area from the sequence of image data acquired by the imaging unit (220). Additionally, the synchronization module (32) may receive the first control command directly or through the compensation calculation module (31), and it is acceptable to receive only the timing information of the first control command.

[0210] The synchronization module (32) can calculate the timing of the second control command by reflecting the timing of the first control command and the movement of the biological body. FIG. 16 is a configuration diagram of the synchronization module (32) according to an embodiment of the present invention.

[0211] In this embodiment, it can be understood that the timing of the second control command, which compensates for the medical device (100) to respond to the movement of the subject, is different from the timing of the first control command.

[0212] The synchronization module (32) can estimate the movement characteristics of the living organism based on the image data of the subject and calculate the time of the second control command by applying an output time difference from the time of the first control command based on the estimated movement characteristics of the living organism.

[0213] FIG. 16 is a configuration diagram of a synchronization module (32) according to an embodiment of the present invention. Referring to FIG. 16, the synchronization module (32) may include an image analysis unit (321) and a synchronization unit (322).

[0214] The image analysis unit (321) receives continuous image data of a subject and can evaluate the quality (IQ) or similarity (IS) of the subject in the continuous image data. The continuous image data may be a sequence of image frames.

[0215] The image analysis unit (321) can perform a function to evaluate the quality of the subject or to evaluate the similarity of the subject. The image analysis unit (321) can analyze the movement caused by breathing among the movement characteristics of the living organism by evaluating the quality or similarity of the image frame sequence in which the subject was captured.

[0216] The image analysis unit (321) can extract an image quality indicator. The image quality (IQ) can be defined as a value that quantitatively extracts how much the image shakes. The image analysis unit (321) calculates a difference value by comparing a sequence of consecutive image frames, and can evaluate the image quality (IQ) as lower the greater the difference value.

[0217] The image analysis unit (321) can extract the similarity (IS) of the image. The similarity (IS) of the image can be calculated as a quantitative value of the degree of similarity by comparing two images in an image frame sequence.

[0218] The image analysis unit (321) may be configured such that the output node consists of one scalar value, or the output node is formed with two or more, and can be performed in a manner that extracts feature vectors and compares the differences between each feature vector. When evaluating image quality or image similarity, when using two images among an image frame sequence, the image data of the other sequence can be sequentially compared based on one image data.

[0219] The image analysis unit (321) can be implemented in a multi-channel structure when evaluating both the quality and similarity of the image or when using two or more evaluation methods in parallel, and can output data with merged evaluation information.

[0220] The quality information (IQ) or similarity information (IS) of the image output from the image analysis unit (321) is transmitted to the synchronization unit (322), and the synchronization unit (322) estimates feature points as the output of a model in which biological movement characteristics are learned. The synchronization unit (322) can execute a machine learning algorithm, and in one embodiment, a model of a multi-channel input-based motion timing estimation algorithm may be used. The feature point information estimated by the synchronization unit (322) may be a point representing exhalation or inhalation among the biological movement characteristics.

[0221] The synchronization unit (322) can calculate the output time difference (TF) as the time of the movement of the living organism corresponding to the time of the feature point and transmit it to the compensation calculation module (31).

[0222] The output time difference (TF) calculated by the synchronization unit (322) can be viewed as timing information that causes the second control command (S2) to be output at a time when the time difference (t') from the first control command (S1) is delayed. Accordingly, the synchronization unit (322) can form the timing of the second control command using the information of the output time difference (TF). The compensation calculation module (31) forms the output timing of the second control command using the timing information of the second control command.

[0223] In this embodiment, the time point refers to a specific point based on the time axis, the time difference refers to the time difference between the time points, and the output timing refers to the timing at which a control command is output to the medical device (100) or surgical device (10).

[0224] The compensation calculation module (31) outputs a second control command (S2) to the surgical instrument device (10) or medical instrument (100), and finally, the medical instrument (100) can be compensated and driven at a time delayed from the first control command (S1).

[0225] FIG. 17 is a functional diagram of a synchronization module (32) according to an embodiment of the present invention. FIG. 17 includes a more detailed embodiment of an image analysis unit (321).

[0226] The image analysis unit (321) can estimate the movement characteristics of a living organism by determining the similarity of a second image data having a different sequence based on a first image data among continuous image data of a subject.

[0227] The image analysis unit (321) can execute a machine learning algorithm for image quality evaluation. One or more of various methods such as SSIM, VIF, DISTS, and FSIM can be applied as algorithms for image quality evaluation. As an example, SSIM evaluates quality by calculating the structure, brightness, and contrast of an image, and can sensitively reflect changes in quality between two images, and is suitable for quantitatively evaluating loss, deformation, or motion information of image data.

[0228] In addition, the image analysis unit (321) may have a deep learning model applied, and various methods such as SVR, SVM, HyperIQA, and MUSIQ may be applied as deep learning models.

[0229] In another embodiment, the image analysis unit (321) may use a Convolutional Neural Network (CNN)-based or Autoencoder-based evaluation model and may be configured to enable real-time evaluation by reflecting characteristics sensitive to changes in the image.

[0230] Additionally, the image analysis unit (321) may apply a deep learning model to evaluate the similarity of the image, and various methods such as a siamese network, attention-based LSTM / GRU, or temporal convolutional network may be applied as the deep learning model. The evaluated image quality (IQ) or similarity (IS) data is transmitted to the synchronization unit (322), and the synchronization unit (322) can estimate the feature points of the breathing cycle by learning the image quality (IQ) or similarity (IS) data as a periodic characteristic (G) among the movement characteristics of the living organism. The synchronization unit (322) can accurately detect variations or feature points of the periodic breathing pattern using the evaluation results of the machine learning or deep learning model.

[0231] FIG. 18 is a functional explanatory diagram of a synchronization unit (322) according to an embodiment of the present invention, FIG. 18 (a) is an explanatory diagram of feature points (P1~4) estimated as information for image quality (IQ) evaluation, FIG. 18 (b) is an explanatory diagram of calculating output parallax (TF) using the breathing cycle relationship with feature points estimated as image quality (IQ) evaluation or similarity (IS) evaluation information. In FIG. 18 (b), the x-axis is an axis that converts the difference calculated per frame into a time unit of ms.

[0232] Figure 18(a) illustrates P1 to P4 as examples of feature points to be estimated from respiratory cycle information of periodic characteristics (G). Respiratory cycle information may be received from a separate device that measures the respiratory characteristics of the living organism. Alternatively, respiratory cycle information may be obtained from the information of a ventilator without a separate device, or it may be manually entered by checking the ventilator's information. In another embodiment, a respiratory sensor may be used, and the respiratory sensor outputs measurement information in which the voltage rises (or falls) by outputting a change in chest impedance during inhalation, or the voltage falls (or rises) by changing pressure or impedance in the opposite direction to inhalation during exhalation. In Figure 18(a), the time when inhalation begins may be feature point P1, the time when exhalation begins may be feature point P3, and the time when exhalation is completed may be feature point P4. Depending on the system settings, any one of the feature points P1 to P4 may be estimated, or two or more feature points may be estimated. The synchronization unit (322) may apply a model that estimates the timing of exhalation or an model that estimates the timing of inhalation to estimate the set feature points.

[0233] The synchronization unit (322) may apply one or more various deep learning models, such as Attention-based LSTM / GRU or Temporal Convolutional Network, as an algorithm for estimating feature points. The synchronization unit (322) sets a region of interest (ROI) in image quality evaluation (IQ) or similarity evaluation (IS) based on respiratory cycle characteristics (G) and estimates feature points. Figure 18 (b) illustrates an example in which a feature point P3, which is an inflection point changing from exhalation to inhalation, is estimated as a feature point, and the second control command (S2) time point is formed as the time point of movement of the body corresponding to the P3 feature point.

[0234] The synchronization unit (322) may set the inflection point where respiration changes in the movement of the living organism as a feature point. The inflection point where respiration changes may be the feature points P1 to P4 exemplified in FIG. 18 (a). The feature point may represent a specific point in time as an inflection point where respiration changes from exhalation to inhalation, and the specific point in time may be labeled for learning. The pre-trained model may be a model trained to estimate the feature point by comparing the input data with the learned specific point in time. The feature points P1 to P4 are used to determine the point in time of the second control command (S2), where the point in time may mean the point in time when the movement of the living organism corresponding to the feature point occurs in the respiration cycle (G).

[0235] Referring to FIG. 18(b), the synchronization module (32) can calculate a first time difference (T1), which is the time difference to the movement of the body corresponding to the time point (S1) of the first control command, by utilizing the periodic characteristic (G) among the movement characteristics of the body. The time point (S1) of the first control command is the time point at which the compensation drive is activated systematically, and if this time point is explained in relation to the breathing cycle (G), the next breathing cycle can be the first time difference (T1). Once the first time difference (T1) is derived, the distance from the last point of the first time difference (T1) to the feature point P4 can be the second time difference (T2).

[0236] The synchronization module (32) can calculate the first time difference (T1) based on data within one cycle from the time of the first control command (S1) in the periodic characteristic (G) related to inhalation or exhalation among the movement characteristics of the living organism. That is, according to the present embodiment, the synchronization module (32) can calculate the time of the second control command (S2) from the breathing cycle (G) within one cycle when determining the time of the second control command by estimating the time of feature point P4.

[0237] The synchronization module (32) can use the respiratory parameter information of the living organism as a periodic characteristic among the movement characteristics of the living organism. The respiratory parameters may include the respiratory cycle, respiratory rate per minute, and inspiratory / expiratory ratio.

[0238] In summary, the synchronization unit (322) can calculate the time point at which the movement of the living organism corresponding to the time point of the feature point occurs using frame-based time information, and can calculate the output time difference (TF) by utilizing the parameters of the breathing cycle. In this embodiment, the synchronization module (32) can estimate a feature point (P) associated with inhalation or exhalation in the movement characteristics of the living organism and calculate a second time difference (T2) which is the time difference from the first time difference (T1) to the feature point. The synchronization module (32) can calculate the output time difference (TF) by summing the first time difference (T1) and the second time difference (T2). In this embodiment, the movement of the living organism corresponding to the time point of the feature point may be an inflection point where respiration changes. Therefore, by reflecting the output time difference (TF) calculated by the synchronization unit (322), the time point of the second control command (S2) can be formed into an inflection region where respiration changes in the movement characteristics of the living organism.

[0239] In this embodiment, the synchronization module (32) calculates an output time difference (TF) based on data within one cycle from the time of the first control command (S1) in periodic characteristics related to inhalation or exhalation among the movement characteristics of the living organism, and the output time difference (TF) can be calculated in a range exceeding one cycle from the time of the first control command (S1). In this embodiment, the output time difference (TF) can be formed in a range within one to three cycles of respiration.

[0240] FIG. 19 illustrates a case in which an output time difference is calculated according to an embodiment of the present invention, where FIG. 19 (a) is a case in which a second control command (S2) is formed within a range of 2 cycles from the time of the first control command (S1), FIG. 19 (b) is a case in which a second control command (S2) is formed within a range of 3 cycles from the time of the first control command (S1), and FIG. 19 (c) is a case in which a second control command (S2) is formed at an arbitrary point.

[0241] FIG. 19 discloses an embodiment in which the second control command (S2) is set at different times in different cycles, taking into account the latency occurring in the system.

[0242] There is a time difference from the occurrence of movement within the body until the same image is displayed on the screen through the camera, endoscope workstation, and operator console (5), and this time difference is called image latency. In addition, the time difference from sending a command from the operator console (5) to the driving unit (150) of the surgical instrument device (10) until the surgical instrument device (10) actually moves is called control latency, and the time delay from the moment the input is received until the result is produced can be collectively referred to as latency. This latency can be calculated separately by reflecting system characteristics. Therefore, when calculating the output time difference (TF) from the time of the first control command (S1) to the time of the second control command (S2) and actually applying it to the robot, this latency must be taken into account and reflected, and depending on this latency, the time of application of the actual compensation drive may differ, as in cases (a), (b), and (c) of FIG. 19. FIGS. 19 (a), (b), and (c) show representative image latency and control latency among various latencies. FIGS. 19 (a) shows the case where the second time difference (T2) is greater than the latency interval, and FIGS. 19 (b) shows the case where the initially calculated second time difference is smaller than the latency interval. In cases where there is a risk that the timing of the second control command (S2) may be formed on the latency interval, one period is added to correct the second time difference (T2), and consequently, the timing of the second control command (S2) can be formed at the time after one period has elapsed. FIGS. 19 (c) shows the case where the latency interval is the same as the second time difference (T2), in which the timing of the second control command (S2) is formed without a period delay.

[0243] FIG. 20 is an explanatory diagram illustrating the calculation of an output disparity (TF) according to another embodiment of the present invention, where FIG. 20 (a) shows feature points (P3, P4) learned from image quality evaluation information, and FIG. 20 (b) is an explanatory diagram illustrating the calculation of an output disparity (TF) using the learned feature points (P3, P4). When calculating an output disparity (TF) to determine the timing of a second control command (S2) based on feature point P3 in the breathing cycle (G) curve, the timing of feature point P3 and the timing of a first control command (S1) may be compared. In this case, the disparity between feature point (P3) and the first control command (S1) can be calculated as a second disparity (T2). Once the second disparity (T2) is calculated, one breathing cycle can be used as the first disparity (T1), and the output disparity (TF) can be calculated as the value of T1 + T2.

[0244] FIG. 21 is a schematic diagram of a bio-motion synchronization method according to an embodiment of the present invention.

[0245] The medical device (100) can be moved to the surgical area to check biological movement. If the movement visible on the screen in the image received in real time through the imaging unit (220) is large, respiratory characteristic parameters can be received from an external respiratory measuring device, such as a ventilator. A step of adjusting the setting value of the respiratory compensation function by referring to the respiratory characteristic parameters received from the ventilator can be performed, and subsequently, a biological movement synchronization method can be executed.

[0246] A method for synchronizing biological motion may include: (a) receiving image data of a subject acquired by a capturing unit; (b) receiving a first control command as a signal for a medical device inserted into the body and reaching the subject in a surgical area to begin motion compensation corresponding to the movement of the subject; (c) estimating the movement characteristics of the body based on the image data of the subject and calculating the motion compensation of the medical device for the subject; and (d) calculating the timing of a second control command by applying an output time difference from the timing of the first control command based on the movement characteristics of the body. An output timing is formed at the calculated timing of the second control command, so that the medical device can be driven by compensation.

[0247] Steps (a) through (d) may be steps performed in the aforementioned processor (3). Step (b) may be a step in which the compensation calculation module (31) receives a first control command generated by the operation part (130) of the operator console (5) or by the aforementioned specific systemic condition. Step (c) may be a step performed in the compensation calculation module (31). Step (d) may be a step performed in the synchronization module (32).

[0248] Although the technical concept of the present invention and disclosure has been explained by the embodiments described above, the technical concept of the present invention includes various substitutions, modifications, and changes that can be made within the scope of understanding of those skilled in the art to which the present invention pertains. Furthermore, it should be understood that such substitutions, modifications, and changes may be included within the scope of the appended claims.

[0249] [Explanation of the symbol]

[0250] 1, 1': Surgical robot system 3: Processor

[0251] 5: Operator Console 10: Surgical Instrument Device

[0252] 13: Manipulator 20: Anatomical structures

[0253] 21: Kidney 23: Ureter

[0254] 25: Bladder 27: Urethra

[0255] 31: Reward Calculation Module 32: Synchronization Module

[0256] 50: Object, subject 51: Input device

[0257] 53: Display device 100: Medical device

[0258] 110: Monitoring unit 130: Control unit

[0259] 140: Robot arm 150: Drive unit

[0260] 151: Translational drive unit 153: Bending drive unit

[0261] 155: Rotary drive unit 172: Access system

[0262] 180: Handle section 200: Overtube

[0263] 220: Camera Department 240: Surgical Instrument Department

[0264] 310: Memory 321: Image Analysis Unit

[0265] 322: Synchronization Unit 330: Orientation Department

[0266] 350: Navigation Unit 500: Motion Estimation Modeling Unit

[0267]

[0268] The present invention can be used in a surgical robot system in which a medical device can be compensated and driven to correspond to the movement of a subject captured in the surgical area.

Claims

1. In surgical robot systems, A shooting unit that acquires image data of a subject; A medical device inserted into the body to reach the subject in the surgical area; A compensation calculation module that receives a first control command as a signal to initiate motion compensation corresponding to the movement of the subject, and forms an output timing at the time of the second control command below to drive the medical device in compensation; and A synchronization module that calculates the timing of the second control command by reflecting the timing of the first control command and the movement of the biological body; A surgical robot system characterized in that the timing of the first control command and the timing of the second control command are different from each other, and the medical device is driven to compensate for the movement of the subject.

2. In Paragraph 1, The above synchronization module is, A surgical robot system characterized by estimating the movement characteristics of a living organism based on image data of the subject, and calculating the timing of the second control command by applying an output time difference from the timing of the first control command based on the estimated movement characteristics of the living organism.

3. In Paragraph 2, The above synchronization module is, A surgical robot system characterized by calculating a first time difference, which is the time difference to the movement of the body corresponding to the time of the first control command, by utilizing the periodic characteristic among the movement characteristics of the body.

4. In Paragraph 3, The above synchronization module is, A surgical robot system characterized by calculating the first time difference based on data within one cycle from the time of the first control command in respiratory characteristics related to inhalation or exhalation among the movement characteristics of the above-mentioned living organism.

5. In Paragraph 3, The above synchronization module is, Among the movement characteristics of the above-mentioned organism, the above-mentioned respiratory characteristics, A surgical robot system characterized by utilizing the above-mentioned biological respiration parameter information.

6. In the above Clause 3, The above synchronization module is, A surgical robot system characterized by estimating a feature point associated with inhalation or exhalation in the movement characteristics of the above-mentioned living organism, and calculating a second time difference, which is the time difference from the first time difference to the feature point.

7. In Paragraph 6, The above synchronization module is, A surgical robot system characterized by calculating the output time difference by summing the first time difference and the second time difference.

8. In Paragraph 6, The above synchronization module is, A surgical robot system characterized by the fact that the inflection point at which respiration changes in the above-mentioned biological movement is set as the above-mentioned feature point.

9. In Paragraph 2, The above synchronization module is, Estimating characteristic points associated with inhalation or exhalation from the movement characteristics of the above-mentioned organism, and The respiratory cycle of the above-mentioned organism is set as the first time difference, A surgical robot system characterized by calculating the time difference from the time of the first control command to the feature point as the second time difference, and calculating the output time difference as the sum of the first time difference and the second time difference.

10. In Paragraph 1, The above compensation calculation module is, A surgical robot system characterized by receiving the first control command generated at the time when it is determined that the medical device has stopped in the surgical area from the sequence of image data acquired by the above-mentioned imaging unit.

11. In Paragraph 2, The above synchronization module is, A surgical robot system characterized by forming the timing of the second control command in the inflection region where respiration changes in the movement characteristics of the above-mentioned living organism.

12. In Paragraph 11, The above synchronization module is, Calculate the output time difference based on data within one cycle from the time of the first control command in respiratory characteristics related to inhalation or exhalation among the movement characteristics of the above-mentioned living organism, and A surgical robot system characterized in that the above output time difference is calculated to be a range exceeding one cycle from the time of the first control command.

13. In Paragraph 1, The above synchronization module is, The above-mentioned imaging unit estimates the movement characteristics of the biological body solely from continuous image data capturing the subject, A surgical robot system characterized by calculating the timing of the above-mentioned second control command through image analysis.

14. In Paragraph 1, The above synchronization module is, A surgical robot system characterized by including an image analysis unit that receives continuous image data capturing the subject and evaluates the quality or similarity of the subject in the continuous image data.

15. In Paragraph 14, The above image analysis unit is, Among the continuous image data capturing the above subject, A surgical robot system characterized by estimating respiratory characteristics in the movement of the living organism by determining the similarity of second image data having a different sequence based on first image data.

16. In Paragraph 1, The above compensation calculation module is, Calculates a movement compensation for moving the medical device so that the subject and the medical device form a fixed distance, and outputs the second control command received from the synchronization module to the medical device. A surgical robot system characterized by the above-mentioned movement compensation being a periodic compensation drive corresponding to the movement of the subject according to the respiration of the above-mentioned living organism.

17. In Paragraph 16, The above synchronization module is, An image analysis unit that receives continuous image data capturing the subject, evaluates the quality or similarity of the subject in the continuous image data, and estimates feature points associated with the inhalation or exhalation of the biological body; and A surgical robot system characterized by including a synchronization unit that calculates an output time difference as the time of the movement of the biological body corresponding to the time of the feature point and transmits it to the compensation calculation module.

18. In Paragraph 1, The above compensation calculation module is, After calculating the movement compensation of the medical device for the above subject, A surgical robot system characterized by outputting the second control command at the time of the second control command received from the synchronization module.

19. In surgical robot systems, A shooting unit that acquires image data of a subject; A medical device inserted into the body to reach the subject in the surgical area; A processor that receives a first control command as a signal to initiate motion compensation corresponding to the movement of the subject, calculates the motion compensation of the medical device for the subject, and outputs a second control command as a signal to drive the motion compensation to drive the medical device; and It includes an operator console having an operating unit that outputs the first control command; The above processor is, A surgical robot system characterized by forming the output timing of the second control command by providing an output time difference from the time when the first control command generated by the operation of the above-mentioned control unit is received.

20. A method for synchronizing the biological movement of a medical device performed in a processor of a surgical robot system, (a) A step of receiving image data of a subject acquired by a camera unit; (b) A medical device inserted into the body and reaching the subject in the surgical area receives a first control command as a signal to initiate movement compensation corresponding to the movement of the subject; (c) a step of estimating the movement characteristics of the living organism based on image data of the subject and calculating the movement compensation of the medical device for the subject; and (d) a step of calculating the timing of a second control command by applying an output time difference from the timing of the first control command based on the movement characteristics of the above-mentioned living organism; and A method for synchronizing the biological movement of a medical device, characterized in that the output timing of the second control command is formed at the time of the second control command calculated above, and the medical device is driven in compensation.

Images