Individualized virtual laparoscopic surgery robot platform and apparatus

Abstract

An individualized virtual laparoscopic surgery robot platform and apparatus, relating to the technical field of medical robots. The platform comprises: an individualized three-dimensional biomimetic model reconstruction module, used for generating an individualized three-dimensional biomimetic model on the basis of clinical medical imaging data; a mechanical arm array module, used for simulating operations of laparoscopic surgery; a virtual surgery feedback module, providing real-time force feedback and visual feedback; and an interaction module, used for controlling coordinated cooperation among the individualized three-dimensional biomimetic model reconstruction module, the mechanical arm array module, and the virtual surgery feedback module, controlling, on the basis of operations of the mechanical arm array module, the individualized three-dimensional biomimetic model to update accordingly, and controlling the virtual surgery feedback module to update visual feedback results.

Description

Personalized Virtual Laparoscopic Surgical Robot Platform and Device Technical Field

[0001] This application relates to the field of medical robot technology, specifically to a personalized virtual laparoscopic surgical robot platform and device. Background Technology

[0002] Laparoscopic surgery, as a minimally invasive surgical procedure, offers advantages such as minimal trauma and rapid recovery. However, the complexity of the surgery demands highly precise surgical planning and training. Existing simulators for laparoscopic surgical planning and training lack individualization capabilities, only capable of simulating operations within the same abdominal cavity setting. They cannot be personalized based on the patient's specific anatomy and pathological characteristics, meaning that training and planning may not fully simulate the complexities encountered in actual surgery, limiting the simulator's potential to improve surgical accuracy and safety. Furthermore, many existing simulators lack real-time feedback mechanisms, failing to simulate the impact of surgical procedures on the patient's internal environment. This feedback is crucial for training surgeons to make rapid decisions and adapt to unforeseen circumstances during surgery. The inability to provide individualized real-time feedback and sufficient interactivity limits their application in surgical training and rehearsals.

[0003] Chinese patent application CN112989449A discloses a tactile force feedback simulation interaction method and device with optimized motion stiffness. It simulates two three-dimensional geometric models, including a static model and a moving model, with their initial centroids coinciding. The moving model and the static model undergo continuous collisions. Based on the principle of local contact force optimization, a predetermined vector direction is assigned to each collision and separation between the models. Furthermore, combined with a penetration depth distance calculation method, the minimum distance between the models at each collision and separation is calculated based on the predetermined vector direction. A dynamic control model is constructed that correlates elastic stiffness, damping constant, and the minimum distance. The minimum distance is then substituted into the dynamic control model to obtain the magnitude of the force applied to the moving model each time. By introducing a variable virtual stiffness nonlinear optimization algorithm for different geometric models, stable rigid body model tactile force feedback simulation interaction is achieved. However, this method still has some shortcomings in real-time calculation and feedback, as well as the overall coordination control of the system platform, and cannot meet the needs of virtual laparoscopic surgery simulation.

[0004] Therefore, there is an urgent need to develop a new virtual laparoscopic surgical robot platform to meet the real-time computing and feedback requirements of virtual laparoscopic surgical simulation. Summary of the Invention

[0005] This application aims to at least partially address one of the technical problems in the related art. To this end, this application provides a personalized virtual laparoscopic surgical robot platform and device, whose unique multi-module interaction and system integration strategy enables highly realistic and personalized surgical simulation.

[0006] To achieve the above objectives, in a first aspect, this application provides a virtual laparoscopic surgical robot platform, comprising:

[0007] The personalized 3D bionic model reconstruction module is used to generate personalized 3D bionic models based on clinical medical imaging data.

[0008] The robotic arm array module includes at least three independent robotic arms for simulating laparoscopic surgery operations;

[0009] A virtual surgical feedback module is used in conjunction with the robotic arm array module to provide real-time force and visual feedback;

[0010] The interaction module is used to control the coordinated operation of the individualized 3D bionic model reconstruction module, the robotic arm array module, and the virtual surgery feedback module. It controls the individualized 3D bionic model to be updated accordingly based on the operation of the robotic arm array module, and controls the virtual surgery feedback module to update the visual feedback results.

[0011] Preferably, the robotic arm array module is equipped with an adaptive gain adjustment strategy for torque control and speed control for each robotic arm. The adaptive gain adjustment strategy enables the PID controller of each robotic arm to dynamically adjust the gain parameters according to the real-time error and system status.

[0012] Preferably, the adaptive gain adjustment strategy is expressed as follows:

[0013] Where u(t) is the control input, torque or speed adjustment signal; t is the time point; e(t) is the error between the setpoint and the current position; and K... p (t), K i (t), K d (t) are the proportional, integral, and derivative gain parameters, respectively, with time and system state as independent variables, and are defined as follows: K i (t)=K i0 +a i ·f i (e(t),∫e(t)dt,S(t));

[0014] Among them, K p0 K i0 K d0It corresponds to the initial gain of proportional, integral, and derivative equations; a p a i a d These are the adjustment coefficients corresponding to the proportional, integral, and derivative operations, used to control the amplitude of adaptive adjustment; f p f i f d It is an adaptive function; S(t) represents the current state information and is a parameter of the virtual surgical scene.

[0015] Preferably, the robotic arm array module is equipped with a joint torque sensor at each joint of the robotic arm to capture angle changes, a piezoelectric sensor to capture the force when in contact with the virtual organ, and an accelerometer to monitor the acceleration of the movement. The robotic arm includes at least three rotational degrees of freedom and one linear degree of freedom. The robotic arm array module applies a Kalman filter algorithm to fuse data from different sensors, wherein the update formula of the Kalman filter is: x k|k =x k|k-1 +K k (Z k -H k x k|k-1 );

[0016] x k|k It is the state estimate after the update at the k-th time point, x k|k-1 It is an estimate of the value at the previous time point, Z. k These are observed values, H k It is the observation model matrix, K k It is the Kalman gain, and the sensor data is processed through the observation model matrix H. k It is integrated into the state estimation and used for dynamic system state estimation of the robotic arm's movements.

[0017] Preferably, the robotic arm array module uses a force model calculation formula to calculate the force when the end of the robotic arm comes into contact with the virtual organ. The force model calculation formula is: F=k·Δx;

[0018] Where F is the contact force, Δx is the deformation, and k is the elastic constant, the value of which is dynamically adjusted according to the physical properties of different virtual organs;

[0019] An impedance-controlled force feedback algorithm is used to enable the operator to feel a force that matches the interaction in the virtual environment. The force feedback algorithm formula is expressed as:

[0020] Among them, F d It is the force of expectation, x d , These are the desired position, velocity, and acceleration, x, It is the actual position, velocity, and acceleration, K d B d M d These are the controlled stiffness, damping, and mass parameters.

[0021] Preferably, affine transformation is used to convert the physical coordinate system of the robotic arm to the coordinate system of the virtual environment:

[0022] Where (x, y, z) is the position of the robotic arm's end effector in the physical coordinate system, and (x′, y′, z′) is the transformed position in the virtual coordinate system. The transformation matrix includes rotation angle R, scaling S, and translation T, where t x , t y , t z These represent the translation amounts along the x, y, and z axes, respectively.

[0023] Preferably, the virtual laparoscopic surgical robot platform further includes a system integration module, which uses timestamp and buffer management technology to align data streams from different sources, enabling synchronous processing of data from the individualized 3D bionic model reconstruction module, the robotic arm array module, and the virtual surgical feedback module.

[0024] Preferably, the system integration module deploys GPU-based computing units to support high-speed data processing and complex signal operations;

[0025] The system integration module is equipped with an embedded real-time operating system to manage task priority and scheduling, so as to ensure that high-priority critical tasks are completed within the time limit. High-priority critical tasks include signal processing and feedback control. The scheduling algorithm of the embedded real-time operating system is set to cyclic scheduling.

[0026] Preferably, the individualized 3D bionic model reconstruction module uses the Loop subdivision algorithm to refine the individualized 3D bionic model. Without changing the topological structure of the tissues and organs, it increases the number of vertices and faces to improve the smoothness and detail of the abdominal organ model. Furthermore, it uses the QEM algorithm for local mesh optimization.

[0027] Preferably, the steps for local mesh optimization using the QEM algorithm include:

[0028] The QEM algorithm is improved by introducing a weighting factor, which is set based on the surgical target location, organ importance, and biomimetic material properties.

[0029] For a liver surgery scenario, Q is defined as a triangular mesh of the liver, with vertex set V and edge set E. For each vertex v in V, the error metric is expressed as follows:

[0030] Among them, K i It is the planar error matrix associated with vertex V; w i The weighting factor is w; i represents all nodes associated with vertex v; the formula is a weighted sum of all planar error matrices associated with vertex v, where w is the weighting factor. i External control over mesh refinement is achieved for different surgical scenarios.

[0031] Preferably, the weighting factor w i The setup strategies include: assigning higher weights to the liver and adjacent organs to reduce simplification of these areas; using lower weights for distal organs and vascular tissues to the surgical target to obtain a simpler tissue mesh; and assigning lower weights to soft tissue areas than to organs, based on the differences in the physical properties of organs and soft tissues.

[0032] Preferably, the personalized 3D biomimetic model reconstruction module further includes adding physical parameters based on real biomechanics to the personalized 3D biomimetic model to simulate the real physical behavior of different tissues; the physical parameters include elastic modulus, density, and coefficient of friction, with the elastic modulus of the liver set to 0.5-1.5 kPa and the elastic modulus of muscle tissue set to 8-12 kPa; in performing parameter mapping, the physical parameters are mapped to each mesh cell of the 3D model using a voxelization method, and the corresponding physical parameters are calculated and assigned on the mesh nodes through cubic linear interpolation to ensure the continuity of the physical properties of the personalized 3D biomimetic model throughout the entire volume.

[0033] Secondly, this application provides a personalized virtual surgical operation device, including the virtual laparoscopic surgical robot platform described above.

[0034] Based on the above technical solution, the virtual laparoscopic surgical robot platform and device of this application have at least one of the following beneficial effects compared with the prior art:

[0035] 1. This application sets up an individualized three-dimensional bionic model reconstruction module, a robotic arm array module, a virtual surgery feedback module, and an interaction module. Through a unique multi-module interaction and system integration strategy, it achieves highly realistic and individualized surgical simulation, which can meet the requirements of virtual laparoscopic surgery simulation in real-time calculation and feedback as well as the overall coordination and control of the system platform.

[0036] 2. The robotic arm array module of this application is equipped with an adaptive gain adjustment strategy for torque control and speed control. Through the PID controller with adaptive gain adjustment, the precise control of the robotic arm in complex surgical scenarios is ensured, thereby improving the realism and accuracy of the surgery.

[0037] 3. The interaction module of this application achieves precise mapping and deep coupling between the robotic arm's movements and the individualized 3D bionic model, ensuring a high degree of realism in human-computer interaction. The system integration module uses timestamp and buffer management technology to align data streams from different sources and sets up an embedded real-time operating system, ensuring efficient communication and real-time computing among modules, and improving the overall coordination and stability of the platform.

[0038] 4. This application introduces weighting factors into the traditional QEM algorithm to preserve details of different tissues and organs. These weighting factors can be set based on the surgical target location, organ importance, and material properties. For example, in a liver surgery scenario, the liver can be designated as a critical organ with a higher weighting coefficient; important organs surrounding the liver can be designated as secondary targets with weaker weighting coefficients; and even weaker coefficients can be used for surrounding soft tissues. This significantly optimizes the number of 3D meshes in laparoscopic surgery scenarios, improving the computational efficiency and real-time performance of virtual surgery.

[0039] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description, or may be realized by practicing the application. The purpose and other advantages of this application can be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings. Attached Figure Description

[0040] Figure 1 is a module connection diagram of the personalized virtual laparoscopic surgical robot platform of this application;

[0041] Figure 2 is a module connection diagram of the individualized three-dimensional bionic model reconstruction module in this application;

[0042] Figure 3 is a module connection diagram of the robotic arm array module in this application;

[0043] Figure 4 is a module connection diagram of the virtual surgery feedback module in this application;

[0044] Figure 5 is a module connection diagram of the interaction module in this application;

[0045] Figure 6 is a physical image of the individualized virtual laparoscopic surgical robot platform in this application;

[0046] Figure 7 is a schematic diagram of the personalized virtual laparoscopic surgical robot platform in this application performing virtual surgery. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0048] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention are also intended to include the plural forms unless the context clearly indicates otherwise.

[0049] To address the shortcomings of existing technologies, the purpose of this invention is to provide an individualized virtual laparoscopic surgical robot platform, which aims to achieve highly realistic and individualized surgical simulation through multi-module interaction and system integration strategies, thereby improving the computational efficiency and real-time performance of virtual surgery.

[0050] The basic idea of ​​this invention is to establish an individualized 3D bionic model reconstruction module, a robotic arm array module, a virtual surgical feedback module, and an interaction module. The interaction module controls the coordinated operation of these modules, including updating the individualized 3D bionic model based on the operation of the robotic arm array module, and updating the visual feedback results through the virtual surgical feedback module. This unique multi-module interaction and system integration strategy achieves highly realistic and individualized surgical simulation.

[0051] Example 1

[0052] In order to develop a highly efficient and realistic virtual laparoscopic surgical robot platform, the inventors conducted in-depth research on related technologies and proposed an individualized virtual laparoscopic surgical robot platform.

[0053] Specifically, as shown in Figure 1, a personalized virtual laparoscopic surgical robot platform is provided, including:

[0054] The personalized 3D bionic model reconstruction module is used to generate personalized 3D bionic models based on clinical medical imaging data.

[0055] The robotic arm array module includes at least three independent robotic arms for simulating laparoscopic surgery operations;

[0056] A virtual surgical feedback module is used in conjunction with the robotic arm array module to provide real-time force and visual feedback;

[0057] The interaction module is used to control the coordinated operation of the individualized 3D bionic model reconstruction module, the robotic arm array module, and the virtual surgery feedback module. It controls the individualized 3D bionic model to be updated accordingly based on the operation of the robotic arm array module, and controls the virtual surgery feedback module to update the visual feedback results.

[0058] This application achieves highly realistic and individualized surgical simulation by setting up an individualized 3D bionic model reconstruction module, a robotic arm array module, a virtual surgical feedback module, and an interaction module, and through a unique multi-module interaction and system integration strategy.

[0059] Figure 2 shows the module connection diagram of the personalized 3D bionic model reconstruction module in this application. This module generates personalized 3D bionic models based on clinical medical imaging data. It is responsible for creating accurate 3D static geometric organ models from original personalized 3D CT images, and adding preset dynamic parameters for different tissues and organs to construct 3D dynamic bionic organ models, providing precise targets for subsequent virtual laparoscopic surgery simulations. This personalized 3D bionic model reconstruction module consists of two main parts:

[0060] 1. Individualized 3D geometric model reconstruction

[0061] (1) Data Preprocessing: This provides clear and accurate data input for subsequent image analysis and model reconstruction. The window width and level are adjusted based on the absorption characteristics of the target tissue to CT rays to enhance image contrast. For example, the window width for liver CT scans can be set to 150-200 HU, and the window level to 30-60 HU. Median filtering is used to denoise the images, and the images are then standardized.

[0062] (2) Organ Segmentation: This scheme combines AI-powered automatic segmentation with manual segmentation to delineate human tissues and organs with biological differences. A pre-trained nnU-Net model is constructed using large datasets of clinical CT images as a sample set to perform the initial organ segmentation. Considering that insufficient training samples may lead to reduced accuracy in lesion segmentation, manual correction is used for lesion areas such as tumors to ensure accurate organ segmentation. Post-processing methods such as morphological manipulations (e.g., expansion and erosion) are applied to refine the segmentation results and improve the continuity and smoothness of the edges of abdominal tissues and organs.

[0063] (3) 3D geometric model reconstruction: The Marching Cubes algorithm is used to extract the isosurface of the 3D model. For each voxel, the gray level of the original 3D CT image corresponding to that voxel is used as a threshold to determine whether the voxel is located inside or outside the model surface. Then, vertices are created on the boundaries of the voxels to form a triangular mesh. All meshes are optimized, including removing isolated elements, smoothing surfaces, and simplifying the mesh, to improve the processing efficiency and visual quality of the 3D model and provide high-quality geometric samples for subsequent organ biomimetic models.

[0064] 2. Individualized 3D bionic model reconstruction

[0065] (1) Geometric Model Subdivision and Optimization: The Loop subdivision algorithm was used to refine the geometric model structure, ensuring that the smoothness and detail of the abdominal organ model were improved by increasing the number of vertices and faces without changing the topology of the tissues and organs. The Quadric Error Metrics (QEM) algorithm was used as a local mesh optimization technique to reduce unnecessary details (such as excessive soft tissue area, broken blood vessel reconstruction, etc.) and optimize the computational efficiency of the model. At the same time, the high resolution of the target feature areas (such as lesions, liver, and the area around tumors) was maintained to ensure the accuracy of the laparoscopic surgery simulation.

[0066] To address the need for dynamically adjusting mesh refinement during virtual laparoscopic surgery simulations, this invention employs a weighting factor improvement strategy to enhance the QEM algorithm, achieving detail preservation for different tissues and organs. In traditional QEM, error metrics are typically calculated using a quadratic error matrix of the vertices. To more precisely control detail in specific regions (such as the liver, soft tissue, or blood vessels), weighting factors are introduced. These weighting factors can be set based on the surgical target location, organ importance, and biomimetic material properties, enabling adaptation to various individualized laparoscopic surgical environments. For a liver surgery scenario, Q is defined as a triangular liver mesh with vertex set V and edge set E. For each vertex v in V, the error metric is expressed as:

[0067] Among them, K i It is the planar error matrix associated with vertex V; w i is the weighting factor; i represents all nodes related to vertex v. This formula is a weighted sum of all planar error matrices related to vertex v. The weighting factor w can be used to externally control mesh refinement for different surgical scenarios. In this invention, w i It can be defined based on the following aspects: 1) Give higher weights to the liver and adjacent organs to reduce simplification of these areas; use lower weights for distal organs and vascular tissues of the surgical target to obtain a simpler tissue grid; 2) Set significantly lower weights for soft tissue areas than for organs based on the differences in the physical properties of organs and soft tissues.

[0068] (2) Physical Parameter Addition: Physical parameters based on real biomechanics, such as elastic modulus, density, and coefficient of friction, are added to the 3D biomimetic model to simulate the real physical behavior of different tissues. First, physical parameters for various tissues are defined based on literature and experimental data. For example, the elastic modulus of the liver can be set in the range of 0.5-1.5 kPa, while that of muscle tissue is 8-12 kPa. Parameter mapping is performed, using voxelization to map the physical parameters to each mesh cell of the 3D model. These parameters are calculated and assigned at the mesh nodes using cubic linear interpolation to ensure the continuity of the model's physical properties across the entire volume.

[0069] The Loop subdivision algorithm is a technique in computer graphics used to smooth and refine 3D geometric models. It increases the detail and smoothness of the model by adding new vertices and faces at each vertex. The Loop subdivision algorithm calculates the positions of new vertices, typically based on a weighted average of the original vertex positions and the positions of adjacent vertices. This calculation ensures a smooth transition between new vertex positions, reducing the jaggedness of the mesh. Although the mesh is refined, the model's topology (i.e., how vertices, edges, and faces are connected) remains unchanged. This means the model's basic shape and features are preserved, only the detail and smoothness are enhanced. Applying the Loop subdivision algorithm results in a smoother surface and finer details on an abdominal organ model, helping to more realistically simulate and represent the morphological features of abdominal organs. For surgical simulation and training, this provides a more realistic surgical experience.

[0070] Finally, the constructed 3D geometric model is transmitted to other models, such as the virtual surgery feedback module, via a communication protocol for further processing.

[0071] This application improves the accuracy of organ segmentation by combining AI-based automatic segmentation with manual segmentation techniques. High-quality 3D organ models are provided by optimizing the 3D geometric model using the Marching Cubes algorithm and an improved QEM algorithm.

[0072] Figure 3 shows the module connection diagram of the robotic arm array module in this application. The robotic arm operation array is the core control unit of the robotic arm array, responsible for sending operation commands to control the robotic arm's movements. It includes three independent robotic arms, each with specific degrees of freedom, including at least three rotational degrees of freedom and one linear degree of freedom, used to perform different operations. The rotational degrees of freedom of the robotic arm allow the joints to rotate in a plane, while the linear degrees of freedom allow the joints to move along a straight line, such as performing extension and retraction. A PID controller is used to automatically adjust the robotic arm's movements based on error signals to achieve precise control. A high-precision encoder is used to accurately measure the position and velocity of the robotic arm joints, providing feedback for control. A sensor signal integration module is responsible for collecting and integrating signals from different sensors for further processing and analysis. An angle sensor measures the angle changes of the robotic arm joints, providing position feedback. A force sensor detects the force when the robotic arm comes into contact with the environment or objects, used for force control and collision avoidance. An accelerometer measures the acceleration of the robotic arm joints, contributing to the control of dynamic response and stability. The communication protocol is responsible for transmitting data and instructions between the robotic arm array module and other system modules, ensuring accurate information transmission. The various components of the robotic arm array module work together to enable it to precisely execute complex operations, such as simulating the movements of surgical instruments in virtual laparoscopic surgery. Through feedback from high-precision encoders and multiple sensors, the PID controller can adjust the robotic arm's movements in real time, ensuring smoothness and accuracy. The communication protocol ensures the coordinated operation of the entire system.

[0073] The robotic arm array module is equipped with an adaptive gain adjustment strategy for each robotic arm to perform torque and speed control, ensuring smooth and precise operation. This adaptive gain adjustment strategy enables the PID controller of each robotic arm to dynamically adjust its gain parameters based on real-time errors and system status.

[0074] Specifically, the formula for the adaptive gain adjustment strategy is expressed as follows:

[0075] Where u(t) is the control input, torque or speed adjustment signal; t is the time point; e(t) is the error between the setpoint and the current position; and K... p (t), K i (t), K d (t) are the proportional, integral, and derivative gain parameters, respectively, with time and system state as independent variables, and are defined as follows: K i (t)=K i0 +a i ·f i (e(t),∫e(t)dt,S(t))

[0076] Among them, K p0 K i0 K d0 It corresponds to the initial gain of proportional, integral, and derivative equations; a p a i a d These are the adjustment coefficients corresponding to the proportional, integral, and derivative operations, used to control the amplitude of adaptive adjustment; f p f i f d It is an adaptive function; S(t) represents the current state information and is a parameter of the virtual surgical scene.

[0077] The adaptive gain adjustment strategy introduced in this application allows the robotic arm's PID controller to dynamically adjust the gain parameters based on real-time errors and system status. This means the robotic arm can automatically optimize its performance in different operational phases (such as cutting, knotting, rapid movement, and delicate operations). Simultaneously, the introduced adaptive gain adjustment can accelerate the system's response to errors, especially in situations requiring rapid adjustment (such as emergency avoidance or abnormal robotic arm trajectory). In complex surgical scenarios, real-time adjustment of the gain parameters ensures the system is always in optimal control.

[0078] Preferably, the robotic arm array module is equipped with a joint torque sensor at each joint of the robotic arm to capture angle changes, a piezoelectric sensor to capture the force when in contact with the virtual organ, and an accelerometer to monitor the acceleration of the movement. The robotic arm includes at least three rotational degrees of freedom and one linear degree of freedom. A Kalman filter algorithm is applied to fuse data from different sensors, wherein the update formula of the Kalman filter is: x k|k =x k|k-1 +K k (Z k -H k x k|k-1 )

[0079] x k|k It is the state estimate after the update at the k-th time point, x k|k-1 It is an estimate of the value at the previous time point, Z. k These are observations, containing the actual measurement data from the sensor, H. k It is the observation model matrix, K k It is the Kalman gain, and the sensor data is processed through the observation model matrix H. k It is integrated into the state estimation and used for dynamic system state estimation of the robotic arm's movements.

[0080] Preferably, the robotic arm array module uses a force model calculation formula to calculate the force when the robotic arm end effector contacts the virtual organ. The force model calculation formula is: F=k·Δx

[0081] Where F is the contact force, Δx is the deformation, and k is the elastic constant, the value of which is dynamically adjusted according to the physical properties of different virtual organs;

[0082] An impedance-controlled force feedback algorithm is used to enable the operator to feel a force that matches the interaction in the virtual environment. The force feedback algorithm formula is expressed as:

[0083] Among them, F d It is the force of expectation, x d , These are the desired position, velocity, and acceleration, x, It is the actual position, velocity, and acceleration, K d B d M d These are the controlled stiffness, damping, and mass parameters.

[0084] Figure 4 shows the module connection diagram of the virtual surgical feedback module in this application. This virtual surgical feedback module provides real-time force feedback, simulating the physical interaction between the robotic arm and virtual organs, enhancing the realism of the surgical simulation. It provides high-definition visual feedback, displaying an individualized abdominal cavity environment and virtual organs, enhancing the immersive experience of the surgical simulation. The force feedback unit employs a force-sensing design, with a high-precision piezoelectric sensor mounted at the end of the robotic arm to capture the force when in contact with the virtual organ. The force model calculation formula uses F = k·Δx to calculate the contact force, where F is the contact force, Δx is the deformation, and k is the elastic constant. The value of k can be dynamically adjusted according to the physical properties (such as the elastic modulus) of different virtual organs. An impedance-controlled force feedback algorithm is implemented to ensure that the operator feels a force matching the interaction with the virtual environment. The force feedback algorithm can be expressed as:

[0085] Among them, F d It is the force of expectation, x d , These are the desired position, velocity, and acceleration, x, It is the actual position, velocity, and acceleration, K d B d M d These are the controlled stiffness, damping, and mass parameters.

[0086] The visual feedback module utilizes a medical-grade high-resolution display to ensure image clarity and detail. Advanced image rendering techniques, such as ray tracing or real-time rendering algorithms, are applied to provide realistic visual effects. A graphics processing unit (GPU) is used to accelerate the rendering process, ensuring real-time performance. A user-friendly interface is developed, supporting functions such as surgical records and risk point alerts. Touchscreen or voice recognition technology is used to enhance interface interactivity. The interface design prioritizes usability and accessibility, ensuring all users can quickly understand and operate the system.

[0087] Figure 5 shows the module connection diagram of the interaction module in this application. This module is the central control unit of the system, responsible for coordinating the operations of all other modules, processing signals from the robotic arm's operations and the deformation signals from the bionic model, ensuring accurate information transmission and feedback, and supporting complex data processing and real-time computing requirements. Specifically, a motion mapping algorithm is designed to achieve accurate mapping from the robotic arm's physical movements to the virtual model's movements. Affine transformation is used to convert between the robotic arm's physical coordinate system and the virtual environment's coordinate system. Where (x, y, z) represents the position of the robotic arm's endpoint in the physical coordinate system, and (x′, y′, z′) represents its transformed position in the virtual coordinate system. The transformation matrix includes rotation angle R, scaling S, and translation T. x , t y , t z These represent the translation amounts along the x, y, and z axes, respectively.

[0088] Model coupling is used to achieve deep coupling between the 3D bionic model and the robotic arm array module, and to adjust the deformation of the bionic model in real time according to the motion signal of the robotic arm to achieve highly realistic human-computer interaction. This involves (1) signal capture and processing: each joint of the robotic arm is equipped with a high-precision encoder to capture joint angles and speeds in real time. Kalman filters are used to smooth and predict the motion trajectory of the robotic arm. (2) Model response algorithm: a real-time bionic model response system based on a physics engine is developed. The finite element method (FEM) is used to simulate the dynamic response of the model. For example, when the robotic arm contacts the bionic model, the stress distribution and deformation at the contact point are calculated to update the shape of the model.

[0089] Preferably, the virtual laparoscopic surgical robot platform further includes a system integration module. This module employs timestamp and buffer management techniques to align data streams from different sources, enabling synchronized data processing from the individualized 3D bionic model reconstruction module, the robotic arm array module, and the virtual surgical feedback module. To ensure synchronized processing of data from deformation sensors, robotic arm sensors, and different functional modules, the virtual laparoscopic robot platform requires extremely high real-time performance. Therefore, this invention uses timestamp and buffer management techniques to align data streams from different sources. An extended Kalman filter (EKF) is applied to fuse data from the robotic arm sensors and bionic model sensors, improving the accuracy of state estimation. A low-pass filter is further designed to reduce the impact of sensor noise and environmental interference.

[0090] Preferably, the system integration module deploys GPU-based computing units to support high-speed data processing and complex signal operations. The system integration module uses an embedded real-time operating system to manage task priority and scheduling, ensuring that high-priority critical tasks, including signal processing and feedback control, are completed within time constraints. The embedded real-time operating system's scheduling algorithm is set to cyclic scheduling to ensure system response time and predictability. An efficient data flow architecture is designed to reduce data transmission latency and avoid data loss. The platform underwent multi-module compatibility and stability verification: the ROS (Robot Operating System) architecture was used to standardize communication between different modules of the overall system. Comprehensive system testing, including unit testing, integration testing, and performance testing, was conducted to ensure that all modules of the virtual laparoscopic surgical robot can work collaboratively, operate stably, and meet design requirements. Dynamic configuration functionality between modules is implemented, allowing the system to automatically reconfigure connections and settings between modules based on current workload and performance indicators, improving system flexibility and scalability.

[0091] Example 2

[0092] This application provides a personalized virtual surgical operation device, including the virtual laparoscopic surgical robot platform described in Embodiment 1 above. As shown in Figures 6 and 7, Figure 6 is a physical diagram of the personalized virtual laparoscopic surgical robot platform of this application, and Figure 7 is a schematic diagram of the personalized virtual laparoscopic surgical robot platform performing virtual surgery. This virtual laparoscopic surgical robot platform has the following functions: a personalized 3D bionic model reconstruction module, a robotic arm array module, a virtual surgical feedback module, and an interaction module. Through a unique multi-module interaction and system integration strategy, it achieves highly realistic and personalized surgical simulation. Its robotic arm array module is equipped with an adaptive gain adjustment strategy for torque and speed control. Through an adaptive gain adjustment PID controller, it ensures precise control of the robotic arm in complex surgical scenarios, improving the realism and accuracy of the surgery. Its interaction module achieves precise mapping and deep coupling between the robotic arm movements and the personalized 3D bionic model, ensuring a high degree of realism in human-computer interaction. The system integration module employs timestamp and buffer management techniques to align data streams from different sources and sets up an embedded real-time operating system, ensuring efficient communication and real-time computation among modules and improving the overall coordination and stability of the platform. It introduces weighting factors into the traditional QEM algorithm to preserve details for different tissues and organs. These weighting factors can be set based on the surgical target location, organ importance, and material properties. For example, in a liver surgery scenario, the liver can be designated as a critical organ with a higher weighting coefficient; important organs surrounding the liver can be designated as secondary targets with weaker weighting coefficients; and even weaker coefficients can be used for surrounding soft tissues. This significantly optimizes the number of 3D meshes in laparoscopic surgery scenarios, improving the computational efficiency and real-time performance of virtual surgery. Affine transformations are used to achieve precise mapping from the physical movements of the robotic arm to the movements of the virtual model. A real-time biomimetic model response system based on a physics engine uses the finite element method (FEM) to simulate the dynamic response of the model. These technological improvements work together to overcome the problems of existing simulators in terms of individualization, real-time feedback, interactivity, technological integration, data utilization, and simulation of surgical complexity, providing a more accurate, efficient, and practical virtual laparoscopic surgical robot platform.

[0093] An adaptive gain function is introduced as a coefficient to improve the traditional PID controller and realize the control process of the robotic arm. The adaptive gain function dynamically adjusts the gain parameter according to the real-time error and system state, and can automatically optimize its performance in different operation stages (such as cutting, knotting, rapid movement and fine operation). This is very important for accelerating the system's response speed to errors and dealing with emergency scenarios (such as emergency avoidance, abnormal robotic arm trajectory). Especially in complex surgical scenarios, it can ensure that the system is always in the optimal control state by adjusting the gain parameter in real time while meeting the requirements of real-time interaction of the laparoscopic surgical robot.

[0094] In constructing the virtual laparoscopic surgical robot platform, the collaborative work of each module is indispensable, forming an innovative overall system. The individualized 3D organ reconstruction module, by combining artificial intelligence and manual segmentation techniques, ensures the accuracy of organ segmentation and provides high-quality 3D geometric models through Marching Cubes algorithms and improved QEM optimization techniques. This provides precise targets for subsequent surgical simulations. The robotic arm array module, through an adaptive gain-adjusting PID controller, ensures precise control of the robotic arm in complex surgical scenarios, improving the realism and accuracy of the surgery. The virtual surgical feedback module provides real-time force feedback and high-definition visual feedback, enhancing the immersive experience of the surgical simulation. The real-time robotic arm control and virtual organ model interaction module achieves precise mapping and deep coupling between robotic arm movements and the bionic model, ensuring a high degree of realism in human-computer interaction. The multi-module interaction and system integration module, through the ROS architecture and real-time operating system, ensures efficient communication and real-time computation among the modules, improving the overall coordination and stability of the system. The absence of any single module would significantly impact the overall performance of the system. For example, without an individualized 3D organ reconstruction module, even if other modules function normally, the surgical simulation will lose its meaning due to the lack of accurate organ models and will fail to provide a realistic surgical environment. The innovation of this system lies in the tight coupling and interdependence between its modules, forming an inseparable whole that ensures the efficiency and accuracy of virtual laparoscopic surgical simulation.

[0095] The foregoing has described specific embodiments of the present invention. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0096] In the description of the embodiments of the present invention, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In the embodiments of the present invention, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in the embodiments of the present invention, as well as the features of the different embodiments or examples.

[0097] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features, excluding any ordering. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature and is used to distinguish it from another. In the description of embodiments of the present invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0098] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of the invention pertain.

[0099] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A virtual laparoscopic surgery robotic platform, characterized in that, include: The personalized 3D bionic model reconstruction module is used to generate personalized 3D bionic models based on clinical medical imaging data. The robotic arm array module includes at least three independent robotic arms for simulating laparoscopic surgery operations; A virtual surgical feedback module is used in conjunction with the robotic arm array module to provide real-time force and visual feedback; The interaction module is used to control the coordinated operation of the individualized 3D bionic model reconstruction module, the robotic arm array module, and the virtual surgery feedback module. It controls the individualized 3D bionic model to be updated accordingly based on the operation of the robotic arm array module, and controls the virtual surgery feedback module to update the visual feedback results.

2. The virtual laparoscopic surgical robot platform according to claim 1, characterized in that, The robotic arm array module is equipped with an adaptive gain adjustment strategy for torque control and speed control for each robotic arm. The adaptive gain adjustment strategy enables the PID controller of each robotic arm to dynamically adjust the gain parameters according to the real-time error and system status.

3. The virtual laparoscopic surgical robot platform according to claim 2, characterized in that, The formula for the adaptive gain adjustment strategy is expressed as follows: Where u(t) is the control input, torque or speed adjustment signal; t is the time point; e(t) is the error between the setpoint and the current position; and K... p (t), K i (t), K d (t) are the proportional, integral, and derivative gain parameters, respectively, with time and system state as independent variables, and are defined as follows: K i (t)=K i0 +a i ·f i (e(t),∫e(t)dt,S(t)); Among them, K p0 K i0 K d0 It corresponds to the initial gain of proportional, integral, and derivative equations; a p a i a d These are the adjustment coefficients corresponding to the proportional, integral, and derivative operations, used to control the amplitude of adaptive adjustment; f p f i f d It is an adaptive function; S(t) represents the current state information and is a parameter of the virtual surgical scene.

4. The virtual laparoscopic surgical robot platform according to claim 3, characterized in that, The robotic arm array module is equipped with joint torque sensors at each joint of the robotic arm to capture angle changes, piezoelectric sensors to capture forces when in contact with virtual organs, and accelerometers to monitor acceleration during movement. Each robotic arm includes at least three rotational degrees of freedom and one linear degree of freedom. The robotic arm array module applies a Kalman filter algorithm to fuse data from different sensors, where the update formula for the Kalman filter is: x k|k =x k|k-1 +K k (Z k -H k x k|k-1 ); x k|k It is the state estimate after the update at the k-th time point, x k|k-1 It is an estimate of the value at the previous time point, Z. k These are observed values, H k It is the observation model matrix, K k It is the Kalman gain, and the sensor data is processed through the observation model matrix H. k It is integrated into the state estimation and used for dynamic system state estimation of the robotic arm's movements.

5. The virtual laparoscopic surgical robot platform according to claim 1, characterized in that, The robotic arm array module uses a force model calculation formula to calculate the force when the end of the robotic arm comes into contact with the virtual organ. The force model calculation formula is: F=k·Δx; Where F is the contact force, Δx is the deformation, and k is the elastic constant, the value of which is dynamically adjusted according to the physical properties of different virtual organs; An impedance-based force feedback algorithm is used to cause the operator to feel forces that match the interaction in the virtual environment. The force feedback algorithm is expressed as: where F d is the desired force, x d 、 These are the desired position, velocity, and acceleration, x, It is the actual position, velocity, and acceleration, K d B d M d These are the controlled stiffness, damping, and mass parameters.

6. The virtual laparoscopic surgical robot platform according to claim 1, characterized in that, Affine transformation is used to convert the physical coordinate system of the robotic arm to the coordinate system of the virtual environment: Where (x,y,z) is the position of the robotic arm's end effector in the physical coordinate system, and (x′,y′,z′) is the transformed position in the virtual coordinate system. The transformation matrix includes rotation angle R, scaling S, and translation T, where t x , t y , t z These represent the translation amounts along the x, y, and z axes, respectively.

7. The virtual laparoscopic surgical robot platform according to claim 1, characterized in that, The virtual laparoscopic surgical robot platform also includes a system integration module, which uses timestamp and buffer management technology to align data streams from different sources, enabling synchronous processing of data from the individualized 3D bionic model reconstruction module, the robotic arm array module, and the virtual surgical feedback module.

8. The virtual laparoscopic surgical robot platform according to claim 7, characterized in that, The system integration module deploys GPU-based computing units to support high-speed data processing and complex signal operations; The system integration module is equipped with an embedded real-time operating system to manage task priority and scheduling, so as to ensure that high-priority critical tasks are completed within the time limit. High-priority critical tasks include signal processing and feedback control. The scheduling algorithm of the embedded real-time operating system is set to cyclic scheduling.

9. The virtual laparoscopic surgical robot platform according to any one of claims 1-8, characterized in that, The individualized 3D bionic model reconstruction module uses the Loop subdivision algorithm to refine the individualized 3D bionic model. Without changing the topology of the tissues and organs, it improves the smoothness and detail of the abdominal organ model by increasing the number of vertices and faces. Furthermore, it uses the QEM algorithm for local mesh optimization.

10. The virtual laparoscopic surgical robot platform according to claim 9, characterized in that, The steps for local mesh optimization using the QEM algorithm include: The QEM algorithm is improved by introducing a weighting factor, which is set based on the surgical target location, organ importance, and biomimetic material properties. For a liver surgery scenario, Q is defined as a triangular mesh of the liver, with vertex set V and edge set E. For each vertex v in V, the error metric is expressed as follows: Among them, K i It is the planar error matrix associated with vertex V; w i The weighting factor is w; i represents all nodes associated with vertex v; the formula is a weighted sum of all planar error matrices associated with vertex v, where w is the weighting factor. i External control over mesh refinement is achieved for different surgical scenarios.

11. The virtual laparoscopic surgical robot platform according to claim 10, characterized in that, The weighting factor w i The setup strategies include: assigning higher weights to the liver and adjacent organs to reduce simplification of these areas; using lower weights for distal organs and vascular tissues to the surgical target to obtain a simpler tissue mesh; and assigning lower weights to soft tissue areas than to organs, based on the differences in the physical properties of organs and soft tissues.

12. The virtual laparoscopic surgical robot platform according to claim 11, characterized in that, The personalized 3D biomimetic model reconstruction module also includes adding physical parameters based on real biomechanics to the personalized 3D biomimetic model to simulate the real physical behavior of different tissues. The physical parameters include elastic modulus, density, and coefficient of friction. The elastic modulus of the liver is set to 0.5-1.5 kPa, and the elastic modulus of muscle tissue is set to 8-12 kPa. In the parameter mapping process, the physical parameters are mapped to each mesh cell of the 3D model using a voxelization method. The corresponding physical parameters are calculated and assigned on the mesh nodes through cubic linear interpolation to ensure the continuity of the physical properties of the personalized 3D biomimetic model throughout the entire volume.

13. A personalized virtual surgical operation device, characterized in that, Includes the virtual laparoscopic surgical robot platform as described in claim 12.

Images