Adaptive force feedback method based on individualized virtual laparoscopic surgery, device, and medium
By introducing a refined surface membrane and a joint segmentation model of Mamba-CNN into a virtual laparoscopic surgery system, adaptive force feedback for different tissues and organs is achieved, solving the problem of a single force feedback mode in virtual surgery systems and improving the realism and computational efficiency of surgical simulation.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- QINGDAO UNIV
- Filing Date
- 2024-12-30
- Publication Date
- 2026-07-02
Smart Images
Figure CN2024144008_02072026_PF_FP_ABST
Abstract
Description
Adaptive force feedback method, equipment, and media based on personalized virtual laparoscopic surgery Technical Field
[0001] This application relates to the field of adaptive force feedback calculation technology, specifically to an adaptive force feedback method, device, and medium based on individualized virtual laparoscopic surgery. Background Technology
[0002] With the advancement of modern clinical medicine and the growing public demand for precision medicine services, digital, personalized, precise, and minimally invasive laparoscopic surgery has become an urgent clinical need and a cutting-edge field in medical technology development. However, the field of laparoscopy worldwide currently faces challenges such as a single system training model, a lack of individualized training modules for different lesions, a lack of adaptive force feedback modules for different tissues and organs, and a lack of realism in surgical simulation, all of which hinder preoperative simulation training for clinicians.
[0003] Chinese patent application CN112528529A discloses a laparoscopic naked-eye simulation method, proposing a finite element simulation contact collision detection method and soft tissue material failure criteria for organ models. It conducts experiments and analyses on the effects of external factors such as inflation pressure, respiratory motion, and surgical instrument traction, compression, and cutting on the deformation of organ models, determining the magnitude of the interaction forces between different factors and tissues, setting contact collision and rupture force thresholds, designing a force response-based load-contact collision detection method for organs and tissues, and performing finite element model simulation, optimization, and verification of complete organs. Soft tissues of organs deform or rupture under load, and the core of the rupture problem is determining the failure criteria for different organs and tissues. This scheme studies the tissue failure criteria of organs based on the distortion energy theory of materials; that is, when the distortion energy density of the organ model tissue reaches its limit value, the material fails. It uses the Abaqus software subroutine Vumat to define the hyperelastic material properties and failure criteria of soft tissues. However, this method cannot achieve rapid and accurate switching of adaptive force feedback parameters between different tissues and organs.
[0004] Therefore, there is an urgent need to develop a new adaptive force feedback method based on virtual laparoscopic surgery to effectively improve the processing speed and accuracy of adaptive force feedback switching, and enhance the quality and realism of virtual surgery. Summary of the Invention
[0005] This application aims to at least partially address one of the technical problems in related technologies. To this end, this application provides an adaptive force feedback method, device, and medium based on individualized virtual laparoscopic surgery, capable of adaptively switching force feedback parameters for different types of tissues and organs, thereby improving the quality and realism of virtual surgery.
[0006] To achieve the above objectives, in a first aspect, this application provides an adaptive force feedback method based on virtual laparoscopic surgery, comprising:
[0007] The system detects whether the tip of the virtual surgical instrument tactile pen comes into contact with the abdominal tissue / organ model. The outermost layer of the abdominal tissue / organ model is covered with a surface membrane. When the tip of the virtual surgical instrument tactile pen comes into contact with the surface membrane, it is determined that it has come into contact with the corresponding abdominal tissue / organ model.
[0008] If the virtual surgical instrument tactile pen tip comes into contact with the abdominal tissue / organ model, the type of the abdominal tissue / organ model in contact is identified, and the force feedback parameter settings of the corresponding abdominal tissue / organ model are switched to perform force feedback calculation.
[0009] The system detects whether the haptic pen tip of the virtual surgical instrument penetrates the surface membrane and enters the abdominal tissue / organ model. If so, it performs force feedback calculation based on the switched force feedback parameter settings and renders the scene.
[0010] Preferably, the process of modeling the abdominal tissue / organ model includes:
[0011] Acquire clinical medical imaging data and preprocess the clinical medical imaging data;
[0012] The preprocessed clinical medical image data is input into the Mamba-CNN joint segmentation model for segmentation processing, and the output is a segmented image of tissues and organs; wherein, the Mamba-CNN joint segmentation model uses a CNN network to capture local and detailed features in the low-order part of the encoder, and uses a Mamba model to extract global dependency features in the high-order part of the encoder.
[0013] Based on three-dimensional image reconstruction technology, a model of the abdominal cavity tissue / organ was constructed.
[0014] Preferably, the surface membrane is generated by dilating the segmented image of the tissue / organ and wraps around the abdominal tissue / organ model; the distance between the surface membrane and the abdominal tissue / organ model is changed by adjusting the dilution coefficient of the surface membrane, and the size and number of the smallest units constituting the surface membrane are adjusted by changing the downsampling parameters.
[0015] Preferably, the surface film is provided with an expansion coefficient adjustment strategy, which is configured such that: when the acceleration of the virtual surgical instrument movement increases, the expansion coefficient of the surface film increases, and when the acceleration of the virtual surgical instrument movement decreases, the expansion coefficient of the surface film decreases; wherein, the expansion coefficient represents the degree of outward expansion centered on the expansion structure core.
[0016] Preferably, the step of generating the surface film by dilating a segmented image of a tissue or organ includes:
[0017] Acquire segmented images of tissues and organs;
[0018] The segmented images of the tissues and organs are subjected to dilation processing centered on the dilated structural kernel;
[0019] Using the surface rendering reconstruction method, the surface of the expanded tissues and organs is reconstructed in three dimensions to generate a surface membrane, and the three-dimensional spatial coordinate range of the surface membrane is calibrated.
[0020] Preferably, the surface drawing and reconstruction method employs one of the following: contour line method, marching plane method, or isosurface method.
[0021] Preferably, the surface film is provided with a spatial resolution adjustment strategy, wherein the spatial resolution represents the number of triangular meshes per unit area of the surface film. The more triangular meshes there are, the higher the spatial resolution of the surface film. The spatial resolution adjustment strategy is configured to improve the spatial resolution of the surface film by increasing the downsampling parameter when the minimum vertical distance between the virtual surgical instrument tactile pen tip and the surface film is less than a first threshold.
[0022] Preferably, the step of detecting whether the tip of the virtual surgical instrument haptic pen comes into contact with the abdominal tissue / organ model includes:
[0023] Real-time acquisition of the three-dimensional spatial coordinates of virtual surgical instruments;
[0024] Based on the three-dimensional spatial coordinates of the virtual surgical instrument and the three-dimensional spatial coordinate range of the surface membrane of each abdominal tissue / organ model, a collision detection algorithm is used to detect the collision between the virtual surgical instrument tactile pen tip and the surface membrane of all abdominal tissue / organ models.
[0025] If the tip of the virtual surgical instrument tactile pen collides with the surface membrane of any abdominal tissue / organ model, it is considered that the tip of the virtual surgical instrument tactile pen has come into contact with the abdominal tissue / organ model.
[0026] Preferably, the step of identifying the type of abdominal tissue / organ model in which the contact occurred includes:
[0027] Record the three-dimensional coordinate position of the virtual surgical instrument tactile pen tip in contact with the abdominal tissue / organ model;
[0028] Determine the relationship between the three-dimensional coordinate position and the three-dimensional spatial coordinate range of the surface membrane of each abdominal tissue / organ model;
[0029] If the three-dimensional coordinate position falls within the three-dimensional spatial coordinate range of the surface membrane of the corresponding abdominal tissue / organ model, it is determined that the virtual surgical instrument tactile pen tip has come into contact with the corresponding abdominal tissue / organ model.
[0030] Preferably, the collision detection algorithm employs one of the following: OBB algorithm, SAT algorithm, or FCL triangular face detection algorithm.
[0031] In a second aspect, this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the adaptive force feedback method described in any of the preceding claims.
[0032] Thirdly, this application provides a computer-readable storage medium including a computer program that, when run on an electronic device, causes the electronic device to perform the adaptive force feedback method described in any of the preceding claims.
[0033] Fourthly, this application provides a personalized virtual surgical operation device, including the aforementioned electronic device.
[0034] Based on the above technical solution, the adaptive force feedback method based on virtual laparoscopic surgery of this application has at least one of the following beneficial effects compared with the prior art:
[0035] 1. To ensure the sensitivity of force feedback generated when virtual surgical instruments contact abdominal tissue / organ models, this invention adds a refined surface film to the volume rendering of the tissues and organs, serving as the boundary for adaptively switching force feedback parameters. This refined surface film has high spatial resolution, which helps reduce false alarms in tissue / organ collision detection.
[0036] 2. This invention adjusts the expansion coefficient of the surface film based on the acceleration of the virtual surgical instrument's movements, and adjusts the spatial resolution of the surface film based on the minimum vertical distance between the virtual surgical instrument's tactile pen tip and the surface film, greatly improving the calculation speed and collision detection accuracy.
[0037] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description, or may be realized by practicing the application. The purpose and other advantages of this application can be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings. Attached Figure Description
[0038] Figure 1 is a flowchart illustrating the adaptive force feedback method based on virtual laparoscopic surgery in this application.
[0039] Figure 2 is a module connection diagram of the individualized virtual surgical operation device in this application;
[0040] Figure 3 is a connection diagram of the virtual surgical instrument signal acquisition module in the personalized virtual surgical operation device of this application;
[0041] Figure 4 is a connection diagram of the force feedback parameter acquisition module in the individualized virtual surgical operation device of this application;
[0042] Figure 5 is a connection diagram of the adaptive parameter adjustment module and the force feedback calculation and scene rendering module in the individualized virtual surgical operation device of this application.
[0043] Figure 6 is a schematic diagram of the generation of a surface film by dilating the segmented image of tissues and organs in this application. Detailed Implementation
[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.
[0045] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention are also intended to include the plural forms unless the context clearly indicates otherwise.
[0046] To address the shortcomings of existing technologies, the purpose of this invention is to provide an adaptive force feedback method based on virtual laparoscopic surgery, aiming to overcome the problems of current virtual laparoscopic systems having the same elastic deformation parameters for different tissues and organs, a single force feedback mode, and a lack of surgical simulation realism.
[0047] The basic idea of this invention is to collect a large amount of offline data on force feedback and puncture depth or stretching length based on real-world tests conducted by clinicians on different tissues and organs. This data is used for training to construct a force feedback prediction model for different tissues and organs. This model is then used in a virtual laparoscopic system to identify tissues and organs in real time when virtual surgical instruments come into contact with them, and to adaptively switch their elastic deformation and force feedback, thereby enhancing the realism of the virtual surgery. To ensure the sensitivity of the force feedback generated by the virtual surgical instruments when they contact tissues and organs, this application adds a refined surface film to the volume rendering of the tissues and organs, serving as the boundary for adaptively switching force feedback parameters. Furthermore, this refined surface film has high spatial resolution, which helps reduce misjudgments of tissues and organs by the contact point. To improve computational speed, this invention adjusts the spatial resolution of the surface film according to the volume of different tissues and organs.
[0048] Example 1
[0049] In order to develop a precise adaptive force feedback method, the inventors conducted in-depth research on artificial intelligence and force feedback technology, and proposed an adaptive force feedback method based on virtual laparoscopic surgery.
[0050] Specifically, as shown in Figure 1, an adaptive force feedback method based on virtual laparoscopic surgery is provided, including the following steps:
[0051] The system detects whether the tip of the virtual surgical instrument tactile pen comes into contact with the abdominal tissue / organ model. The outermost layer of the abdominal tissue / organ model is covered with a surface membrane. When the tip of the virtual surgical instrument tactile pen comes into contact with the surface membrane, it is determined that it has come into contact with the corresponding abdominal tissue / organ model.
[0052] If the virtual surgical instrument tactile pen tip comes into contact with the abdominal tissue / organ model, the type of the abdominal tissue / organ model in contact is identified, and the force feedback parameter settings of the corresponding abdominal tissue / organ model are switched to perform force feedback calculation.
[0053] The system detects whether the haptic pen tip of the virtual surgical instrument penetrates the surface membrane and enters the abdominal tissue / organ model. If so, it performs force feedback calculation based on the switched force feedback parameter settings and renders the scene.
[0054] Preferably, the process of modeling the abdominal tissue / organ model includes:
[0055] Acquire individualized clinical medical imaging data and preprocess the individualized clinical medical imaging data;
[0056] The preprocessed individualized clinical medical image data is input into the Mamba-CNN joint segmentation model for segmentation processing, and the output is a segmented image of tissues and organs; wherein, the Mamba-CNN joint segmentation model uses a CNN network to capture local and detailed features in the low-order part of the encoder, and uses a Mamba model to extract global dependency features in the high-order part of the encoder.
[0057] Based on three-dimensional image reconstruction technology, a model of the abdominal cavity tissue / organ was constructed.
[0058] Preferably, the surface membrane is generated by dilating the segmented image of the tissue / organ and wraps around the abdominal tissue / organ model; the distance between the surface membrane and the abdominal tissue / organ model is changed by adjusting the dilution coefficient of the surface membrane, and the size and number of the smallest units constituting the surface membrane are adjusted by changing the downsampling parameters.
[0059] Preferably, the surface film is provided with an expansion coefficient adjustment strategy, which is configured such that: when the acceleration of the virtual surgical instrument movement increases, the expansion coefficient of the surface film increases, and when the acceleration of the virtual surgical instrument movement decreases, the expansion coefficient of the surface film decreases; wherein, the expansion coefficient represents the degree of outward expansion centered on the expansion structure core.
[0060] Preferably, the step of generating the surface film by dilating a segmented image of a tissue or organ includes:
[0061] Acquire segmented images of tissues and organs;
[0062] The segmented images of the tissues and organs are subjected to dilation processing centered on the dilated structural kernel;
[0063] Using the surface rendering reconstruction method, the surface of the expanded tissues and organs is reconstructed in three dimensions to generate a surface membrane, and the three-dimensional spatial coordinate range of the surface membrane is calibrated.
[0064] Specifically, when a virtual surgical instrument collides with a tissue or organ based on its spatial position obtained from the acquired signals, the system automatically identifies the specific tissue or organ and switches its force feedback parameters to calculate the force feedback. To improve the sensitivity of adaptively switching force feedback parameters for different tissues and organs and calculating force feedback, this application innovatively introduces a surface membrane as the boundary for parameter adjustment. This surface membrane is generated by expanding the segmented three-dimensional image and wraps around the tissue or organ. By changing the expansion coefficient inf, the distance from the surface membrane to the tissue or organ can be changed. By changing the downsampling parameter ds, the size and number of the smallest unit constituting the surface membrane can be adjusted, thereby improving the calculation speed while ensuring a low misjudgment rate of the contact point for the tissue or organ. This invention uses the collision recognition between the virtual surgical instrument and the surface membrane as the signal for switching force feedback parameters. Before constructing the surface membrane, artificial intelligence technology is used to automatically detect the three-dimensional spatial boundaries of different tissues and organs, constructing a voxel-based three-dimensional model to calibrate the three-dimensional spatial coordinate range of different tissues and organs, and also to prepare for subsequent simulated cutting, burning, and other operations. When the virtual surgical instrument passes through the surface membrane and enters the three-dimensional space of the tissue or organ, force feedback is calculated based on the switched force feedback parameters, and scene rendering is performed. During scene rendering, this application employs a multi-layered rendering method to represent the process of the virtual surgical instrument passing through the surface membrane to the tissue or organ, thereby improving rendering speed and performance. Specifically, when the virtual surgical instrument approaches the surface membrane, a smaller number of triangle models are used for simplification; when the virtual surgical instrument penetrates the surface membrane to reach the tissue or organ, a larger number of triangle models are used for detailed scene representation to provide a more realistic visual experience. By adaptively switching the force feedback parameters for different tissues and organs in real time, the feedback forces of different tissues and organs can be obtained, thereby enhancing the realism of the virtual surgery.
[0065] Preferably, the surface drawing and reconstruction method employs one of the following: contour line method, marching cubes method, or opaque cube method.
[0066] Preferably, the surface film is provided with a spatial resolution adjustment strategy, wherein the spatial resolution represents the number of triangular meshes per unit area of the surface film. The more triangular meshes there are, the higher the spatial resolution of the surface film. The spatial resolution adjustment strategy is configured to improve the spatial resolution of the surface film by increasing the downsampling parameter when the minimum vertical distance between the virtual surgical instrument tactile pen tip and the surface film is less than a first threshold.
[0067] Preferably, the step of detecting whether the tip of the virtual surgical instrument haptic pen comes into contact with the abdominal tissue / organ model includes:
[0068] Real-time acquisition of the three-dimensional spatial coordinates of virtual surgical instruments;
[0069] Based on the three-dimensional spatial coordinates of the virtual surgical instrument and the three-dimensional spatial coordinate range of the surface membrane of each abdominal tissue / organ model, a collision detection algorithm is used to detect the collision between the virtual surgical instrument tactile pen tip and the surface membrane of all abdominal tissue / organ models.
[0070] If the tip of the virtual surgical instrument tactile pen collides with the surface membrane of any abdominal tissue / organ model, it is considered that the tip of the virtual surgical instrument tactile pen has come into contact with the abdominal tissue / organ model.
[0071] Preferably, the step of identifying the type of abdominal tissue / organ model in which the contact occurred includes:
[0072] Record the three-dimensional coordinate position of the virtual surgical instrument tactile pen tip in contact with the abdominal tissue / organ model;
[0073] Determine the relationship between the three-dimensional coordinate position and the three-dimensional spatial coordinate range of the surface membrane of each abdominal tissue / organ model;
[0074] If the three-dimensional coordinate position falls within the three-dimensional spatial coordinate range of the surface membrane of the corresponding abdominal tissue / organ model, it is determined that the virtual surgical instrument tactile pen tip has come into contact with the corresponding abdominal tissue / organ model.
[0075] As shown in Figure 6, this application illustrates the process of dilating the segmented tissue / organ image to generate a surface membrane. Figure 6(a) shows the liver boundary after segmentation by the joint segmentation model, Figure 6(b) shows a schematic diagram of the dilated liver, Figure 6(c) shows a schematic diagram of the liver constructed with the dilated surface membrane, and Figure 6(d) shows a schematic diagram of the liver after downsampling with the surface membrane. Since the surface membrane encapsulates the tissue / organ, it collides with the virtual surgical instrument before the instrument operates on the abdominal tissue / organ model. To improve the sensitivity of adaptively switching different tissue / organ force feedback parameters and calculating force feedback, this application uses the collision between the virtual surgical instrument and the surface membrane as a signal for tissue / organ identification and initiating the switching of force feedback parameters. The system acquires the three-dimensional spatial coordinates of virtual surgical instruments in real time. Based on the calibrated three-dimensional spatial range of the surface membranes of different tissues and organs, a collision detection algorithm is used to detect collisions between the virtual surgical instrument's haptic pen tip and the surface membranes of each tissue and organ. If a collision occurs with a tissue or organ, the system identifies that tissue or organ and begins switching force feedback parameters. The collision detection algorithm used can be OBB, SAT algorithm, FCL-triangulation, etc. When the virtual surgical instrument's haptic pen tip continues to enter the calibrated three-dimensional spatial boundary range of a specific tissue or organ, it indicates that the virtual surgical instrument has collided with that tissue or organ. Based on the switched force feedback parameters and the following modules, force feedback is calculated, and operations such as cutting and cauterization are performed. The higher the spatial resolution of the surface membrane, the lower the error rate in identifying different tissues and organs when colliding with the surface membrane, especially for adjacent tissues and organs.
[0076] To ensure high sensitivity in adaptive force feedback parameter adjustment, a low error rate in organ and tissue recognition, and simultaneously improve computational speed, this invention downsamples and reconstructs the acquired surface membrane at a downsampling ratio of ds. After downsampling, the number of triangular meshes constituting the surface membrane is ds*N, where N represents the number of triangular meshes before downsampling. When the minimum vertical distance between the virtual surgical instrument's haptic pen tip and the surface membrane is less than a first threshold, downsampling is performed at the downsampling ratio ds to improve the spatial resolution of the surface membrane, i.e., to increase the number of triangular meshes constituting the surface membrane.
[0077] When the tactile pen tip of the virtual surgical instrument penetrates the surface membrane and enters the three-dimensional spatial boundary of a specific tissue or organ, the force feedback parameters obtained above are brought into the corresponding physical deformation model. The elastic deformation and force feedback of each radiation layer in the three dimensions of X, Y, and Z are calculated in real time and transmitted to the surgical instrument signal acquisition module. At the same time, the spatial coordinates of the deformed data nodes are updated and visual feedback in the virtual scene is provided.
[0078] It should be noted that in step S2, when switching to the force feedback parameter settings of the corresponding abdominal tissue / organ model for force feedback calculation, the force feedback parameter settings of different abdominal tissue / organ models can be collected and calculated in the following manner.
[0079] To characterize and capture the motion trajectory of the handpiece during actual laparoscopic surgery, the virtual surgical instrument uses a spherical coordinate system to establish the spatial movement and position representation of the actual handpiece. This includes capturing the horizontal rotation angle α and vertical rotation angle β when the handpiece rotates, the diameter R when the handpiece is pulled up, and the spin angle θ when the handpiece performs operations such as cutting, ligation, and cauterization. To capture the motion signals of the virtual surgical instrument in real time, high-precision encoders are installed at the joints of the virtual surgical instrument to capture angle changes. Simultaneously, force sensors monitor contact forces, and accelerometers are installed to monitor the acceleration of the motion. These signals are then accurately transmitted to the central control unit for processing. The specific force feedback acquisition and calculation process is as follows:
[0080] a. For different tissues and organs, using multiple experienced clinicians as test subjects, force sensors and strain gauges were used to measure the depth of application, such as puncture depth or stretching length L. 测试 The required force F 测试 Overall longitudinal deformation l 测试 Centered on the contact point, set m marker points from the inside out, and record the deformation r in the tangential direction. i and vertical deformation y i Where i∈{1,2,…m}. The force sensor can be selected from resistive strain gauge, capacitive, or other types of sensors;
[0081] b. Initial physical deformation models are constructed for different tissues and organs. The physical deformation model used is a spring-mass model based on a hexagonal network topology. When a force is applied, the tissue deformation spreads outward from the contact point in a radial concentric hexagonal structure. The specific formula is as follows: α i =2i-1 (4)
[0082] Where F represents the total force acting on the tissue or organ, m is the number of radiation layers, initialized to 2; Δy i K represents the spring deformation, i.e., the longitudinal deformation, of the i-th layer of mass. i The spring constants in different radiation layers; l i The vertical spring deformation of the i-th layer mass point is represented by r; r represents the radius of the hexagonal network topology, initialized to 1; α i This is an intermediate coefficient. The physical deformation model used in this application can also be other spring-mass models.
[0083] c. Based on the artificial intelligence model, train the model shown in step b using the offline data collected in step a to obtain the optimal elasticity coefficient K. i And calculate the error ε=F 测试 -F. The training model used can be an artificial intelligence and machine learning method such as CNN.
[0084] d. Increase the number of radiation layers m, repeating steps b and c until the error ε is less than a set threshold or the number of radiation layers m reaches the set threshold. Determine the optimal number of radiation layers m and the optimal elastic coefficient K. i And the optimal network radius r, thereby determining the physical deformation model of a specific tissue or organ;
[0085] e. Traverse all organs and tissues, calculate and save the force feedback parameters for different organs and tissues, including the optimal number of radiation layers m and the optimal elastic coefficient K. i And the optimal network radius r, so that it can be retrieved later.
[0086] It should be noted that the training process of the above Mamba-CNN joint segmentation model includes: preprocessing and data augmentation of the pre-set clinical image data of abdominal organs, and performing geometric transformation, brightness and contrast intensity transformation and elastic deformation on the data to increase the dataset and enhance the robustness and generalization of the artificial intelligence model.
[0087] The algorithm divides the image into training and testing sets. A joint Mamba-CNN segmentation model is trained on the training set and tested on the testing set to achieve automatic segmentation of different tissues and organs. The selected AI model is the Mamba-CNN joint segmentation algorithm. In the low-order part of the encoder, a CNN network is used to capture local and detailed features, while in the high-order part, a Mamba model is used to extract global dependency features. In the low-order part, the CNN is used to capture local and detailed features. Convolutional layers can automatically learn and extract local features in the image, such as edges and textures. In the high-order part, the Mamba model is used to extract global dependency features. The Mamba model can capture the global dependencies of temporal or spatial sequences in image data.
[0088] It should be noted that the termination condition for model training is that the rate of decrease of loss is less than a preset threshold for multiple consecutive training cycles.
[0089] Using 3D image reconstruction technology, 3D models of different tissues and organs are constructed based on voxels, and smoothing and other processing are performed. The 3D spatial coordinate range of different tissues and organs is calibrated, and the volume of different tissues and organs is calculated. The size of the voxels is adjusted to improve the calculation speed while ensuring that the reconstructed objects are not distorted.
[0090] The specific steps of using the surface rendering and reconstruction method to perform three-dimensional surface reconstruction of the expanded abdominal tissue / organ model and generate a surface membrane include:
[0091] The segmented images of tissues and organs are dilated, with the dilation kernel being K and the dilation coefficient being inf, representing the degree of outward dilation centered on the kernel. A larger inf indicates a greater degree of image dilation and a larger boundary distance between the tissues and organs before and after dilation. The selected dilation coefficient is related to the acceleration of movements during actual laparoscopic surgery. The greater the acceleration, the larger the selected dilation coefficient is needed to ensure sensitivity for adaptive switching. A surface rendering reconstruction method is used to perform 3D surface reconstruction of the dilated tissues and organs, which serves as the boundary for collision detection and defines the 3D spatial coordinate range of the surface membrane. The selected surface rendering reconstruction method can be contour line method, Marching cubes, Opaque Cube, etc., initializing the number of triangular meshes constituting the surface membrane; a larger number of meshes indicates a higher spatial resolution for the surface membrane.
[0092] Example 2
[0093] This embodiment provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to implement the adaptive force feedback method described in Embodiment 1 above, and to achieve the following functions: To ensure the sensitivity of force feedback generated when the virtual surgical instrument contacts the abdominal tissue / organ model, a refined surface film is added on top of the volume rendering of the tissue / organ as a boundary for adaptively switching force feedback parameters. This refined surface film has high spatial resolution, which helps reduce misjudgments in tissue / organ collision detection. The expansion coefficient of the surface film is adjusted according to the acceleration of the virtual surgical instrument's movement, and the spatial resolution of the surface film is adjusted according to the minimum vertical distance between the virtual surgical instrument's tactile pen tip and the surface film, greatly improving the calculation speed and collision detection accuracy, and enhancing the quality and realism of the virtual surgery. Specifically, this electronic device can be a controller in a specific device.
[0094] Example 3
[0095] Based on the same technical concept, embodiments of this application also provide a computer-readable storage medium storing a computer program. When the computer program runs on a computer or processor, it causes the computer or processor to execute the steps of the aforementioned adaptive force feedback method. It achieves the following function: To ensure the sensitivity of force feedback generated when the virtual surgical instrument contacts the abdominal tissue / organ model, a refined surface film is added on top of the volume rendering of the tissue / organ, serving as the boundary for adaptively switching force feedback parameters. This refined surface film has high spatial resolution, helping to reduce misjudgments in tissue / organ collision detection. The expansion coefficient of the surface film is adjusted according to the acceleration of the virtual surgical instrument's movement, and the spatial resolution of the surface film is adjusted according to the minimum perpendicular distance between the virtual surgical instrument's tactile pen tip and the surface film, greatly improving the calculation speed and collision detection accuracy, thus enhancing the quality and realism of the virtual surgery.
[0096] Example 4
[0097] As shown in Figure 2, this application provides a personalized virtual surgical operation device, comprising:
[0098] The virtual surgical instrument signal acquisition module acquires the motion trajectory of the virtual surgical instrument handle during virtual laparoscopic surgery, including the acquisition of the handle's spatial position and movement signals, horizontal and vertical rotation angles, handle lifting diameter, handle operation spin angle, contact force, and motion acceleration signals, and transmits the acquired signals to the central control processing unit for processing.
[0099] The force feedback parameter acquisition module is used to acquire the test deformation of different tissues and organs based on clinical experience, initialize the physical deformation model, then train the physical deformation model for different tissues and organs to determine the optimal parameters, and finally save the force feedback parameters of different tissues and organs for subsequent switching of force feedback parameter settings.
[0100] The adaptive parameter adjustment module and the force feedback calculation and scene rendering module are used to implement the adaptive force feedback method in the above embodiment one.
[0101] As shown in Figures 3-5, in the virtual surgical instrument signal acquisition module, the virtual surgical instrument uses a spherical coordinate system to establish the spatial movement and position representation of the actual handle. This includes the horizontal rotation angle α and vertical rotation angle β acquired when the handle rotates, the diameter R acquired when the handle is lifted, and the spin angle θ acquired when the handle performs operations such as cutting, ligation, and cauterization. To capture the motion signals of the virtual surgical instrument in real time, high-precision encoders are installed at the joints of the virtual surgical instrument to capture angle changes. Simultaneously, force sensors are used to monitor contact forces, and accelerometers are installed to monitor the acceleration of the motion. These signals are then accurately transmitted to the central control unit for processing.
[0102] The force feedback parameter acquisition module, targeting different tissues and organs, and using multiple experienced clinicians as test subjects, employs force sensors and strain gauges to measure the force at different depths, such as puncture depth or stretching length L. 测试 The required force F 测试 Overall longitudinal deformation l 测试 Centered on the contact point, set m marker points from the inside out, and record the deformation r in the tangential direction. i and vertical deformation y i Where i∈{1,2,…m}. The force sensor can be selected from resistive strain gauge, capacitive, or other types of sensors;
[0103] Then, initial physical deformation models were constructed for different tissues and organs. The physical deformation model used was a spring-mass model based on a hexagonal network topology. When a force is applied, the tissue deformation spreads outward in a radial concentric hexagonal structure centered on the contact point.
[0104] Based on an artificial intelligence model, an initial physical deformation model is trained using collected offline data to obtain the optimal elastic coefficients. The training model used can be an artificial intelligence and machine learning method such as CNN.
[0105] Determine the optimal number of radiation layers, the optimal elastic coefficient, and the optimal network radius, thereby determining the physical deformation model for a specific tissue or organ;
[0106] Finally, all organs and tissues are traversed, and force feedback parameters for different organs and tissues are calculated and saved, including the optimal number of radiation layers, the optimal elastic coefficient, and the optimal network radius, for later retrieval.
[0107] The adaptive parameter adjustment module and the force feedback calculation and scene rendering module first construct abdominal tissue / organ models of different tissues and organs based on clinical medical imaging data and calibrate their three-dimensional spatial coordinate range. Then, the segmented tissue / organ images are dilated and the surface is reconstructed in three dimensions to serve as the boundary for collision detection, and the three-dimensional spatial coordinate range of the surface membrane is calibrated. Next, the motion information of the virtual surgical instrument handle during virtual laparoscopic surgery is acquired by the virtual surgical instrument signal acquisition module, and the system calculates whether the virtual surgical instrument tactile pen tip contacts the surface membrane and whether it penetrates the surface membrane into the abdominal tissue / organ model. If the virtual surgical instrument tactile pen tip contacts the surface membrane, the corresponding tissue / organ is identified and its force feedback parameters are called. If the virtual surgical instrument tactile pen tip penetrates the surface membrane into the abdominal tissue / organ model, force feedback calculation is performed based on the switched force feedback parameter settings, and scene rendering is performed.
[0108] To ensure the sensitivity of force feedback generated by virtual surgical instruments when contacting abdominal tissue / organ models, this application adds a refined surface membrane to the volume rendering of the tissues and organs, serving as the boundary for adaptively switching force feedback parameters. This refined surface membrane has high spatial resolution, helping to reduce false positives in tissue / organ collision detection. Furthermore, the expansion coefficient of the surface membrane is adjusted based on the acceleration of the virtual surgical instrument's movements, and the spatial resolution of the surface membrane is adjusted based on the minimum perpendicular distance between the virtual surgical instrument's haptic pen tip and the surface membrane. This significantly improves computational speed and collision detection accuracy, enhancing the quality and realism of the virtual surgery.
[0109] The foregoing has described specific embodiments of the present invention. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.
[0110] In the description of the embodiments of the present invention, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In the embodiments of the present invention, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in the embodiments of the present invention, as well as the features of the different embodiments or examples.
[0111] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features, excluding any ordering. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature and is used to distinguish it from another. In the description of embodiments of the present invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0112] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, as should be understood by those skilled in the art to which embodiments of the invention pertain.
[0113] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. An adaptive force feedback method based on virtual laparoscopic surgery, characterized in that, include: The system detects whether the tip of the virtual surgical instrument tactile pen comes into contact with the abdominal tissue / organ model. The outermost layer of the abdominal tissue / organ model is covered with a surface membrane. When the tip of the virtual surgical instrument tactile pen comes into contact with the surface membrane, it is determined that it has come into contact with the corresponding abdominal tissue / organ model. If the virtual surgical instrument tactile pen tip comes into contact with the abdominal tissue / organ model, the type of the abdominal tissue / organ model in contact is identified, and the force feedback parameter settings of the corresponding abdominal tissue / organ model are switched to perform force feedback calculation. The system detects whether the haptic pen tip of the virtual surgical instrument penetrates the surface membrane and enters the abdominal tissue / organ model. If so, it performs force feedback calculation based on the switched force feedback parameter settings and renders the scene.
2. The adaptive force feedback method according to claim 1, characterized in that, The process of modeling the abdominal tissue / organ model includes: Acquire clinical medical imaging data and preprocess the clinical medical imaging data; The preprocessed clinical medical image data is input into the Mamba-CNN joint segmentation model for segmentation processing, and the output is a segmented image of tissues and organs; wherein, the Mamba-CNN joint segmentation model uses a CNN network to capture local and detailed features in the low-order part of the encoder, and uses a Mamba model to extract global dependency features in the high-order part of the encoder. Based on three-dimensional image reconstruction technology, a model of the abdominal cavity tissue / organ was constructed.
3. The adaptive force feedback method of claim 2, wherein, The surface membrane is generated by dilating the segmented image of the tissue and organ, and wraps around the abdominal tissue / organ model. The distance between the surface membrane and the abdominal tissue / organ model is changed by adjusting the dilution coefficient of the surface membrane, and the size and number of the smallest unit constituting the surface membrane are adjusted by changing the downsampling parameters.
4. The adaptive force feedback method of claim 3, wherein, The surface film is provided with an expansion coefficient adjustment strategy, which is configured such that: when the acceleration of the virtual surgical instrument movement increases, the expansion coefficient of the surface film increases, and when the acceleration of the virtual surgical instrument movement decreases, the expansion coefficient of the surface film decreases; wherein, the expansion coefficient represents the degree of outward expansion centered on the expansion structure core.
5. The adaptive force feedback method of claim 4, wherein, The surface film is generated by dilating segmented images of tissues and organs, including the following steps: Acquire segmented images of tissues and organs; The segmented images of the tissues and organs are subjected to dilation processing centered on the dilated structural kernel; Using the surface rendering reconstruction method, the surface of the expanded tissues and organs is reconstructed in three dimensions to generate a surface membrane, and the three-dimensional spatial coordinate range of the surface membrane is calibrated.
6. The adaptive force feedback method according to claim 5, characterized in that, The surface drawing and reconstruction method adopts one of the following: contour line method, marching method, and isosurface method.
7. The adaptive force feedback method according to claim 5, characterized in that, The surface film is equipped with a spatial resolution adjustment strategy. The spatial resolution represents the number of triangular meshes per unit area of the surface film. The more triangular meshes there are, the higher the spatial resolution of the surface film. The spatial resolution adjustment strategy is configured to improve the spatial resolution of the surface film by increasing the downsampling parameter when the minimum vertical distance between the virtual surgical instrument tactile pen tip and the surface film is less than a first threshold.
8. The adaptive force feedback method according to claim 5, characterized in that, The steps for detecting whether the tip of the virtual surgical instrument haptic pen is in contact with the abdominal tissue / organ model include: Real-time acquisition of the three-dimensional spatial coordinates of virtual surgical instruments; Based on the three-dimensional spatial coordinates of the virtual surgical instrument and the three-dimensional spatial coordinate range of the surface membrane of each abdominal tissue / organ model, a collision detection algorithm is used to detect the collision between the virtual surgical instrument tactile pen tip and the surface membrane of all abdominal tissue / organ models. If the tip of the virtual surgical instrument tactile pen collides with the surface membrane of any abdominal tissue / organ model, it is considered that the tip of the virtual surgical instrument tactile pen has come into contact with the abdominal tissue / organ model.
9. The adaptive force feedback method of claim 8, wherein, The steps for identifying the type of abdominal tissue / organ model in which contact occurred include: Record the three-dimensional coordinate position of the contact between the virtual surgical instrument tactile pen tip and the abdominal tissue / organ model; Determine the relationship between the three-dimensional coordinate position and the three-dimensional spatial coordinate range of the surface membrane of each abdominal tissue / organ model; If the three-dimensional coordinate position falls within the three-dimensional spatial coordinate range of the surface membrane of the corresponding abdominal tissue / organ model, it is determined that the virtual surgical instrument tactile pen tip has come into contact with the corresponding abdominal tissue / organ model.
10. The adaptive force feedback method of claim 9, wherein, The collision detection algorithm employs one of the following: OBB algorithm, SAT algorithm, or FCL triangular face detection algorithm.
11. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement the adaptive force feedback method as described in any one of claims 1-10.
12. A computer-readable storage medium, the computer-readable storage medium storing a computer program, characterized in that, When the computer program runs on a computer or processor, it causes the computer or processor to perform the adaptive force feedback method as described in any one of claims 1-10.
13. A personalized virtual surgical operation device, characterized in that, Including the electronic device as described in claim 11.