Method for acquiring state data of interventional instrument, and device and medium

Abstract

A method for acquiring state data of an interventional instrument, and a device and a medium. The method comprises: constructing a virtual interventional scene corresponding to a real interventional scene; acquiring a control instruction generated by a master controller; on the basis of the control instruction, controlling the movement of an interventional instrument in the real interventional scene, acquiring a corresponding DSA image, and on the basis of the DSA image, obtaining first state data of the interventional instrument in the real interventional scene; on the basis of the control instruction, controlling the movement of a virtual interventional instrument in the virtual interventional scene, and acquiring second state data of the virtual interventional instrument; and fusing the first state data and the second state data, so as to obtain fused state data. The finally obtained fused state data is more accurate and reliable, enabling a doctor to perceive the state of an interventional instrument during surgery more precisely, thereby improving the safety and accuracy of the surgical operation, and providing a more robust basis for surgical risk prediction.

Description

Methods, equipment, and media for acquiring status data of interventional devices

[0001] This application claims priority to Chinese Patent Application No. 202411688079.1, filed on November 22, 2024, entitled “Method, Device and Medium for Acquiring Status Data of Interventional Devices”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of vascular interventional surgical robots, and in particular to a method, device and medium for acquiring status data of interventional devices. Background Technology

[0003] During interventional surgical robotic procedures, the positional information and tip force information of the interventional instruments are crucial data. Surgeons or the master controller can use this data to predict surgical risks. Therefore, the accuracy of this data is paramount. Currently, the method for acquiring this data typically involves analyzing the force feedback system of the end-user robot and DSA (Digital Subtraction Angiography) images. However, this approach is prone to errors. For instance, when analyzing the positional and tip force information of the interventional instruments based on DSA images, because only image information is available, blood vessels may sometimes be mistaken for the interventional instruments, resulting in erroneous data. This can affect the accuracy of the surgeon's or controller's risk prediction.

[0004] Therefore, how to accurately obtain the status data of interventional devices is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] This application provides a method, device, and medium for acquiring status data of interventional devices, aiming to solve the technical problem that existing technologies cannot accurately acquire status data of interventional devices.

[0006] To achieve the aforementioned objectives, the first aspect of this application proposes a method for acquiring state data of interventional devices, comprising the steps of: constructing a virtual interventional scenario corresponding to a real interventional scenario; acquiring control commands generated by a master controller; controlling the movement of the interventional device in the real interventional scenario according to the control commands, and acquiring corresponding digital subtraction angiography images, and obtaining first state data of the interventional device in the real interventional scenario based on the digital subtraction angiography images; controlling the movement of the virtual interventional device in the virtual interventional scenario according to the control commands, and acquiring second state data of the virtual interventional device; and fusing the first state data and the second state data to obtain fused state data.

[0007] Further, after fusing the first state data and the second state data to obtain fused state data, the process includes: correcting the first state data based on the fused state data to correct recognition errors in digital subtraction angiography image analysis; and correcting the second state data based on the fused state data to optimize model parameters and calculation biases in the virtual interventional scenario.

[0008] Furthermore, the steps for constructing a virtual interventional scenario corresponding to a real interventional scenario include: acquiring interventional device data, slave robot data, and preoperative computed tomography angiography data of the patient in the real interventional scenario; constructing a virtual interventional device in the virtual interventional scenario based on the interventional device data; constructing virtual slave robot data in the virtual interventional scenario based on the slave robot data; and constructing a vascular model in the virtual interventional scenario based on the computed tomography angiography data.

[0009] Furthermore, the steps of constructing a vascular model in a virtual interventional scenario based on computed tomography angiography data include: reconstructing a three-dimensional vascular model based on computed tomography angiography data; increasing the vascular wall thickness in the three-dimensional vascular model to obtain a volume model; and importing the volume model into a preset simulation software to obtain a vascular model, wherein the simulation software assigns elastic physical collision parameters to the vascular model.

[0010] Furthermore, the step of fusing the first state data and the second state data to obtain fused state data includes: fusing the first state data and the second state data using a multimodal data fusion method to obtain fused state data.

[0011] Furthermore, the multimodal data fusion method includes a Kalman filtering algorithm, which is used to assign weights based on the credibility of the first state data and the second state data in order to fit accurate fused state data.

[0012] Furthermore, after the step of constructing a virtual interventional scenario corresponding to the real interventional scenario, the method includes: using computer graphics technology and visualization algorithms to generate a visualized virtual interventional scenario to synchronously display the real-time position of the virtual interventional device within the vascular model and the force data at the tip of the virtual interventional device.

[0013] Furthermore, the method for acquiring the status data of the interventional device also includes: acquiring the sensor data of each sensor on the slave robot; determining whether the component corresponding to each sensor data has malfunctioned based on the sensor data; if so, mapping the malfunctioning component to the visualized virtual interventional scene and displaying the fault information through a flashing icon or warning box; if not, displaying the components of the slave robot normally in the virtual interventional scene.

[0014] Further, after fusing the first state data and the second state data to obtain fused state data, the process includes: using the fused state data to predict whether there is a risk in the surgery; if so, generating a reminder message, which includes a visual warning and / or an audible alarm; if not, acquiring new fused state data to continue predicting whether there is a risk in the surgery.

[0015] The second aspect of this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the interventional device status data acquisition method as described above.

[0016] Furthermore, a computer device includes a memory and a processor. The memory stores a computer program, wherein the processor executes the computer program to perform the following steps: constructing a virtual interventional scenario corresponding to a real interventional scenario; acquiring control instructions generated by a master controller; controlling the movement of interventional devices in the real interventional scenario according to the control instructions, and acquiring corresponding digital subtraction angiography images, and obtaining first state data of interventional devices in the real interventional scenario based on the digital subtraction angiography images; controlling the movement of virtual interventional devices in the virtual interventional scenario according to the control instructions, and acquiring second state data of virtual interventional devices; and fusing the first state data and the second state data to obtain fused state data.

[0017] Furthermore, after the processor executes the computer program to fuse the first state data and the second state data to obtain fused state data, it also performs the following steps: correcting the first state data based on the fused state data, and correcting the second state data based on the fused state data.

[0018] Furthermore, when the processor executes the computer program to construct a virtual interventional scenario corresponding to the real interventional scenario, it specifically implements the following steps: acquiring interventional device data, slave robot data, and preoperative computed tomography angiography data of the patient in the real interventional scenario; constructing virtual interventional devices in the virtual interventional scenario based on the interventional device data; constructing virtual slave robot data in the virtual interventional scenario based on the slave robot data; and constructing a vascular model in the virtual interventional scenario based on the computed tomography angiography data.

[0019] Furthermore, when the processor executes the computer program to construct a vascular model in a virtual interventional scenario based on computed tomography angiography data, the specific steps are as follows: reconstruct a three-dimensional vascular model based on computed tomography angiography data; increase the thickness of the vascular wall in the three-dimensional vascular model to obtain a volume model; import the volume model into a preset simulation software to obtain a vascular model, wherein the simulation software assigns elastic physical collision parameters to the vascular model.

[0020] Furthermore, when the processor executes the computer program to fuse the first state data and the second state data to obtain fused state data, the specific steps are as follows: the first state data and the second state data are fused using a multimodal data fusion method to obtain fused state data.

[0021] Furthermore, after the processor executes the computer program to construct a virtual interventional scene corresponding to the real interventional scene, it also performs the following steps: using computer graphics technology and visualization algorithms to generate a visualized virtual interventional scene to synchronously display the real-time position of the virtual interventional device within the vascular model and the force data at the tip of the virtual interventional device.

[0022] Furthermore, when the processor executes the computer program, it also performs the following steps: acquiring data from each sensor on the slave robot; determining whether the component corresponding to each sensor data has malfunctioned based on the sensor data; if so, mapping the malfunctioning component to the visualized virtual intervention scene and displaying the fault information through a flashing icon or warning box; if not, displaying the components of the slave robot normally in the virtual intervention scene.

[0023] Furthermore, after the processor executes the computer program to fuse the first state data and the second state data to obtain fused state data, it also performs the following steps: using the fused state data, predicting whether there is a risk in the surgery; if so, generating a reminder message, which includes a visual warning and / or an audible alarm; if not, acquiring new fused state data and continuing to predict whether there is a risk in the surgery.

[0024] A third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the interventional device status data acquisition method as described above.

[0025] Furthermore, a computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it performs the following steps: constructing a virtual interventional scenario corresponding to a real interventional scenario; acquiring control instructions generated by a master controller; controlling the movement of interventional devices in the real interventional scenario according to the control instructions, and acquiring corresponding digital subtraction angiography images, and obtaining first state data of interventional devices in the real interventional scenario based on the digital subtraction angiography images; controlling the movement of virtual interventional devices in the virtual interventional scenario according to the control instructions, and acquiring second state data of virtual interventional devices; and fusing the first state data and the second state data to obtain fused state data.

[0026] Furthermore, after the processor executes the computer program to fuse the first state data and the second state data to obtain fused state data, it also performs the following steps: correcting the first state data based on the fused state data, and correcting the second state data based on the fused state data.

[0027] Furthermore, when the processor executes the computer program to construct a virtual interventional scenario corresponding to the real interventional scenario, it specifically implements the following steps: acquiring interventional device data, slave robot data, and preoperative computed tomography angiography data of the patient in the real interventional scenario; constructing virtual interventional devices in the virtual interventional scenario based on the interventional device data; constructing virtual slave robot data in the virtual interventional scenario based on the slave robot data; and constructing a vascular model in the virtual interventional scenario based on the computed tomography angiography data.

[0028] Furthermore, when the processor executes the computer program to construct a vascular model in a virtual interventional scenario based on computed tomography angiography data, the specific steps are as follows: reconstruct a three-dimensional vascular model based on computed tomography angiography data; increase the thickness of the vascular wall in the three-dimensional vascular model to obtain a volume model; import the volume model into a preset simulation software to obtain a vascular model, wherein the simulation software assigns elastic physical collision parameters to the vascular model.

[0029] Furthermore, when the processor executes the computer program to fuse the first state data and the second state data to obtain fused state data, the specific steps are as follows: the first state data and the second state data are fused using a multimodal data fusion method to obtain fused state data.

[0030] Furthermore, after the processor executes the computer program to construct a virtual interventional scene corresponding to the real interventional scene, it also performs the following steps: using computer graphics technology and visualization algorithms to generate a visualized virtual interventional scene to synchronously display the real-time position of the virtual interventional device within the vascular model and the force data at the tip of the virtual interventional device.

[0031] Furthermore, when the processor executes the computer program, it also performs the following steps: acquiring data from each sensor on the slave robot; determining whether the component corresponding to each sensor data has malfunctioned based on the sensor data; if so, mapping the malfunctioning component to the visualized virtual intervention scene and displaying the fault information through a flashing icon or warning box; if not, displaying the components of the slave robot normally in the virtual intervention scene.

[0032] Beneficial Effects: This application provides a method, device, and medium for acquiring the status data of interventional devices. The method constructs a virtual interventional scenario, simulating the surgical procedure in a virtual environment and providing comparison and reference for real surgery. Based on control commands from a master controller, the synchronous movement of the device in both the real and virtual interventional scenarios is achieved, acquiring status data for each scenario. Fusing the two data allows for full utilization of the rich physical simulation information in the virtual interventional scenario and the actual imaging information in the real interventional scenario, complementing each other's strengths. The resulting fused status data is more accurate and reliable, helping physicians to more precisely grasp the status of interventional devices during surgery, improving the safety and accuracy of surgical procedures, providing stronger evidence for surgical risk prediction, and thus enhancing the overall quality and success rate of interventional surgery. Attached Figure Description

[0033] Figure 1 is a flowchart illustrating a method for acquiring status data of an interventional device according to an embodiment of the invention;

[0034] Figure 2 is a flowchart illustrating a method for acquiring status data of an interventional device according to an embodiment of the invention;

[0035] Figure 3 is a schematic diagram of the structure of a computer device according to an embodiment of the invention.

[0036] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0038] Referring to Figure 1, this application proposes a method for acquiring the status data of an interventional device, including the following steps:

[0039] S1. Construct a virtual intervention scenario that corresponds to the real intervention scenario;

[0040] S2. Obtain the control commands generated by the master controller;

[0041] S3. Control the movement of the interventional device in the real interventional scenario according to the control command, and acquire the corresponding digital subtraction angiography images, and obtain the first state data of the interventional device in the real interventional scenario based on the digital subtraction angiography images.

[0042] S4. Control the movement of the virtual interventional device in the virtual interventional scenario according to the control command, and obtain the second state data of the virtual interventional device;

[0043] S5. The first state data and the second state data are fused to obtain fused state data.

[0044] For step S1, the aforementioned real interventional scenario refers to the actual interventional scenario when the interventional surgical robot performs the interventional task. An interventional surgical robot generally includes a master controller and a slave robot. The relationship between the master controller and the slave robot is as follows: the master controller controls the actions of the slave robot, and the slave robot, in turn, feeds back relevant signals generated by its actions to the master controller. In this embodiment, using surgical-related data acquired preoperatively, including interventional device data, slave robot data, and CTA (Computed Tomography Angiography) data, a virtual interventional scenario similar to the real surgical scenario is constructed in a virtual environment. Interventional device data is used to construct virtual interventional devices, slave robot data is used to construct a virtual slave robot, and CTA data is used to construct a vascular model, thereby providing a foundation for subsequent simulation operations and data comparison.

[0045] In step S2, the surgeon operates the master controller with force feedback, which generates corresponding control commands. These commands serve as the command signals for the entire surgical procedure, determining the movement direction, speed, rotation, and other operational parameters of the instruments in both the real and virtual interventional scenarios. In this embodiment, the control signals from the master controller are transmitted to the slave robot and the virtual interventional scenario respectively via a CAN bus.

[0046] For step S3, in a real interventional scenario, the slave robot receives control commands and executes actions to move the interventional device, including controlling its axial and radial movements. Simultaneously, the DSA device acquires images in real time. Based on image recognition algorithms, it extracts information such as the two-dimensional position and tip bending state of the interventional device from the DSA two-dimensional images, and can estimate the force on the tip of the interventional device. This information constitutes the first-state data. For example, in vascular interventional surgery, the real-time position of the interventional device within the blood vessel and the force exerted on its tip by factors such as the blood vessel wall are both considered first-state data.

[0047] In step S4, within the virtual interventional scenario, the virtual environment drives the virtual interventional device to move according to the master control commands. In this embodiment, the virtual interventional scenario is constructed using SOFA (Simulation Open Framework Architecture, an open-source software framework for real-time physics simulation). Through the elastic physics engine and collision algorithm within the SOFA simulation software, the three-dimensional position information and the force state at the tip of the virtual interventional device are obtained using the built-in function interfaces of the software function library. This information constitutes the second state data. Examples include the motion trajectory and position coordinates of the virtual interventional device in the virtual blood vessel model, as well as the simulated magnitude and direction of the force at the tip.

[0048] For step S5, a multimodal data fusion method, such as the Kalman filter algorithm, is used to fuse the first-state data and the second-state data to obtain fused state data. This fused state data generally includes more precise information such as the force and position of the interventional device's tip. Taking the tip force as an example, in the virtual interventional environment, a tip force A is obtained through the API program interface and the elasticity simulation environment, and another tip force B is obtained based on the DSA image. The Kalman filter method assigns different weights according to the credibility of different data to fit a more accurate tip force value.

[0049] In this implementation, a data acquisition system based on virtual-reality interaction was constructed. By building a virtual interventional scenario, the surgical procedure can be simulated in a virtual environment, providing comparison and reference for real surgery. Control commands from the master controller synchronize the movement of instruments in both the real and virtual interventional scenarios, acquiring state data for each scenario. Integrating these two systems allows for the full utilization of the rich physical simulation information in the virtual interventional scenario and the actual image information in the real interventional scenario, achieving complementary advantages. For example, the three-dimensional position information and force calculations in the virtual interventional scenario can supplement the deficiencies in spatial information and force prediction of the two-dimensional images in the real interventional scenario, while the actual images in the real interventional scenario can correct potential model biases in the virtual interventional scenario. The resulting fused state data is more accurate and reliable, helping doctors to more precisely grasp the state of interventional instruments during surgery, improving the safety and accuracy of surgical operations, providing stronger evidence for surgical risk prediction, and thus improving the overall quality and success rate of interventional surgery.

[0050] Referring to Figure 2, in one embodiment, after step S5 of fusing the first state data and the second state data to obtain fused state data, the method includes:

[0051] S6. Correct the first state data based on the fused state data; and

[0052] S7. Correct the second state data based on the fusion state data.

[0053] For step S6, after obtaining the fused state data, the first state data obtained in the actual interventional scenario is corrected using the accurate information in the fused state data as a reference. During interventional device segmentation, traditional algorithms based solely on AI models may make recognition errors due to having only image information, such as mistaking a blood vessel for an interventional device. The fused state data, however, includes physical attribute information from the virtual environment (such as the collision between the interventional device and the blood vessel model). By comparing the two-dimensional image obtained in the virtual environment based on registration (annotated with the interventional device projection and incorporating physical attributes) with the actual DSA image, erroneous judgments regarding interventional device identification, position, and force in the first state data can be corrected, thus making the first state data more accurate.

[0054] For step S7, the second state data in the virtual interventional scenario is also optimized based on the fused state data. Although the virtual environment can simulate and calculate the state data of the interventional device, there may be problems such as model errors or calculation deviations. The fused state data integrates the actual situation in the real interventional scenario, such as the motion feedback of the real interventional device. Using this information, the parameters in the virtual interventional device model can be adjusted to correct the second state data such as the three-dimensional position and tip force of the virtual interventional device, making it more consistent with the actual surgical situation.

[0055] In this embodiment, by correcting the first and second state data based on fused state data, the accuracy of both types of state data can be continuously improved. For the first state data, the corrected result can provide doctors with more reliable decision-making basis in real surgical operations, reducing surgical risks caused by erroneous data. For the second state data, the optimized virtual interventional scenario data can more accurately simulate the surgical process, improving the guiding significance of the virtual environment for real surgery. For example, in subsequent surgical simulations and rehearsals, the corrected virtual interventional scenario data can more realistically reflect the situations that may occur during surgery, helping doctors better plan surgical procedures, improving the success rate and safety of the surgery, while also making the virtual-reality state synchronization more accurate, further enhancing the intelligent perception capability of the entire interventional surgical robot system.

[0056] In one embodiment, step S1 of constructing a virtual intervention scenario corresponding to a real intervention scenario includes:

[0057] S11. Acquire interventional device data, end-device robot data, and patient CTA data from real interventional scenarios;

[0058] S12. Based on the interventional device data, construct the virtual interventional device in the virtual interventional scenario;

[0059] S13. Construct virtual slave robot data in the virtual intervention scenario based on the slave robot data;

[0060] S14. Based on the computed tomography angiography data, construct a vascular model in the virtual interventional scenario.

[0061] The aforementioned interventional device data encompasses information such as the device's model, specifications, and physical characteristics. This information is crucial for accurately constructing virtual interventional devices in a virtual environment. The slave robot data includes specific parameters of the robot's delivery cart, delivery rollers, delivery rails, and other structural components, used to construct the virtual slave robot to achieve synchronized motion simulation between the virtual and real robots. Preoperative CTA data provides detailed structural information about the patient's blood vessels, serving as the foundational data for constructing vascular models. For example, for a specific vascular interventional procedure, the interventional device data specifies the device's length and diameter, the slave robot data determines how the virtual robot simulates the motion of the real robot, and the CTA data clearly shows the direction and branches of the patient's blood vessels.

[0062] Using acquired interventional device data, a virtual interventional device model is created in a virtual environment that is similar to the real interventional device in terms of shape and physical properties. This model can simulate movement according to control commands in the virtual interventional scenario, and its interaction with virtual vascular models accurately reflects the situation during real surgery. For example, based on the material parameters such as the elastic modulus of the real interventional device, corresponding properties are set in the virtual model to make its behavior, such as stress deformation, consistent with the real device in the virtual environment.

[0063] Based on data from the slave robot, a virtual slave robot model and its associated motion control logic are constructed. This virtual slave robot maintains the same motion and state as the real slave robot in the virtual intervention scenario, accepting the same control commands and providing similar state information. For example, the movement of the delivery cart of the real slave robot is accurately simulated in the virtual slave robot model, enabling a comprehensive display of the surgical robot system's operational status in the virtual environment.

[0064] Based on CTA data, a vascular model is constructed for the virtual interventional scenario. First, a three-dimensional vascular model is reconstructed from the CTA data, representing the three-dimensional structure of the blood vessels. Then, the vessel wall thickness is increased in the three-dimensional model to create a volumetric model. This step helps to more accurately simulate the physical properties of the blood vessels, such as elasticity. Finally, the volumetric model is imported into pre-set simulation software (such as SOFA simulation software), and elastic physical collision parameters are assigned to the vascular model, thus constructing a complete vascular model. This constructed vascular model can perform realistic collision simulations with virtual interventional instruments, providing a more realistic environment for surgical simulation.

[0065] In this embodiment, by accurately acquiring and utilizing various data from real interventional scenarios to construct corresponding elements in the virtual interventional scenario, a high degree of correspondence between the virtual and real interventional scenarios is achieved. The construction of virtual interventional instruments forms a digital twin system of the real interventional scenario. The construction of virtual slave robot data helps optimize the robot's control algorithms and motion planning. Furthermore, the accurate vascular model not only simulates the deformation and stress of blood vessels during surgery but also provides accurate environmental references for the operation of interventional instruments, improving the precision and safety of the surgery. The construction of the entire virtual interventional scenario provides doctors with an intuitive and realistic surgical platform, helping to improve their understanding and control of the surgical process, thereby increasing the success rate of the surgery.

[0066] In one implementation, step S14, which involves constructing a vascular model in the virtual interventional scenario based on the computed tomography angiography data, includes:

[0067] S141. Reconstruct a three-dimensional vascular model based on the computed tomography angiography data;

[0068] S142. Increase the thickness of the blood vessel wall in the three-dimensional blood vessel model to obtain a volume model;

[0069] S143. Import the volume model into a preset simulation software to obtain the blood vessel model, wherein the simulation software assigns elastic physical collision parameters to the blood vessel model.

[0070] CTA data contains tomographic information of the patient's blood vessels. Using specific image processing algorithms and 3D reconstruction techniques, this two-dimensional tomographic information is transformed into a three-dimensional vascular model. This process requires precise extraction and reconstruction of information such as vascular contours and branching structures from the CTA data to generate a 3D model that accurately reflects the true morphology of the patient's blood vessels. For example, for complex cerebral vascular structures, the reconstruction algorithm needs to identify the origin, direction, and connection relationships of each vascular branch to construct a complete 3D cerebral vascular model. Based on the reconstructed 3D vascular model, the thickness of the vessel wall is increased, transforming it from a simple surface model into a volumetric model. The importance of this step lies in the fact that real vessel walls have a certain thickness and physical properties; the increased thickness of the volumetric model allows for a more accurate simulation of the mechanical behavior of the vessel wall under the influence of interventional instruments, such as elastic deformation and collision response. For example, in vascular interventional surgery, contact and interaction between interventional instruments and the vessel wall are common, and the volumetric model can more realistically reflect this interaction process. The constructed volumetric model is then imported into selected simulation software (such as SOFA simulation software), which has powerful physical simulation capabilities. During the import process, the software assigns elastic physical collision parameters to the vascular model based on preset parameter configurations, including material parameters such as the elastic modulus and Poisson's ratio of the vessel wall, as well as collision detection and response algorithms. This allows the vascular model to realistically simulate elastic collisions with virtual interventional instruments in the virtual environment. When the virtual interventional instrument comes into contact with or collides with the vascular model, the software can accurately calculate the deformation and stress on the vessel wall based on the set parameters, providing a highly realistic vascular environment for surgical simulation.

[0071] This embodiment proposes a specific construction process focused on vascular models, which plays a crucial role in improving the realism and practicality of virtual interventional scenarios. Through accurate 3D vascular model reconstruction, doctors can be provided with detailed information about the patient's vascular structure, aiding in surgical planning and risk assessment. Increasing the thickness of the vessel wall to obtain a volumetric model makes the virtual blood vessel more physically similar to a real blood vessel, improving the accuracy of surgical simulation. Furthermore, importing the volumetric model into simulation software and assigning elastic physical collision parameters enables realistic interactive simulation of blood vessels and interventional instruments in the virtual environment. Doctors can more intuitively observe the operation of interventional instruments within the blood vessel in the virtual interventional scenario, assess the impact of different operations on the blood vessel, identify potential risks in advance, optimize surgical plans, and thus improve the safety and success rate of interventional surgery.

[0072] In one embodiment, step S5, which fuses the first state data and the second state data to obtain fused state data, includes:

[0073] S51. The first state data and the second state data are fused using a multimodal data fusion method to obtain the fused state data.

[0074] For step S51, the multimodal data fusion method is a technical means of comprehensively processing data from different data sources or different modes to obtain more accurate and comprehensive information. In this embodiment, the first state data comes from DSA image analysis and force detection at the end of interventional instruments in real interventional scenarios, while the second state data is obtained through simulation calculations in virtual interventional scenarios. These data from different sources have their own characteristics and advantages, but also certain limitations. For example, DSA images provide two-dimensional image information in real surgical scenarios, but may not be accurate enough in terms of spatial positioning and force calculation; while data from virtual interventional scenarios can perform accurate three-dimensional calculations, there may be problems with the model not completely matching the actual situation. Multimodal data fusion methods such as the Kalman filtering algorithm are used to fuse the two state data. The Kalman filtering algorithm is based on the state equation of a linear system and estimates the optimal state of the system by predicting the system state and correcting the measured values. In this application, it takes first-state data (such as the head force B estimated based on DSA images) and second-state data (such as the head force A obtained through the virtual environment API) as inputs, assigns different weights based on the reliability of the two types of data (such as the calculation accuracy of the data in the virtual environment, the resolution of the DSA images, etc.), and then calculates a fused head force value. A similar fusion method is used for other state data such as the position information of interventional devices to obtain fused state data.

[0075] In this embodiment, by integrating data from both real and virtual interventional scenarios, the advantages of both types of data are fully utilized, while their respective shortcomings are compensated for. The fused state data can provide doctors with more accurate surgical decision-making basis. For example, during the operation of interventional instruments, more accurate prediction of the force applied to the tip can help doctors better control the operating force and avoid unnecessary damage to blood vessels; more precise tip position information helps improve the positioning accuracy of interventional instruments within blood vessels and reduce operational errors. This not only improves the safety of the surgery but also enhances surgical efficiency, shortens the operation time, and makes interventional surgical robots more reliable and intelligent in complex surgical operations, further promoting the development of interventional surgical technology.

[0076] In one embodiment, after step S1 of constructing a virtual intervention scenario corresponding to the real intervention scenario, the following is included:

[0077] S10. Using computer graphics technology and visualization algorithms, generate visualized virtual intervention scenes.

[0078] In step S10, after constructing the virtual interventional scene corresponding to the real interventional scenario, computer graphics technology and visualization algorithms are used to display various elements in the virtual interventional scene, such as virtual interventional instruments, virtual blood vessel models, and virtual end-effector robots, in an intuitive graphical interface. This includes converting the 3D model into a 2D image displayed on the screen, while enhancing the realism and three-dimensionality of the scene through effects such as color and lighting. For example, the virtual blood vessel model can be represented by colors such as red or blue, and the virtual interventional instrument can be displayed in gray or silver, and dynamically rendered according to its position and state in the scene. For example, when the interventional instrument moves within the blood vessel, its position in the visualization interface is updated in real time. Furthermore, various data can be displayed on the screen, such as the specific values ​​of the force applied to the tip.

[0079] Visualizing virtual interventional scenarios has multiple important implications. For physicians, a visualized virtual interventional scenario provides an intuitive platform for surgical rehearsal and real-time monitoring. Before surgery, physicians can observe instrument operation and vascular responses within the virtual scenario to plan surgical pathways, assess surgical risks, and develop optimized surgical plans. During the surgery, the visualization interface can display the interaction between virtual instruments and the vascular model in real time, helping physicians better understand the surgical progress and adjust their operational strategies promptly. Simultaneously, visualized virtual interventional scenarios facilitate communication and collaboration between physicians and other medical personnel, enabling them to jointly discuss improvements and optimizations to the surgical plan. From the perspective of the overall surgical system, visualization enhances the user-friendliness of human-computer interaction, making it easier for physicians to accept and operate interventional surgical robot systems, improving the system's practicality and ease of use, and contributing to the widespread application of interventional surgical robots in clinical practice.

[0080] In one embodiment, the above-mentioned method for acquiring the status data of interventional devices further includes:

[0081] S7. Acquire data from various sensors on the slave robot;

[0082] S8. Based on the data from each sensor, determine whether the component corresponding to each sensor data has malfunctioned;

[0083] S91. If so, the faulty component will be mapped to the visualized virtual intervention scene.

[0084] S92. If not, the components of the slave robot are displayed normally in the virtual intervention scenario.

[0085] The end-user robot is equipped with various sensors, such as position sensors and force sensors, which collect real-time operational status data from various components of the robot. The data collected by each sensor is analyzed using preset fault diagnosis algorithms and thresholds to determine whether the corresponding component has malfunctioned. For example, if the force sensor data shows a force value exceeding the normal operating range, or an abnormal force trend (such as a sudden increase or decrease), it may indicate a fault risk in the component related to that force sensor, such as the transmission device. If the position sensor data shows excessive deviation or unstable fluctuations, it may suggest inaccurate positioning or movement stagnation in the corresponding moving parts. Once a component malfunction is identified, it is prominently displayed in the virtual interventional scenario using specific mapping algorithms and graphic labeling technology. For instance, in the visualization interface, the faulty motor is marked with a flashing red icon, or a warning box is displayed around the faulty component, along with the fault type and relevant prompts. This allows doctors observing the virtual interventional scenario to intuitively see the location and type of the fault and take timely measures, such as pausing surgery, performing emergency repairs, or adjusting the surgical strategy. Of course, if all sensor data are normal, it means that no component is malfunctioning. Therefore, in the virtual intervention scenario, the components of the slave robot can be displayed normally without special marking.

[0086] This embodiment, by acquiring sensor data from the slave robot in real time and performing fault diagnosis, can promptly identify potential faults and prevent them from worsening during surgery, thus avoiding serious harm to the patient. Mapping faulty components to a visualized virtual interventional scenario provides doctors with an intuitive and convenient way to display fault information, enabling them to quickly locate faults without complex data analysis, greatly improving the efficiency of fault handling. During surgery, this rapid and accurate fault feedback mechanism helps doctors adjust surgical procedures in a timely manner, reducing surgical risks caused by robot malfunctions and ensuring the smooth progress of the surgery. Simultaneously, it facilitates real-time monitoring and maintenance of the robot by technical personnel within the surgical team, improving the stability and maintainability of the entire surgical system and promoting the safety and effectiveness of interventional surgical robots in clinical applications.

[0087] In one embodiment, after step S5 of fusing the first state data and the second state data to obtain fused state data, the following steps are included:

[0088] S51. Using the fusion state data, predict whether there are risks associated with the surgery;

[0089] S52. If so, generate a reminder message;

[0090] S53. If not, obtain new fusion state data and continue to predict whether there is a risk in the surgery.

[0091] Fusion status data encompasses comprehensive information about the interventional device in both real and virtual interventional scenarios, such as tip position, stress levels, and movement trajectory. Analyzing this data allows for the assessment of potential risks during the procedure. For instance, based on the fused tip stress data, if the stress value approaches or exceeds the limits that the blood vessel wall can withstand, or if the stress changes abnormally drastically, it may indicate a risk of blood vessel rupture. Regarding the device's position and movement trajectory, if it deviates from the pre-set safe operating area, or is too close to the blood vessel wall or other tissues and organs, it may also lead to risks such as collision or perforation. Furthermore, trend analysis of fusion status data over a period of time, such as sudden changes in the speed of the interventional device's movement or a continuous increase in stress, can also help predict surgical risks in advance.

[0092] When a surgical risk is predicted, the system automatically generates an alert. This alert can take various forms, such as displaying a prominent warning box on the aforementioned visual screen showing the risk type and severity; simultaneously issuing an audible alarm to attract the doctor's attention; or providing detailed text prompts explaining the cause of the risk and possible countermeasures. For example, if a risk of blood vessel rupture is predicted, the warning box might display "Risk of blood vessel rupture: Excessive force on the tip, please adjust the operating force," accompanied by an audible alarm and specific operational suggestions in the adjacent prompt bar, such as "Slowly withdraw the interventional device to reduce pressure on the tip." When no risk is predicted, the system maintains its current state and continues to acquire new fusion status data to predict surgical risks, without issuing any other corresponding alerts or alarms.

[0093] Utilizing fused state data for risk prediction fully leverages the advantages of multimodal data fusion, improving the accuracy and reliability of risk prediction. Timely generation of alerts allows surgeons to be aware of potential risks during surgery immediately, enabling them to react quickly, adjust surgical procedures, and avoid risk events. This helps reduce the incidence of surgical complications, increase surgical success rates, and protect patient safety. Simultaneously, this risk prediction and alert mechanism reduces the psychological stress on surgeons during procedures, allowing them to focus more on the surgical operation and improve surgical quality. Within the entire interventional surgical robot system, this function enhances the system's intelligence and safety, driving interventional surgery towards greater precision and safety.

[0094] Referring to Figure 3, this application embodiment also provides a computer device, the internal structure of which can be as shown in Figure 3. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor in this computer device is designed to provide computing and control capabilities. The memory of the computer device includes non-volatile storage media, internal memory, and volatile storage media. The non-volatile storage media stores operating devices, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used to store data related to this application, etc. The network interface of the computer device is used for communication with external terminals via network connection. Furthermore, the above-mentioned computer device may also be equipped with input devices and a display screen, etc. When the aforementioned computer program is executed by a processor, it implements a method for acquiring the state data of an interventional device, comprising the following steps: constructing a virtual interventional scene corresponding to a real interventional scene; generating control commands through a master controller; controlling the movement of the interventional device in the real interventional scene according to the control commands, and acquiring corresponding DSA images, and obtaining first state data of the interventional device in the real interventional scene based on the DSA images; controlling the movement of the virtual interventional device in the virtual interventional scene according to the control commands, and acquiring second state data of the virtual interventional device; and fusing the first state data and the second state data to obtain fused state data.

[0095] One embodiment of this application also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, storing a computer program thereon. When the computer program is executed by a processor, it implements a method for acquiring the state data of an interventional device, including the following steps: constructing a virtual interventional scene corresponding to a real interventional scene; generating control instructions through a master controller; controlling the movement of the interventional device in the real interventional scene according to the control instructions, and acquiring corresponding DSA images, and obtaining first state data of the interventional device in the real interventional scene based on the DSA images; controlling the movement of a virtual interventional device in the virtual interventional scene according to the control instructions, and acquiring second state data of the virtual interventional device; fusing the first state data and the second state data to obtain fused state data.

[0096] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media provided in this application and in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0097] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for acquiring status data of an interventional device, wherein, The method includes the following steps: Construct virtual intervention scenarios that correspond to real intervention scenarios; Obtain control commands generated by the master controller; The control commands control the movement of the interventional device in the real interventional scenario and acquire the corresponding digital subtraction angiography images. Based on the digital subtraction angiography images, the first state data of the interventional device in the real interventional scenario is obtained. Control the movement of the virtual interventional device in the virtual interventional scenario according to the control command, and obtain the second state data of the virtual interventional device; The first state data and the second state data are fused together to obtain fused state data.

2. The method for acquiring status data of interventional devices according to claim 1, wherein, After the step of fusing the first state data and the second state data to obtain the fused state data, the following steps are included: The first state data is corrected based on the fused state data to correct recognition errors in digital subtraction angiography image analysis; and the second state data is corrected based on the fused state data to optimize model parameters and calculation biases in the virtual interventional scenario.

3. The method for acquiring status data of interventional devices according to claim 1, wherein, The steps for constructing a virtual intervention scenario corresponding to the real intervention scenario include: Acquire interventional device data, end-device robot data, and preoperative computed tomography angiography data of patients in real interventional scenarios; Based on the interventional device data, the virtual interventional device in the virtual interventional scenario is constructed; Based on the slave robot data, construct virtual slave robot data in the virtual intervention scenario; Based on the computed tomography angiography data, a vascular model in the virtual interventional scenario is constructed.

4. The method for acquiring status data of interventional devices according to claim 3, wherein, The step of constructing a vascular model in the virtual interventional scenario based on the computed tomography angiography data includes: Based on the computed tomography angiography data, a three-dimensional vascular model is reconstructed. Increase the thickness of the blood vessel wall in the three-dimensional blood vessel model to obtain a volume model; The volume model is imported into a preset simulation software to obtain the blood vessel model, wherein the simulation software assigns elastic physical collision parameters to the blood vessel model.

5. The method for acquiring status data of interventional devices according to claim 1, wherein: The step of fusing the first state data and the second state data to obtain fused state data includes: The first state data and the second state data are fused using a multimodal data fusion method to obtain the fused state data.

6. The method for acquiring status data of interventional devices according to claim 5, wherein: The multimodal data fusion method includes a Kalman filtering algorithm, which is used to assign weights based on the reliability of the first state data and the second state data in order to fit accurate fused state data.

7. The method for acquiring status data of interventional devices according to claim 3 or 4, wherein: Following the step of constructing a virtual intervention scenario corresponding to the real intervention scenario, the following steps are included: Using computer graphics technology and visualization algorithms, a visualized virtual interventional scene is generated to synchronously display the real-time position of the virtual interventional device within the vascular model and the force data at the tip of the virtual interventional device.

8. The method for acquiring status data of interventional devices according to claim 7, wherein: The method further includes: Acquire data from various sensors on the slave robot; Based on the data from each sensor, determine whether the component corresponding to each sensor data has malfunctioned; If so, the faulty component will be mapped to the visualized virtual intervention scene, and the fault information will be displayed by flashing icons or warning boxes; If not, the components of the slave robot will be displayed normally in the virtual intervention scenario.

9. The method for acquiring status data of an interventional device according to any one of claims 1 to 6, wherein: After fusing the first state data and the second state data to obtain the fused state data, the following steps are included: Using the fused state data, the potential risks of the surgery can be predicted; If so, a reminder message is generated, which includes a visual alert and / or an audible alarm; If not, new fusion state data is obtained to continue predicting whether there are risks associated with the surgery.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein... When the processor executes the computer program, it performs the following steps: Construct virtual intervention scenarios that correspond to real intervention scenarios; Obtain control commands generated by the master controller; The control commands control the movement of the interventional device in the real interventional scenario and acquire the corresponding digital subtraction angiography images. Based on the digital subtraction angiography images, the first state data of the interventional device in the real interventional scenario is obtained. Control the movement of the virtual interventional device in the virtual interventional scenario according to the control command, and obtain the second state data of the virtual interventional device; The first state data and the second state data are fused together to obtain fused state data.

11. The computer device according to claim 10, wherein, When the processor executes the computer program to implement the step of constructing a virtual intervention scenario corresponding to the real intervention scenario, the following steps are specifically implemented: Acquire interventional device data, end-device robot data, and preoperative computed tomography angiography data of patients in real interventional scenarios; Based on the interventional device data, the virtual interventional device in the virtual interventional scenario is constructed; Based on the slave robot data, construct virtual slave robot data in the virtual intervention scenario; Based on the computed tomography angiography data, a vascular model in the virtual interventional scenario is constructed.

12. The computer device according to claim 10, wherein, When the processor executes the computer program to implement the step of constructing a vascular model in the virtual interventional scenario based on the computed tomography angiography data, the specific steps are as follows: Based on the computed tomography angiography data, a three-dimensional vascular model is reconstructed. Increase the thickness of the blood vessel wall in the three-dimensional blood vessel model to obtain a volume model; The volume model is imported into a preset simulation software to obtain the blood vessel model, wherein the simulation software assigns elastic physical collision parameters to the blood vessel model.

13. The computer device according to claim 11 or 12, wherein, After the processor executes the computer program and implements the step of constructing a virtual intervention scene corresponding to the real intervention scene, it also implements the following steps: Using computer graphics technology and visualization algorithms, a visualized virtual interventional scene is generated to synchronously display the real-time position of the virtual interventional device within the vascular model and the force data at the tip of the virtual interventional device.

14. The computer device according to claim 13, wherein, When the processor executes the computer program, it also performs the following steps: Acquire data from various sensors on the slave robot; Based on the data from each sensor, determine whether the component corresponding to each sensor data has malfunctioned; If so, the faulty component will be mapped to the visualized virtual intervention scene, and the fault information will be displayed by flashing icons or warning boxes; If not, the components of the slave robot will be displayed normally in the virtual intervention scenario.

15. The computer device according to any one of claims 10 to 12, wherein, When the processor executes the computer program, after implementing the step of fusing the first state data and the second state data to obtain fused state data, it also implements the following steps: Using the fused state data, the potential risks of the surgery can be predicted; If so, a reminder message is generated, which includes a visual alert and / or an audible alarm; If not, new fusion state data is obtained to continue predicting whether there are risks associated with the surgery.

16. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by the processor, it performs the following steps: Construct virtual intervention scenarios that correspond to real intervention scenarios; Obtain control commands generated by the master controller; The control commands control the movement of the interventional device in the real interventional scenario and acquire the corresponding digital subtraction angiography images. Based on the digital subtraction angiography images, the first state data of the interventional device in the real interventional scenario is obtained. Control the movement of the virtual interventional device in the virtual interventional scenario according to the control command, and obtain the second state data of the virtual interventional device; The first state data and the second state data are fused together to obtain fused state data.

17. The computer-readable storage medium of claim 16, wherein, When the processor executes the computer program to implement the step of constructing a virtual intervention scenario corresponding to the real intervention scenario, the following steps are specifically implemented: Acquire interventional device data, end-device robot data, and preoperative computed tomography angiography data of patients in real interventional scenarios; Based on the interventional device data, the virtual interventional device in the virtual interventional scenario is constructed; Based on the slave robot data, construct virtual slave robot data in the virtual intervention scenario; Based on the computed tomography angiography data, a vascular model in the virtual interventional scenario is constructed.

18. The computer-readable storage medium according to claim 16, wherein, When the processor executes the computer program to implement the step of constructing a vascular model in the virtual interventional scenario based on the computed tomography angiography data, the specific steps are as follows: Based on the computed tomography angiography data, a three-dimensional vascular model is reconstructed. Increase the thickness of the blood vessel wall in the three-dimensional blood vessel model to obtain a volume model; The volume model is imported into a preset simulation software to obtain the blood vessel model, wherein the simulation software assigns elastic physical collision parameters to the blood vessel model.

19. The computer-readable storage medium according to claim 17 or 18, wherein, After the processor executes the computer program and implements the step of constructing a virtual intervention scene corresponding to the real intervention scene, it also implements the following steps: Using computer graphics technology and visualization algorithms, a visualized virtual interventional scene is generated to synchronously display the real-time position of the virtual interventional device within the vascular model and the force data at the tip of the virtual interventional device.

20. The computer-readable storage medium according to claim 19, wherein, When the processor executes the computer program, it also performs the following steps: Acquire data from various sensors on the slave robot; Based on the data from each sensor, determine whether the component corresponding to each sensor data has malfunctioned; If so, the faulty component will be mapped to the visualized virtual intervention scene, and the fault information will be displayed by flashing icons or warning boxes; If not, the components of the slave robot will be displayed normally in the virtual intervention scenario.

Images