AR Skeletal Overlay With Multimodal 3D Fracture Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current medical imaging technologies, such as X-ray, CT, and MRI, fail to provide real-time visual guidance during fracture diagnosis and treatment, leading to misdiagnosis, inaccurate treatment plans, and increased surgical complexity and risk.
Innovation Solution
A method and system using augmented reality (AR) for overlay presentation of skeletal images, integrating CT, X-ray, and light-field imaging data through a feature fusion network with ray consistency constraints to reconstruct a 3D model, which is aligned and calibrated with the actual surgical scene.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional two-dimensional imaging techniques (X-ray, CT, MRI) are used for fracture diagnosis, then imaging data can be obtained, but depth perception is lost and three-dimensional structure cannot be accurately perceived
Solution Approach 1:
The patent transforms traditional two-dimensional imaging data into three-dimensional visual information by parsing ray models from light-field images and fusing them with CT and X-ray data. This dimensional transition enables doctors to directly perceive the three-dimensional structure of fractures without relying on spatial imagination, thereby resolving the contradiction between diagnosis accuracy and depth perception loss.
2Measurement precision
If multiple cross-sectional images are used to reconstruct three-dimensional structure, then more detailed information is obtained, but the process is time-consuming and error-prone
Solution Approach 1:
The patent replaces the manual process of piecing together multiple cross-sectional images with an automated computer-based reconstruction system. By using ray model parsing and multimodal feature fusion networks, the system automatically generates three-dimensional models from imaging data, eliminating time-consuming manual reconstruction while maintaining high accuracy.
3Ease of manufacture
If preoperative imaging data is used to devise surgical plans, then treatment planning is possible, but real-time visual guidance during surgery is not provided
Solution Approach 1:
The patent performs preliminary action by reconstructing three-dimensional models and parsing ray models before surgery. These preprocessed data are then stored and ready for immediate use during the surgical procedure, enabling real-time visual guidance through augmented reality overlays without requiring manual processing during the operation.
4Productivity
If doctors rely on experience and memory during surgery, then surgical decisions can be made, but surgical complexity and risk increase
Solution Approach 1:
The patent introduces an intermediary system consisting of augmented reality glasses and ray model parsing modules. This intermediary provides real-time visual guidance and three-dimensional anatomical information directly to the surgeon during the procedure, reducing reliance on memory and experience while improving surgical reliability and reducing errors.
Data Source
AI summary
The present disclosure provides a method and system for overlay presentation of a skeletal image based on augmented reality, relating to the technical field of smart medical systems. The method includes obtaining fracture imaging data for a fracture region of a patient, the fracture imaging data including a CT image, an X-ray image, and a light-field image; parsing a ray model corresponding to the light-field image, the ray model providing spatial propagation information of a light in the fracture region of the patient; determining a multimodal fusion feature corresponding to the fracture imaging data based on the ray model and a feature fusion network; reconstructing a three-dimensional model of a fracture part based on the multimodal fusion feature; and aligning and calibrating the reconstructed three-dimensional model of the fracture part with the fracture region of the patient in an actual surgical scene.