Markerless AR Medical Image Registration Using Facial Landmarks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional marker-based augmented reality (AR) navigation in medical imaging is prone to human error due to the precise installation requirements of fiducial markers, which is challenging in surgeries involving soft tissues like the brain or skin where marker placement is difficult.
Innovation Solution
A marker-less medical image processing system using image registration based on facial feature points, combining tomography and 3D imaging to generate a composite image that overlays blood vessel and bone tissue information onto a patient's face in real-time, utilizing algorithms like PPF and ICP for accurate registration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If marker-based AR navigation is used to achieve accurate image registration, then registration precision is improved, but the complexity of operation and susceptibility to human error increases due to precise marker installation requirements
Solution Approach 1:
The patent extracts and removes the fiducial markers from the AR navigation system, transitioning from marker-based to marker-less registration. The system achieves accurate registration by extracting and matching facial feature points (eyes, nose, mouth, ears) directly from the patient's anatomy, eliminating the need for manual marker placement while maintaining registration precision through algorithmic feature point detection and matching
Solution Approach 2:
The system enables self-service registration by automatically detecting and matching facial feature points without requiring manual marker installation. The AR navigation system performs self-calibration by identifying anatomical landmarks and establishing coordinate transformations autonomously, reducing human intervention and potential errors associated with marker placement
2Measurement precision
If fiducial markers are installed on soft tissues like brain or skin for AR navigation, then image registration can be performed, but the reliability decreases due to difficulty in maintaining marker positions during surgery
Solution Approach 1:
The patent removes fiducial markers from the system entirely and replaces them with direct detection of anatomical feature points on soft tissues. By extracting and tracking natural anatomical landmarks (eyes, nose, mouth, ears) rather than artificial markers, the system maintains registration capability while eliminating the reliability issues associated with marker displacement during soft tissue surgery
Solution Approach 2:
Instead of placing artificial markers on the patient's body, the system inverts the approach by using the patient's own anatomical features as registration references. The system projects AR images and detects feature points directly on the soft tissue surfaces, turning the anatomical structures themselves into the registration framework rather than requiring external markers
Data Source
AI summary
Disclosed is an operating method of a medical image processing device using augmented reality (AR) including obtaining, from a first external device, a first image including blood vessel information and bone tissue information according to tomography, obtaining, from a second external device, a second image including a facial structure and a facial curve according to 3D imaging, extracting a facial feature point from facial unique information including the blood vessel information, the bone tissue information, the facial structure, and the facial curve, registering the first image to the second image based on the facial feature point, generating a composite image by performing 3D-modeling on the first image registered with the second image, projecting the composite image onto a face of a patient, and displaying, on a projected image, a first layer corresponding to the blood vessel information and a second layer including the bone tissue information.