AR HMD Tracking With SLAM and IR Markers for Stable Overlay
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing augmented reality head-mountable devices (HMDs) for medical applications suffer from precision and latency issues, with SLAM methods accumulating errors over time and IR marker tracking having high latency and requiring additional equipment installation.
Innovation Solution
An HMD system that combines IR object tracking with SLAM tracking by updating AR visualization between IR iterations using the HMD pose, leveraging efficient algorithms to maintain high precision and low latency without additional equipment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If SLAM tracking is used for AR visualization, then low latency is achieved, but measurement precision deteriorates due to error accumulation
Solution Approach 1:
The patent combines SLAM tracking and IR marker tracking into a unified system. SLAM provides continuous low-latency pose estimation, while IR markers provide periodic high-precision corrections. The system merges these two tracking methods by using IR marker detections to correct accumulated SLAM errors, achieving both low latency and high precision simultaneously.
2Measurement precision
If IR marker tracking is used for AR visualization, then measurement precision is improved, but loss of time worsens due to high latency
Solution Approach 1:
The system uses periodic IR marker tracking instead of continuous tracking. IR markers are detected at specific intervals to correct SLAM drift, rather than requiring continuous IR camera monitoring. This periodic correction approach maintains high precision when needed while reducing the overall time loss associated with IR processing.
3Measurement precision
If IR markers are used for tracking, then measurement precision is improved, but device complexity increases due to additional equipment requirements
Solution Approach 1:
The HMD device performs multiple functions with its existing components. The IR camera, already present for other purposes, is utilized for marker detection. The processing means handles both SLAM algorithms and IR marker recognition. This multi-functionality approach enables high-precision tracking without adding dedicated external tracking equipment.
4Loss of time
If computational algorithms are optimized for speed, then loss of time is reduced, but measurement precision deteriorates
Solution Approach 1:
The system applies different levels of computational processing to different tracking methods. SLAM uses optimized fast algorithms for continuous tracking, while IR marker processing uses more computationally intensive algorithms only when markers are detected. This partial application of excessive computation maintains precision when needed while minimizing overall processing time.
Data Source
Figure 1~2
Figure 3~4C
Figure 5
AI summary
Augmented reality method comprising the following steps: receiving continuously from a SLAM system in an HMD SLAM data and perform a SLAM fracking, wherein the SLAM fracking based on the SLAM data defects an environment of the HMD in a world coordinate system and simultaneously defects the pose of the HMD in the world coordinate system using a SLAM algorithm, wherein the SLAM fracking at each SLAM iteration determines the pose of the HMD in the world coordinate system at the time of the respective SLAM iteration based on SLAM data received from the SLAM system at the time of the SLAM iteration; performing an IR object fracking comprising at each IR iteration of the IR object fracking: to receive from an IR image sensor in the HMD an IR image captured at a time of the IR iteration; to defect IR object markers in the IR image of the IR iteration; to determine the pose of the object in an HMD coordinate system at the IR iteration based on 2D positions of the IR object markers defected in the IR image of this IR iteration and based on a known spatial relationship of a set of IR object markers attached to an object to be defected, and to determine the pose of the object in the world coordinate system of the IR iteration based on the pose of the object in an HMD coordinate system of the IR iteration and based on the pose of the HMD in the world coordinate system determined in the SLAM fracking; and displaying at each IR iteration an AR visualization on an AR display in the HMD such that the AR visualization appears on an AR position or an AR pose in the world coordinate system depending on the pose of the object in the world coordinate system of the current IR iteration, wherein, until the pose of the object of the next IR iteration is available, the displaying of the AR visualization is updated based on the HMD poses in the world coordinate system obtained in the SLAM fracking such that the AR visualization is displayed stably at the AR position or AR pose in the world coordinate system of the current IR iteration.