HMD Camera-Inertial Sensor Calibration via Marker Pose Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Head-mounted displays (HMDs) face challenges in providing an immersive augmented reality experience due to inaccurate object tracking caused by limited computational resources and the complexity of integrating cameras and inertial sensors, which requires a more efficient calibration method for end-users.
Innovation Solution
A method for calibrating the relationship between a camera and an inertial sensor in a head-mounted display using images of a real marker in different poses, acquired through both the camera and inertial sensor data, to derive a spatial relationship and adjust calibration settings based on user feedback for improved accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If optical tracking using the camera is used to track real-world objects, then the system structure is simple, but the tracking accuracy deteriorates when angular velocity of head rotation is fast or rotation angle is large due to time difference (delay time)
Solution Approach 1:
The patent combines optical tracking (camera) and inertial tracking (motion sensors) into a hybrid tracking system. The camera captures visual data while motion sensors measure angular velocity and head rotation, and both data streams are fused through coordinate transformation to achieve accurate real-time tracking that overcomes the limitations of optical tracking alone during fast head movements.
Solution Approach 2:
The patent introduces coordinate transformation as an intermediary process that bridges the camera coordinate system and the inertial sensor coordinate system. This transformation mediates between the two different tracking modalities, allowing data from both sources to be integrated and reconciled into a unified tracking framework that maintains accuracy during dynamic head movements.
2Measurement precision
If motion sensors are used in conjunction with the camera to predict relative motion of the HMD and the environment, then the tracking accuracy is improved, but the device complexity increases due to integration of multiple sensors and calibration requirements
Solution Approach 1:
The patent performs calibration of the camera and motion sensors before actual AR operation. During this preliminary phase, the spatial relationship between sensors is established and stored as calibration data. This pre-calibration work prepares the system in advance, so that during normal operation the calibrated parameters can be directly applied without real-time adjustment, reducing operational complexity.
Solution Approach 2:
The patent implements a feedback mechanism where the system continuously compares the position of visual elements derived from camera data with positions derived from inertial sensor data. When discrepancies are detected, the system can trigger recalibration or adjust tracking parameters, creating a closed-loop system that self-corrects and maintains accuracy while adapting to changes in sensor performance over time.
3Measurement precision
If end-user calibration is implemented to improve calibration accuracy, then the measurement precision is improved, but the ease of operation deteriorates due to the difficulty of performing calibration
Solution Approach 1:
The patent enables end-users to perform their own calibration procedures without requiring specialized equipment or expert intervention. The system provides automated calibration routines that guide users through the process, allowing the HMD itself to serve as the calibration tool. This self-service approach empowers users to achieve accurate calibration while maintaining operational simplicity through automation.
Solution Approach 2:
The patent uses a camera to capture images of a calibration marker, creating a visual copy or representation of the marker's position and orientation. This captured image data is then processed to derive the spatial relationship between sensors. By using this visual copy rather than requiring direct physical measurement or complex sensor alignment, the system simplifies the calibration process while maintaining high precision.
4Measurement precision
If the camera and inertial sensor are calibrated using images of a real marker in different poses, then the calibration accuracy is improved, but the time required for calibration increases
Solution Approach 1:
The patent employs a calibration marker with specific geometric features (such as circles arranged in a pattern) that can be quickly identified and processed by image recognition algorithms. The marker is designed with sufficient features to enable accurate calibration but not so many features that the processing time becomes excessive. This optimized approach performs partial calibration using only the essential features needed for sensor alignment, achieving good accuracy without requiring exhaustive measurement of all possible marker properties.
Data Source
AI summary
A head-mounted display device includes: a camera; an inertial sensor in a fixed or adjustably fixed spatial relationship with the camera; a display; and a processor. The processor is configured to derive a first position of a feature element in an image frame in an image data sequence using an image frame, and derive a second position of the feature element using a sensor data sequence and the spatial relationship. The processor is further configured to display, with the display, a display image containing: i) a first visual element at a first location corresponding to the first position, and ii) a second visual element at a second location corresponding to the second position. The processor is further configured to perform calibration when a received signal indicates that the camera and inertial sensor are not sufficiently calibrated.