Camera-IMU Joint Calibration for Accurate AR Localization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing augmented reality (AR) devices face inaccuracies in position estimation and map generation due to changes in the relative positions of multiple cameras and inertial measurement units (IMUs) within the device, which are not accurately accounted for in traditional bundle adjustment processes.
Innovation Solution
A method that jointly optimizes camera and IMU calibrations using visual and inertial measurements, integrating inertial data to correct relative positions and improve the accuracy of device localization and environment mapping, allowing for real-time recalibration without factory intervention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional bundle adjustment processes are used without integrating inertial data, then the processing complexity is lower, but the position estimation accuracy and map generation accuracy deteriorate
Solution Approach 1:
The patent merges visual data from cameras with inertial data from IMUs into a unified bundle adjustment process. The system jointly optimizes camera poses, map points, and sensor calibrations by combining projection equations from visual features with motion constraints from inertial measurements, achieving improved position estimation accuracy while managing processing complexity through integrated optimization.
Solution Approach 2:
The patent introduces an intermediary calibration parameter set that bridges camera and IMU data. By optimizing extrinsic parameters (relative position and orientation between sensors) and intrinsic parameters (sensor-specific characteristics) as intermediaries, the system enables accurate fusion of visual and inertial measurements, resolving the contradiction between accuracy improvement and processing complexity.
2Adaptability or versatility
If factory calibration is performed for sensor positions, then the initial calibration accuracy is higher, but the ability to adapt to position changes deteriorates
Solution Approach 1:
The patent transforms static factory calibration into a dynamic, adaptive calibration process. The system continuously optimizes sensor calibration parameters during operation by integrating inertial constraints with visual observations, allowing the calibration to adapt to position changes while maintaining accuracy through real-time optimization of extrinsic and intrinsic parameters.
Solution Approach 2:
The patent implements feedback mechanisms where the bundle adjustment process continuously refines calibration parameters based on discrepancies between predicted and observed visual features. The optimized calibration parameters feed back into the sensor fusion process, creating a closed-loop system that maintains both adaptability and calibration accuracy.
3Reliability
If multiple sensors are integrated in the AR device, then the data fusion capability is improved, but the relative position stability between sensors deteriorates
Solution Approach 1:
The patent segments the calibration problem into distinct components: extrinsic parameters (relative position and orientation between sensors) and intrinsic parameters (sensor-specific characteristics). This segmentation allows independent optimization of each component, enabling the system to handle sensor position variations while maintaining reliable data fusion through separate calibration streams.
Solution Approach 2:
The patent employs parameter changes by optimizing calibration parameters (extrinsic and intrinsic) to compensate for physical changes in sensor relative positions. The system dynamically adjusts these parameters based on observed discrepancies between inertial predictions and visual measurements, maintaining data fusion reliability despite physical instability in sensor mounting.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for calibrating an augmented reality device using camera and inertial measurement unit data. In some implementations, a bundle adjustment process jointly optimizes or estimates states of the augmented reality device. The process can use, as input, visual and inertial measurements as well as factory-calibrated sensor extrinsic parameters. The process performs bundle adjustment and uses non-linear optimization of estimated states constrained by the measurements and the factory calibrated extrinsic parameters. The process can jointly optimize inertial constraints, IMU calibration, and camera calibrations. Output of the process can include most likely estimated states, such as data for a 3D map of an environment, a trajectory of the device, and/or updated extrinsic parameters of the visual and inertial sensors (e.g., cameras and IMUs).