Rolling Shutter Camera Pose Tracking via Inertial Acceleration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing augmented reality systems using rolling shutter cameras face challenges in accurately estimating camera poses due to the assumption of constant camera velocity during image readout, which is often violated by unpredictable camera movements, leading to distorted images and failed tracking.
Innovation Solution
The proposed solution involves using inertial sensor data to estimate camera poses for each scanline during rolling shutter image readout, modeling velocity changes instead of assuming constant velocity, thereby improving pose estimation accuracy without significant computational overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If constant velocity assumption is used during rolling shutter image readout, then computational complexity is reduced, but pose estimation accuracy deteriorates due to unpredictable camera movements
Solution Approach 1:
The patent applies dynamics by transitioning from a static constant velocity model to a dynamic acceleration model that adapts to changing camera motion. The system estimates acceleration between scanlines and updates the motion model accordingly, allowing the pose estimation to dynamically respond to unpredictable camera movements while maintaining computational efficiency through the use of simple acceleration calculations rather than complex full physics simulations.
2Measurement precision
If acceleration modeling is used to account for velocity changes during rolling shutter readout, then pose estimation accuracy is improved, but computational overhead increases
Solution Approach 1:
The patent implements partial action by applying acceleration compensation only to the rolling shutter readout process rather than to the entire image processing pipeline. The system calculates acceleration estimates specifically for the duration of the rolling shutter exposure and applies corrections only where needed during scanline processing, avoiding the computational overhead of applying complex motion models to the entire image or performing full physics-based simulations.
3Reliability
If inertial sensor data is integrated into pose estimation, then tracking robustness is improved, but system complexity increases
Solution Approach 1:
The patent uses inertial sensor data as an intermediary to bridge the gap between visual tracking and physical motion. Rather than directly integrating complex sensor fusion algorithms, the system uses inertial measurements as a mediator to estimate acceleration and inform the rolling shutter compensation model, simplifying the overall system architecture while maintaining tracking robustness through the additional motion information.
Data Source
AI summary
A method and apparatus of tracking poses of a rolling-shutter camera in an augmented reality (AR) system is provided. The method and apparatus use camera information and inertial sensor readings from Inertial Measurement Unit (IMU) to estimate the pose of the camera at a reference line. Thereafter, relative pose changes at scanlines may be calculated using the inertial sensor data. The estimated reference pose of the camera is then further refined based on the visual information from the camera, the relative pose changes and the optimized reference line pose of a previous image. Thereafter, the estimate of the scanline poses may be updated using the relative pose changes obtained in the earlier steps.