Rolling-Shutter VINS for Inaccurate Camera-IMU Timestamps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing vision-aided inertial navigation systems (VINS) face challenges with commercial-grade hardware in mobile devices due to unsynchronized camera and IMU clocks and the rolling-shutter effect, leading to navigation accuracy degradation.
Innovation Solution
A linear-complexity technique using an interpolation-based measurement model and Observability-Constrained Extended Kalman filter (OC-EKF) to compensate for time misalignment and rolling-shutter effects, enhancing navigation accuracy and speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If offline calibration methods are used to calibrate constant time offset between camera and IMU, then calibration accuracy can be improved, but the equipment requirements increase and the method becomes less adaptable to jittering time offsets
Solution Approach 1:
The patent replaces the mechanical/offline calibration process with a computational/online estimation approach. Instead of using physical calibration equipment to determine constant time offsets, the system employs algorithms that continuously estimate and compensate for time synchronization errors in real-time, eliminating the need for complex calibration hardware while adapting to dynamic conditions
Solution Approach 2:
The patent transitions from static calibration (constant time offset assumption) to dynamic calibration (time-varying offset modeling). The system models time synchronization errors as dynamic parameters that change over time, allowing the calibration process to adapt to jittering offsets without requiring rigid equipment setups, thereby improving adaptability while reducing equipment complexity
2Ease of manufacture
If rolling-shutter camera is used in mobile devices, then cost is reduced, but navigation accuracy degrades due to time misalignment and rolling-shutter effects
Solution Approach 1:
The patent converts the harmful rolling-shutter effect (time misalignment between camera rows and IMU measurements) into a beneficial opportunity for improved navigation. By explicitly modeling and incorporating the rolling-shutter timing information into the navigation algorithm, the system transforms the distortion caused by sequential row scanning into a predictable pattern that can be compensated for, thereby maintaining accuracy while using cost-effective rolling-shutter cameras
Solution Approach 2:
The patent changes the approach to time synchronization from assuming constant offsets to modeling time-varying parameters. The system estimates time synchronization errors as dynamic parameters that change during operation, allowing the navigation algorithm to adapt to the rolling-shutter timing characteristics in real-time, thus maintaining navigation accuracy despite the cost advantages of rolling-shutter cameras
3Device complexity
If conventional VINS methods are used with unsynchronized camera and IMU clocks, then system simplicity is maintained, but navigation accuracy significantly degrades
Solution Approach 1:
The patent introduces time synchronization error estimation as an intermediary component between the camera and IMU measurement fusion process. Instead of directly fusing measurements with unsynchronized timestamps, the system first estimates and compensates for time synchronization errors, creating a bridging mechanism that reconciles the timing discrepancies between camera and IMU clocks while maintaining system simplicity
Solution Approach 2:
The patent implements feedback mechanisms that continuously monitor and adjust for time synchronization errors. The system uses the navigation data to estimate time offset errors and feeds this information back into the measurement fusion process, allowing dynamic compensation for clock desynchronization. This feedback approach maintains accuracy without requiring complex pre-synchronization hardware, as the system self-corrects timing mismatches during operation
Data Source
AI summary
Vision-aided inertial navigation techniques are described. In one example, a vision-aided inertial navigation system (VINS) comprises an image source to produce image data at a first set of time instances along a trajectory within a three-dimensional (3D) environment, wherein the image data captures features within the 3D environment at each of the first time instances. An inertial measurement unit (IMU) to produce IMU data for the VINS along the trajectory at a second set of time instances that is misaligned with the first set of time instances, wherein the IMU data indicates a motion of the VINS along the trajectory. A processing unit comprising an estimator that processes the IMU data and the image data to compute state estimates for 3D poses of the IMU at each of the first set of time instances and 3D poses of the image source at each of the second set of time instances along the trajectory. The estimator computes each of the poses for the image source as a linear interpolation from a subset of the poses for the IMU along the trajectory.