Camera-IMU Fusion for Object Tracking Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems for object detection and pose estimation in three-dimensional models using cameras suffer from latency issues, especially when user motion is high, leading to delayed object tracking and re-initialization challenges, which degrade user experience in applications like robotics and augmented reality.
Innovation Solution
The integration of an inertial sensor (IMU) with a camera-based system for object detection and pose estimation, allowing for real-time tracking and re-initialization by fusing sensor data with image data to reduce latency and improve tracking accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a camera-based system is used for object detection and pose estimation, then object tracking can be performed, but latency increases as user motion or velocity increases
Solution Approach 1:
The patent combines camera-based object detection with IMU-based pose estimation into a unified tracking system. The camera provides accurate object detection while the IMU provides high-frequency pose data, merging both data sources to achieve both accuracy and low latency. The system fuses camera detection results with IMU sensor data to estimate object pose, creating a hybrid tracking approach that leverages the strengths of both sensors.
Solution Approach 2:
The IMU acts as an intermediary between the camera detection system and the final pose estimation output. When camera-based tracking experiences latency or loses track, the IMU provides continuous pose estimates that bridge the gap, maintaining tracking continuity. The system uses the IMU as a mediator to smooth transitions and maintain accurate tracking during rapid motion or when the camera view is temporarily obscured.
2Reliability
If the system loses track of an object and needs to re-initialize detection, then tracking can continue, but considerable processing time is required to search for the closest limited views
Solution Approach 1:
The system performs preliminary actions by maintaining a library of pre-captured object views from multiple angles and using the IMU to predict the current object pose before re-initialization is needed. When tracking is lost, the system can quickly search through pre-processed view data rather than capturing and processing new images, significantly reducing re-initialization time. The IMU data provides advance information about object orientation to guide the search for appropriate pre-captured views.
Solution Approach 2:
The system creates and stores multiple copies of object views from different angles and positions in advance. Instead of capturing new views during re-initialization, the system selects from pre-captured view copies that match the current object pose estimated by the IMU. This copying approach eliminates the need for real-time view capture and processing during re-initialization, dramatically reducing the time required to resume tracking.
3Device complexity
If only camera data is used for pose estimation, then the system is simpler, but latency increases during rapid user motion
Solution Approach 1:
The IMU serves as an intermediary sensor that bridges the latency gap in camera-based tracking during rapid motion. The IMU's high-frequency data stream provides continuous pose information that compensates for camera latency, allowing the system to maintain accurate tracking during fast user motion without requiring a complete system redesign.
Solution Approach 2:
The system changes the sampling rate parameter by using the IMU's high-frequency data (typically hundreds of Hz) to supplement the lower-frequency camera data (typically 30-60 Hz). This parameter change in data acquisition frequency provides temporal resolution during rapid motion that the camera alone cannot achieve, reducing effective tracking latency without significantly increasing overall system complexity.
Data Source
AI summary
A method may include the following steps: acquiring, from a camera, an image frame; acquiring, from an inertial sensor, a sensor data sequence; tracking a first pose of an object in a real scene based at least on the image frame; deriving a sensor pose of an inertial sensor based on the sensor data sequence; determining whether the first pose is lost; retrieving from one or more memories, or generating from a 3D model stored in one or more memories, a training template corresponding to a view that is based on the sensor pose obtained on or after the first pose is lost; and deriving a second pose of the object using the training template.