360-Degree Video Stabilization Using IMU and Feature Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for stabilizing 360-degree videos captured by electronic devices fail to effectively correct tilt and maintain heading direction due to noisy inertial measurement unit data and lack of accurate real-time tilt correction, leading to an unpleasant viewing experience, especially in scenarios with curved paths or initial camera tilts.
Innovation Solution
A method and apparatus that compute relative rotation angles for 360-degree image frames by tracking feature points and using IMU rotation data, estimate universal offsets, and apply relative rotations to stabilize and correct tilt, while maintaining the head direction by calculating angle differences and applying filters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If computer vision based method is used to stabilize 360 degree video, then video stabilization is achieved, but tilt correction is not effective and real-time processing is difficult
Solution Approach 1:
The patent segments the stabilization process into two independent modules: feature-point-based stabilization and IMU-based tilt correction. This segmentation allows each module to specialize in one aspect (stabilization or tilt correction) and operate in real-time without the complexity of combining both functions in a single computer vision pipeline.
Solution Approach 2:
The patent introduces an intermediary IMU sensor system that provides independent tilt angle measurements to complement the computer vision stabilization. This intermediary system bridges the gap between visual stabilization and physical orientation, enabling real-time tilt correction without overloading the computer vision processing.
2Measurement precision
If IMU rotation data is used for tilt correction, then tilt correction is applied, but noisy sensor data reduces accuracy
Solution Approach 1:
The patent merges IMU-based tilt angle measurements with feature-point-based rotation calculations through a unified optimization model. This combination allows the system to leverage the complementary strengths of both sources: IMU provides direct tilt measurements while feature points provide visual verification, resulting in more accurate and reliable tilt correction despite noisy sensor data.
Solution Approach 2:
The patent implements feedback mechanisms where the stabilized video output is continuously monitored and used to adjust the tilt correction parameters. The system compares the expected orientation from IMU data with the actual visual features, and uses this feedback to refine the tilt correction in real-time, improving accuracy despite noisy sensor inputs.
3Stability of the object's composition
If frames are aligned with respect to a first frame for stabilization, then video stabilization is achieved, but heading direction is lost in curved path scenarios
Solution Approach 1:
The patent applies preliminary tilt correction using IMU data before performing the frame alignment stabilization. By pre-correcting the tilt orientation based on sensor measurements, the system preserves heading direction information in the reference frame, allowing subsequent stabilization to maintain both stability and directional accuracy even in curved path scenarios.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
An apparatus and method are provided for handling 360 degree image content captured by a 360 degree camera. A method includes obtaining, by an electronic device, a sequence of 360 degree projection format image frames associated with the 360 degree image content and inertial measurement unit (IMU) rotation data associated with the 360 degree image content, computing, by the electronic device, a relative rotation of the 360 degree image content, and applying, by the electronic device, the relative rotation to at leastone of the 360 degree projection format image frames to stabilize the 360 degree image content.