Hybrid 3D-2D Stabilization for 360-Degree Video
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current video stabilization algorithms for 360-degree video data are not robust, slow, and ineffective in removing shake and geometric distortions, particularly in Virtual Reality applications, due to their reliance on 2D motion models that do not translate well to the spherical domain.
Innovation Solution
A hybrid 3D-2D stabilization model that determines feature points, tracks motion, estimates relative rotations using 3D reasoning, and optimizes inner frames for visual smoothness, while also applying a speed adjustment model to maintain constant scene change, resulting in a stabilized 360-degree video with higher accuracy and processing speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full 3D reconstruction is performed to stabilize 360-degree video data, then stabilization accuracy is improved, but processing complexity and time increase significantly
Solution Approach 1:
The patent segments the video processing into key frame selection and inner frame processing. Key frames are processed using 3D reasoning for accurate rotation estimation, while inner frames use 2D optimization for efficiency. This segmentation allows high-accuracy 3D methods to be applied only where necessary, reducing overall computational complexity while maintaining stabilization accuracy.
Solution Approach 2:
Instead of performing full 3D reconstruction on all frames, the patent applies partial 3D reasoning only to selected key frames. This partial action approach provides sufficient stabilization accuracy for the entire video sequence while dramatically reducing processing complexity and time compared to full-frame 3D reconstruction.
2Productivity
If 2D motion models are used for video stabilization, then processing speed is improved, but stabilization effectiveness deteriorates in 360-degree video data
Solution Approach 1:
The patent transitions from 2D motion models to 3D reasoning by representing video frames as projections of 3D points on a unit sphere. This dimensional change allows the system to capture the spherical geometry of 360-degree video data, significantly improving stabilization effectiveness while maintaining reasonable processing speed through selective application to key frames only.
Solution Approach 2:
The patent applies different processing qualities to different parts of the video data: 3D reasoning with high computational cost is applied to key frames where accuracy is most critical, while 2D optimization with lower computational cost is applied to inner frames. This local quality differentiation maintains overall stabilization effectiveness while preserving processing speed.
3Device complexity
If 2D motion models are used for video stabilization, then device complexity is reduced, but adaptability to spherical domain deteriorates
Solution Approach 1:
The patent changes the fundamental parameters of motion modeling from 2D translation and rotation to 3D spherical coordinates (azimuth and elevation angles). This parameter change enables the algorithm to naturally adapt to the spherical domain of 360-degree video data, improving versatility while keeping the overall system complexity manageable through selective application.
Data Source
AI summary
An image processing system generates 360-degree stabilized videos with higher robustness, speed, and smoothing ability using a hybrid 3D-2D stabilization model. The image processing system first receives an input video data (e.g., a 360-degree video data) for rotation stabilization. After tracking feature points through the input video data, the image processing system determines key frames and estimates rotations of key frames using a 3D reasoning based on the tracked feature points. The image processing system also optimizes inner frames between key frames using a 2D analysis based on the estimated key frame rotation. After the 3D reasoning and the 2D analysis, the image processing system may reapply a smoothed version of raw rotations to preserve desirable rotations included in the original input video data, and generates a stabilized version of the input video data (e.g., a 360-degree stabilized video).