Panoramic Video Anti-Shake Using Virtual Lens Reprojection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge of maintaining the original lens focus and viewing angle stability in panoramic videos shot with handheld devices due to camera jitter and shaking is not adequately addressed by existing pan-tilt stabilization methods, which are costly and voluminous.
Innovation Solution
A panoramic video anti-shake method using an extended Kalman filter to smooth camera motion, decompose it into virtual lens motion, and perform reprojection to generate a stable video by retaining the original shooting angle.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If a pan-tilt mechanism is used to stabilize the panoramic shooting device, then picture stability is improved, but device cost and volume increase
Solution Approach 1:
The patent replaces the mechanical pan-tilt stabilization system with a computational approach using extended Kalman filter algorithms. The system uses sensor data (accelerometer, gyroscope, magnetometer) to calculate compensation rotation matrices and applies these through image processing (reprojection and warping) rather than mechanical movement, thereby eliminating the need for complex mechanical stabilization components.
Solution Approach 2:
The patent introduces a virtual lens as an intermediary between the physical camera and the final image output. The extended Kalman filter estimates the virtual lens position and orientation, and this intermediate representation is used to generate compensation transformations that stabilize the image without requiring physical stabilization mechanisms.
2Reliability
If a pan-tilt mechanism is used to stabilize the panoramic shooting device, then picture jitter is reduced, but the device becomes larger and more expensive
Solution Approach 1:
The patent replaces the mechanical pan-tilt stabilization system with a computational approach using extended Kalman filter algorithms. The system uses sensor data (accelerometer, gyroscope, magnetometer) to calculate compensation rotation matrices and applies these through image processing (reprojection and warping) rather than mechanical movement, thereby eliminating the need for complex mechanical stabilization components.
3Adaptability or versatility
If the camera moves to acquire panoramic viewing angles, then complete panoramic coverage is achieved, but the original lens focus is lost
Solution Approach 1:
The patent performs preliminary estimation of the virtual lens state using the extended Kalman filter before the actual image processing. By continuously predicting and updating the virtual lens position and orientation based on sensor data, the system prepares compensation transformations in advance that will maintain focus consistency across the panoramic sequence.
Solution Approach 2:
The patent implements a feedback mechanism where the extended Kalman filter continuously updates the virtual lens state estimation based on incoming sensor measurements. This feedback loop allows the system to adapt to camera movements and maintain accurate focus information throughout the panoramic acquisition process.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The method generates a stable panoramic video by maintaining the original shooting angle and smooth motion, effectively reducing artificial jitters and retaining the viewing angle, even in noisy and dynamic environments.
Implementation Method 1
establishing the rotation vector of the current state by utilizing an extended Kalman filter combined with the accelerometer count value and the angular speed value
Implementation Method 2
acquiring in real time the timestamp of a current state, an accelerometer count value, and an angular speed value of a portable terminal
Implementation Method 3
acquiring in real time the timestamp of a current state, an accelerometer count value, and an angular speed value of a portable terminal
Implementation Method 4
calculating a current rotation matrix via the Rodrigues' rotation formula on the basis of the rotation vector of the current state
Data Source
Figure 1~2
AI summary
The present invention is applicable to the field of videos, and provides a panoramic video anti-shake method and a portable terminal. The method comprises: obtaining world coordinates of any reference point in a world coordinate system in real time, and simultaneously obtaining camera coordinates corresponding to the reference point in the portable terminal and an angular velocity value of a current state of a gyroscope in the portable terminal; smoothing the motion of a camera using an extended Kalman filter; decomposing the smoothed motion to synthesize virtual lens motion of a free lens mode, and calculating the rotation amount of the virtual lens; and re-projecting an original video to generate a stable video. In the present invention, by decomposing the motion of the camera, keeping an original photographing view angle of the camera, and synthesizing the virtual lens motion, the stable video is generated.