Rolling Shutter Removal via Camera Motion Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Rolling shutter effects in video capture, such as wobble, skew, and smear, cause distortions in recorded videos, especially in handheld shots or when the camera is in motion, and existing methods require calibration or prior knowledge of camera characteristics to correct these issues.
Innovation Solution
A calibration-free method and system that determines features in video frames, calculates projective and mixture transforms to estimate camera motion, and accounts for distortion caused by rolling shutter, allowing for the removal of rolling shutter effects without prior calibration, using a computing device with modules for feature extraction, transformation, and camera path estimation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If rolling shutter capture method is used, then video recording capability is improved, but distortion and quality degradation occur
Solution Approach 1:
The patent converts the harmful rolling shutter distortion effects into beneficial information by analyzing the distortion patterns themselves. The distortion caused by rolling shutter is used to infer camera motion and characteristics, transforming what was previously a quality degradation into a useful signal for calibration and stabilization without requiring external calibration equipment.
Solution Approach 2:
The system performs self-calibration by using the rolling shutter distortion to automatically determine camera characteristics and motion models. The video recording process itself provides the calibration data needed, eliminating the need for separate calibration procedures or external reference materials.
2Manufacturing precision
If calibration is performed to remove rolling shutter effects, then video quality is improved, but device complexity and calibration requirements increase
Solution Approach 1:
The system eliminates external calibration requirements by using the rolling shutter distortion itself as the calibration signal. The camera automatically calibrates its own characteristics and motion model by analyzing the distortion patterns in the recorded video, removing the need for external calibration equipment or procedures.
Solution Approach 2:
The patent extracts calibration information directly from the video recording process by analyzing rolling shutter distortion patterns. The calibration data is extracted from the actual video content rather than from separate calibration sequences or reference materials, simplifying the overall system requirements.
3Measurement precision
If feature extraction and transform calculation are performed, then camera motion estimation is improved, but computational time and processing complexity increase
Solution Approach 1:
The patent applies local quality by processing different regions of the video frame with appropriate detail levels. Feature extraction focuses on salient points and edges rather than processing every pixel uniformly, and the transform calculations are optimized for local motion characteristics, reducing overall computational burden while maintaining precision.
Solution Approach 2:
The system segments the video processing into distinct stages: feature extraction, motion estimation, and distortion correction. By dividing the complex computation into manageable segments and processing them in sequence rather than simultaneously, the patent reduces total processing time while maintaining the precision of camera motion estimation.
Data Source
AI summary
Methods and systems for rolling shutter removal are described. A computing device may be configured to determine, in a frame of a video, distinguishable features. The frame may include sets of pixels captured asynchronously. The computing device may be configured to determine for a pixel representing a feature in the frame, a corresponding pixel representing the feature in a consecutive frame; and determine, for a set of pixels including the pixel in the frame, a projective transform that may represent motion of the camera. The computing device may be configured to determine, for the set of pixels in the frame, a mixture transform based on a combination of the projective transform and respective projective transforms determined for other sets of pixels. Accordingly, the computing device may be configured to estimate a motion path of the camera to account for distortion associated with the asynchronous capturing of the sets of pixels.