Rolling Shutter Artifact Repair via Feature Mesh Warping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Digital cameras with rolling shutter sensors capture images as scan lines, leading to distortion in moving objects due to time delays between scan lines, causing artifacts in video frames.
Innovation Solution
A method involving feature identification, mesh generation, and warping of frames to correct distortions by determining pixel movements and using algorithms like Kanade-Lucas-Tomasi for feature extraction and thin plate spline for interpolation to generate a warped frame.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If scan lines are read in succession with time delay, then data can be captured and stored in memory, but moving objects appear distorted due to position changes during the time delay
Solution Approach 1:
The system performs preliminary actions by capturing feature positions in multiple frames before the distortion becomes apparent, then uses these pre-captured positions to calculate and apply corrective transformations that compensate for the rolling shutter distortion
Solution Approach 2:
The system creates a copy of the distorted frame and applies warping transformations to the copy rather than the original, generating a corrected version that eliminates distortion while preserving the captured image data
2Shape
If feature tracking and mesh warping are applied to correct distortion, then object shape accuracy is improved, but computational complexity and processing time increase
Solution Approach 1:
The system segments the image into a mesh of connected vertices and edges, applying transformations to individual mesh elements rather than the entire image at once, which simplifies the computational complexity while maintaining shape accuracy
Solution Approach 2:
The system applies different warping transformations to different regions of the image based on local feature movements, rather than applying a uniform transformation across the entire frame, which improves shape accuracy while managing computational complexity
Data Source
AI summary
Systems and methods for rolling shutter artifact repair are disclosed. For example, one disclosed method includes the steps of receiving two frames (F0, F2) of a video, identifying a first plurality of features in frame F0 and a second plurality of features in frame F2, determining movements of corresponding features in the first and second pluralities of features, generating a mesh based on the first and second pluralities of features and the movements, and generating a warped frame (F0′) based on the mesh