Rolling Shutter Distortion Removal via Spatially Varying Homography
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
CMOS imaging sensors with rolling shutter mechanisms capture images with distortions such as skews and wobbles when in motion, complicating computer vision tasks and requiring either front-end distortion removal or back-end task-dependent algorithms, with few holistic solutions available for simultaneous image stitching and rectification.
Innovation Solution
A computer-implemented method and system using rolling shutter-aware spatially varying differential homography fields for simultaneous distortion removal and image stitching, involving keypoint detection, outlier filtering, and computation of a spatially varying differential homography field to produce distortion-free stitched images.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If rolling shutter mechanism is used in CMOS imaging sensors, then manufacturing cost and simplicity are improved, but image distortion occurs when the device is in motion
Solution Approach 1:
The patent applies preliminary action by estimating camera motion parameters and computing compensation transformations before the image stitching process. The system calculates the rolling shutter distortion parameters and pre-computes the geometric correction needed, then applies these corrections during stitching to eliminate distortions before they affect the final output.
Solution Approach 2:
The patent changes parameters by introducing scanline-dependent homography transformations that account for the temporal progression of the rolling shutter readout. Instead of using a single global homography, the system varies the transformation parameters across different scanlines based on their exposure time, thereby correcting the geometric distortions caused by camera motion during the rolling shutter sequence.
2Device complexity
If conventional image stitching algorithms are applied to rolling shutter images, then processing simplicity is improved, but stitching accuracy deteriorates due to uncorrected distortions
Solution Approach 1:
The patent applies preliminary action by performing motion estimation and distortion parameter calculation before the actual stitching operation. The system computes the rolling shutter effect parameters from the input images and pre-determines the correction transformations, so that when stitching is performed, the distortions have already been accounted for, improving alignment accuracy without significantly increasing overall complexity.
3Manufacturing precision
If rolling shutter distortion correction is applied, then image geometric accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent reduces computational complexity by performing motion estimation and distortion parameter calculation in advance, before the actual correction and stitching operations. By pre-computing the rolling shutter parameters and correction transformations, the system avoids repeated complex calculations during stitching, thereby maintaining geometric accuracy while reducing overall computational burden.
Solution Approach 2:
The patent manages computational complexity by parameterizing the rolling shutter model with a limited set of motion parameters (rotation, translation, timing). Instead of computing full per-pixel distortion corrections, the system uses these compact parameters to generate scanline-dependent homography transformations, which are computationally more efficient while still achieving accurate distortion correction.
4Measurement precision
If scanline-varying transformations are used for distortion removal, then distortion correction accuracy is improved, but algorithm complexity increases
Solution Approach 1:
The patent achieves accurate distortion correction with managed complexity by parameterizing the scanline-varying transformations through a compact motion model. Instead of using fully independent per-scanline transformations, the system expresses all transformations in terms of a small set of camera motion parameters (rotation, translation, timing), which simplifies the algorithm structure while maintaining the ability to accurately correct distortions across different scanlines.
Data Source
AI summary
A computer-implemented method executed by at least one processor for applying rolling shutter (RS)-aware spatially varying differential homography fields for simultaneous RS distortion removal and image stitching is presented. The method includes inputting two consecutive frames including RS distortions from a video stream, performing keypoint detection and matching to extract correspondences between the two consecutive frames, feeding the correspondences between the two consecutive frames into an RS-aware differential homography estimation component to filter out outlier correspondences, sending inlier correspondences to an RS-aware spatially varying differential homography field estimation component to compute an RS-aware spatially varying differential homography field, and using the RS-aware spatially varying differential homography field in an RS stitching and correction component to produce stitched images with removal of the RS distortions.