Rolling Shutter Distortion Correction via Device-Specific Affine Transforms
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
CMOS-based camera sensors suffer from rolling shutter distortion, causing images to appear tilted or stretched when the camera is in motion, especially in DSLR cameras with large sensors, due to the sequential reading of pixels which results in varying acquisition times across the frame.
Innovation Solution
A method is implemented to correct rolling shutter distortion by using predefined affine transforms associated with known video capture devices, with an optional calibration process for unknown devices, allowing for automatic correction based on unique device identifiers and shutter speed characteristics, reducing the need for extensive parameter determination.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a global shutter like CCD-based sensors is used, then rolling shutter distortion is eliminated, but CMOS-based sensors cannot achieve this due to their sequential pixel reading mechanism
Solution Approach 1:
The patent replaces the mechanical/global shutter mechanism with a computational approach. Instead of using a global shutter hardware solution (which would require switching to CCD technology or complex CMOS global shutter mechanisms), the invention applies digital image processing techniques - specifically affine transforms - to correct the rolling shutter distortion after the sequential readout has occurred. This substitutes a hardware/mechanical solution with a software/computational one.
2Reliability
If the reading rate of CMOS sensor pixels is increased to reduce distortion, then rolling shutter effect is reduced, but inexpensive CMOS sensors have slow reading rates
Solution Approach 1:
The patent converts the harmful rolling shutter distortion effect into beneficial information. By analyzing the characteristic distortion patterns in the captured images, the system determines device-specific parameters (such as distortion coefficients) that are then used to create correction transforms. The harmful distortion becomes a diagnostic tool for characterizing the device and establishing the appropriate correction methodology.
Solution Approach 2:
The patent applies parameter changes through affine transforms that modify the geometric parameters of the distorted image. The correction process adjusts position, orientation, and scale parameters of image features to counteract the rolling shutter distortion. By changing these geometric parameters computationally, the system corrects the distortion without needing to change the physical reading rate of the sensor.
3Measurement precision
If extensive parameter determination is performed for each device, then accurate correction is achieved, but computational complexity and time increase
Solution Approach 1:
The patent performs preliminary action by pre-determining device-specific parameters during an initial calibration phase. The system captures calibration images, analyzes rolling shutter distortion patterns, and computes device-specific correction parameters in advance. These pre-computed parameters are then stored and reused for subsequent correction operations, eliminating the need to perform extensive parameter determination for every correction operation.
Solution Approach 2:
The patent uses copying by creating a digital model or profile of the specific device's distortion characteristics during calibration. This copied device profile containing the determined parameters is then reused across multiple correction operations. Instead of re-determining parameters each time, the system copies and applies the previously established device-specific correction model to new images from the same device.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing rolling shutter distortion corrections are described. A video clip captured by a user is received and each of a plurality of predefined affine transforms for rolling shutter distortion correction is applied to the received video clip. Further, a visual indication of results from each of the plurality of the predefined affine transforms is presented to the user and input is received from the user selecting one of the visual indications. Furthermore, the predefined affine transform corresponding to the selected visual indication is associated with a device that acquired the received video clip. Additionally, the association can be stored, and the stored association can be used later to automatically perform a rolling shutter distortion correction on another video clip upon detecting that the other video clip comes from same device that already went through a calibration sequence.