Joint Rolling Shutter Correction and Deblurring via Synthesized Training Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Rolling shutter (RS) distortions and blur artifacts in images captured by CMOS sensors, particularly in low-light conditions, are challenging to address due to the sequential exposure of sensor rows, leading to distorted and blurred images.
Innovation Solution
An end-to-end learning approach using a structure-and-motion-aware RS distortion and blur rendering module and a single-view joint RS correction and deblurring network to generate synthesized RS blurred images and predict rectified and deblurred images from a single input RS and blurred image, leveraging global shutter sharp images, depth maps, and synthesized camera motions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If rolling shutter mechanism is used in CMOS sensors, then cost advantages are maintained, but RS distortions and blur artifacts occur in captured images
Solution Approach 1:
The patent creates a virtual copy of the imaging process by synthesizing RS blurred images from GS sharp images and depth maps. This synthetic training data allows the network to learn RS correction without requiring expensive GS hardware, resolving the contradiction between cost-effective RS sensors and image quality correction capability
Solution Approach 2:
The patent introduces depth maps as an intermediary element that bridges the gap between GS sharp images and RS blurred images. The depth information serves as a mediator to guide the synthesis process and enable the neural network to understand the relationship between sharp and blurred images, allowing correction of RS artifacts while maintaining cost advantages of RS sensors
2Device complexity
If sequential exposure of sensor rows is used, then device complexity is reduced, but image distortion and blur increase
Solution Approach 1:
The patent replaces the complex mechanical solution of global shutter mechanisms with a computational approach using neural networks. Instead of modifying the physical sensor to eliminate sequential exposure, the system uses software-based RS correction that processes the sequentially exposed images to remove distortions, maintaining simple sensor mechanics while recovering image quality
Solution Approach 2:
The patent changes the parameter space by introducing depth maps as an additional input modality. This additional parameter provides geometric information that helps the neural network understand scene structure and accurately correct RS distortions, compensating for the information loss caused by sequential exposure while keeping the sensor mechanism simple
3Ease of operation
If RS correction and deblurring are performed separately, then processing steps are simplified, but correction accuracy decreases
Solution Approach 1:
The patent merges RS correction and deblurring into a single joint neural network that processes both tasks simultaneously. This unified approach allows the network to learn the coupled relationships between RS distortion and motion blur, achieving more accurate correction than separate processing while maintaining operational simplicity through a single end-to-end trained model
Data Source
AI summary
A method for jointly removing rolling shutter (RS) distortions and blur artifacts in a single input RS and blurred image is presented. The method includes generating a plurality of RS blurred images from a camera, synthesizing RS blurred images from a set of GS sharp images, corresponding GS sharp depth maps, and synthesized RS camera motions by employing a structure-and-motion-aware RS distortion and blur rendering module to generate training data to train a single-view joint RS correction and deblurring convolutional neural network (CNN), and predicting an RS rectified and deblurred image from the single input RS and blurred image by employing the single-view joint RS correction and deblurring CNN.