Burst Image Super-Resolution With CNN Deghosting and Noise Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Photographs taken with mid-range cameras or smartphones suffer from high noise levels and low spatial resolution, which affect detail visibility and are exacerbated by color artefacts from bright colors, and existing methods like Lucas-Kanade reloaded fail to account for image heterogeneity and are computationally intensive.
Innovation Solution
A device and method using a convolutional neural network (CNN) for image alignment, deghosting, and optimization with Lucas-Kanade algorithm to form high-resolution images from a burst of low-resolution images, incorporating exposure times and distortion parameters to minimize a quadratic energy function through iterative gradient descent.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If pixel size is reduced to increase spatial resolution, then spatial resolution is improved, but noise level in dark areas increases
Solution Approach 1:
The image processing is segmented into multiple frames captured in a burst sequence. Each frame captures different temporal information, and by processing multiple segmented frames together, the system achieves super-resolution while maintaining low noise levels through temporal averaging and selective pixel combination.
Solution Approach 2:
The invention transitions from spatial dimension alone to temporal dimension by capturing multiple frames over time. This adds the time dimension to the processing, allowing the system to resolve fine spatial details through temporal sampling while reducing noise through multiple measurements at different moments.
2Measurement precision
If existing methods like Lucas-Kanade reloaded are used for super-resolution, then spatial resolution is improved, but computational time and complexity increase significantly
Solution Approach 1:
The system performs preliminary actions by capturing multiple frames in advance during the burst sequence. This pre-capture of temporal data allows the processing algorithm to work with pre-prepared information, reducing the computational burden during the actual super-resolution processing compared to methods that require intensive real-time calculations.
Solution Approach 2:
The invention extracts only the essential distortion parameters and alignment information from each frame using simplified models, rather than performing full-blown optical flow calculations. This extraction of key features reduces computational complexity while maintaining the necessary precision for super-resolution.
3Measurement precision
If existing methods are used for processing burst images, then spatial resolution is improved, but image heterogeneity and ghosting artifacts are not addressed
Solution Approach 1:
The system applies local quality control by detecting and correcting ghosting artifacts in specific regions where they occur, rather than applying uniform processing across the entire image. This allows the system to maintain high resolution in clean areas while selectively addressing degradation in affected regions.
Solution Approach 2:
The invention implements feedback mechanisms by evaluating the quality and consistency of aligned frames during processing. This feedback allows the system to identify and correct issues such as ghosting and misalignment, ensuring that the final super-resolution image maintains consistent quality across all regions.
Data Source
AI summary
A device for forming a high-resolution image from pairs formed of a low-resolution image and of an exposure time. The device includes: an estimator arranged to estimate, using a Lucas-Kanade algorithm, a distortion parameter for each low-resolution image; a processing unit arranged to obtain an initial high-resolution image; an alignment unit arranged to align a reference low-resolution image according to each distortion parameter; a convolutional neural network arranged to generate, for each low-resolution image, a confidence factor; a computer arranged to calculate, for each low-resolution image, an interference-removal weight; and an optimization module arranged to minimize a quadratic energy function with two variables: a high-resolution image estimation variable and an auxiliary variable initialized by the initial high-resolution image.