RAW Burst Image Restoration with Feature Alignment and Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Smartphone cameras face challenges in capturing high-quality images due to small image sensors and limited lens sizes, leading to issues like misalignment, noise, and low dynamic range in burst photography, which existing methods struggle to address effectively.
Innovation Solution
A neural network engine in a mobile device processes a RAW image burst through edge boosting feature alignment to correct spatial and color misalignment, followed by pseudo-burst feature fusion and adaptive group upsampling to enhance image quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Illumination intensity
If longer exposure is used to gather more light in smaller sensors, then light gathering capability is improved, but image sharpness deteriorates due to blur from handshake and scene movement
Solution Approach 1:
The patent captures multiple images in rapid succession (periodic action) rather than using a single long exposure. This allows the system to gather sufficient light across multiple shorter exposures while avoiding motion blur that would occur with prolonged exposure times.
Solution Approach 2:
The patent merges multiple short-exposure images into a single high-quality output image. By combining information from multiple frames captured in quick succession, the system achieves both adequate light gathering and maintained sharpness, resolving the contradiction between these two requirements.
2Illumination intensity
If wider aperture lenses are used to take in more light, then light gathering capability is improved, but lens distortion artifacts and field of view alterations are introduced
Solution Approach 1:
Instead of changing the physical aperture parameter to increase light gathering, the patent changes the temporal parameter by capturing multiple images in quick succession. This allows achieving adequate illumination without the quality degradation that would result from using wider aperture lenses.
3Volume of moving object
If smaller image sensors are used to maintain compact form factor, then device portability is improved, but spatial resolution and image quality deteriorate
Solution Approach 1:
The patent uses rapid sequential capture of multiple images to compensate for the limited light-gathering capability of small sensors. This periodic capture approach enables small sensors to achieve image quality comparable to larger sensors while maintaining compact device form factor.
Solution Approach 2:
The patent combines information from multiple images captured by the small sensor to produce a high-resolution output. This merging process recovers spatial resolution that would otherwise be lost due to the sensor's small size, while the device remains compact.
4Manufacturing precision
If burst photography is used to capture multiple frames for processing, then image quality potential is improved, but inter-frame misalignment and processing complexity increase
Solution Approach 1:
The patent performs alignment and feature extraction operations on individual frames before merging them. This preliminary processing of each frame separately simplifies the overall workflow and reduces the complexity of handling the entire burst sequence at once.
Solution Approach 2:
The patent segments the burst image processing into distinct stages: alignment, feature extraction, and merging. By dividing the complex task into manageable segments, the system handles inter-frame misalignment systematically while maintaining image quality.
Data Source
AI summary
A mobile device and mobile application, in which the mobile device includes a camera having an image capture circuit operating in a mode to capture a RAW image burst, and processing circuitry, including a neural network engine, to generate a single enhanced image from the RAW image burst. The neural network engine executing program instructions including an edge boosting feature alignment stage to remove inter-frame spatial and color misalignment from the RAW image burst to obtain aligned burst frames, a pseudo-burst feature fusion stage to create a set of pseudo-burst features that combine complementary information from the aligned burst frames, and an adaptive group upsampling stage to progressively increase spatial resolution while merging the set of pseudo-burst features and output the single enhanced image. The mobile application and mobile device perform super-resolution, low-light image enhancement, and burst denoising using a RAW image burst.