Deep Learning Image Fusion for Noise Reduction and High Dynamic Range
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image fusion techniques struggle with noise reduction and ghosting artifacts in images captured under varying conditions, particularly in low-light situations and with small image sensors, due to limitations in computational efficiency and the need for complex optimizations.
Innovation Solution
The use of machine learning techniques, specifically deep neural networks, to generate synthetic intermediate assets such as synthetic reference and synthetic long images, which are then pyramically decomposed and fused to reduce noise and improve image quality, while maintaining computational and memory efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple images are fused to reduce noise and improve signal-to-noise ratio, then image quality improves, but computational complexity and processing time increase
Solution Approach 1:
The patent segments the image fusion process into distinct stages: image registration, quality assessment, weight calculation, and pixel-level fusion. By dividing the complex fusion operation into manageable segments, the system can apply optimized algorithms at each stage, reducing overall computational complexity while maintaining noise reduction effectiveness.
Solution Approach 2:
The patent dynamically adjusts fusion parameters including weight thresholds, distance metrics, and quality assessment criteria based on input image characteristics. This adaptive parameter adjustment allows the system to optimize the balance between noise reduction and computational efficiency for different imaging scenarios, particularly for small sensor images with varying noise levels.
2Measurement precision
If image registration is performed to align candidate images with reference image, then fusion accuracy improves, but processing time increases due to motion compensation requirements
Solution Approach 1:
The patent performs preliminary image registration and quality assessment before the actual fusion operation. By pre-aligning images and identifying quality metrics in advance, the system reduces the computational burden during the fusion stage, thereby decreasing overall processing time while maintaining alignment accuracy.
Solution Approach 2:
The patent introduces an intermediary quality assessment step that evaluates image suitability before fusion. This intermediary process identifies and weights high-quality images, allowing the fusion algorithm to focus computational resources on aligning and combining only the most relevant images, thus reducing unnecessary processing time.
3Area of stationary object
If small pixel size is used to reduce device form factor, then device portability improves, but light capture per pixel decreases resulting in increased noise
Solution Approach 1:
The patent merges multiple noisy images captured by small pixels into a single fused image with improved signal-to-noise ratio. By combining information from multiple exposures and angles, the system compensates for the limited light capture capability of small pixels, effectively reducing noise while maintaining the compact device form factor.
Solution Approach 2:
The patent creates multiple copies of the scene through sequential image captures at different exposures and angles. These multiple copies are then processed and fused to produce a high-quality final image, allowing small-pixel sensors to achieve results comparable to larger sensors by leveraging temporal and angular redundancy.
4Illumination intensity
If multiple images with different exposures are fused for high dynamic range, then dynamic range improves, but difficulty in determining appropriate fusion weights increases
Solution Approach 1:
The patent implements a feedback mechanism where quality assessment metrics from previously fused images inform the weight calculation for subsequent fusion operations. The system learns from past fusion results and adjusts weights dynamically based on image quality, exposure levels, and noise characteristics, simplifying the determination of appropriate weights for high dynamic range fusion.
Solution Approach 2:
The patent automatically adjusts fusion parameters including weight thresholds and distance metrics based on the exposure differences and quality characteristics of input images. This adaptive parameter adjustment simplifies the weight determination process by eliminating the need for manual optimization, while still achieving optimal high dynamic range results across varying lighting conditions.
Data Source
AI summary
Electronic devices, methods, and program storage devices for leveraging machine learning to perform improved image fusion and/or noise reduction are disclosed. An incoming image stream may be obtained from an image capture device, wherein the incoming image stream comprises a variety of differently-exposed captures, e.g., EV0 images, EV− images, EV+ images, long exposure images, EV0/EV− image pairs, etc., which are received according to a particular pattern. When a capture request is received, two or more intermediate assets may be generated based on determined combinations of images from the incoming image stream, and the intermediate assets may then be fed into a neural network that has been trained to determine one or more sets of parameters to optimally fuse and/or noise reduce the intermediate assets. In some embodiments, the network may be trained to operate on levels of pyramidal decompositions of the intermediate assets independently, for increased efficiency and memory utilization.