HDR Image Generation Using Neural Subnetworks for Ghost Artifact Removal
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing HDR image processing methods fail to effectively handle misalignment and exposure differences between low dynamic range (LDR) images, leading to ghost artifacts in high dynamic range (HDR) images, especially in dynamic scenes with camera or scene motion.
Innovation Solution
The use of convolutional neural networks (CNNs) to process multiple LDR images simultaneously, incorporating color mapping, HDR merging, and guided de-ghosting sub-processes, which considers all possible combinations of input images and their exposures to generate a ghost-free HDR image.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If conventional de-ghosting methods are used to handle misalignment in LDR images, then computational complexity is reduced, but ghost artifacts remain visible in cases of non-rigid motion or large perspective differences
Solution Approach 1:
The patent segments the image processing into multiple specialized neural network modules: color mapping sub-networks for exposure alignment, HDR merging sub-network for combining images, and guided de-ghosting sub-network for artifact removal. Each module handles a specific aspect of the processing, allowing complex operations to be divided into manageable segments that can be processed in parallel or sequence.
Solution Approach 2:
The patent introduces an intermediary guided de-ghosting process that uses the initially generated HDR image as a guide to identify and remove ghost artifacts. This intermediary step takes the output from the HDR merging stage and uses it to inform the de-ghosting operation, creating a feedback loop that improves overall effectiveness without requiring complete redesign of the entire system.
2Reliability
If all possible combinations of input LDR images are processed to generate HDR image, then image quality and dynamic range are improved, but processing time and computational load increase
Solution Approach 1:
The patent performs preliminary color mapping to align all input LDR images to a reference exposure level before HDR merging. This preliminary alignment ensures that all images are properly registered and reduces the need for iterative adjustments during the merging process, saving computational time while maintaining image quality.
Solution Approach 2:
The patent merges multiple aligned LDR images simultaneously using the HDR merging sub-network, which processes all input images in a single integrated operation rather than sequentially combining them pairwise. This merging approach efficiently utilizes all available input data to generate the final HDR image while reducing the total number of processing steps required.
3Stability of the object's composition
If color mapping is performed to align LDR images to reference exposure, then exposure consistency is improved, but processing complexity increases
Solution Approach 1:
The patent creates a copy of the reference LDR image at the target exposure level through color mapping, rather than modifying the original reference image or all input images. This copying approach allows the reference to remain unchanged while generating an aligned version for merging, simplifying the overall process and reducing computational overhead.
Solution Approach 2:
The color mapping sub-network changes the exposure parameter of the reference image to match the target exposure level. By adjusting this single parameter through learned transformations, the system achieves exposure alignment without requiring complex geometric transformations or manual calibration, reducing processing complexity while maintaining consistency.
Data Source
AI summary
The invention relates to an image processing apparatus for generating an HDR image associated with a first view on the basis of a plurality of LDR images, including a first LDR image and a second LDR image. The first LDR image is associated with the first view and a first exposure, i.e. a first dynamic range, and the second LDR image is associated with a second view and a second exposure, i.e. a second dynamic range. The image processing apparatus comprises a processor configured to provide a neural network having a plurality of neural subnetworks including a first neural subnetwork. The first neural subnetwork is configured to generate the HDR image on the basis of: (i) the first LDR image, (ii) the second LDR image, and (iii) a modified first LDR image. The modified first LDR image is associated with the first view and the second exposure.