Array Image Reconstruction With Neural Matching Refinement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image processing methods for array cameras struggle to effectively merge sub-images into a high-resolution output image, often resulting in artifacts and inadequate resolution enhancement due to insufficient pixel matching and refinement techniques.
Innovation Solution
An image processing method utilizing a neural network model to estimate optical flow and perform pixel-to-pixel matching, followed by geometric consistency refinement to enhance resolution and reduce artifacts by merging sub-images into a single view.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of moving object
If a capturing device uses a compact array lens structure to reduce volume, then the device size is reduced, but the image resolution deteriorates due to division into smaller sub-units
Solution Approach 1:
The capturing device uses an array lens composed of multiple sub-lenses arranged in a grid pattern, where each sub-lens captures a portion of the light. This segmentation allows the device to maintain a compact form factor while gathering light from multiple angles, which is then processed to reconstruct a high-resolution image that overcomes the resolution limitations of individual small sub-lenses.
Solution Approach 2:
The patent transforms the problem from a two-dimensional sensor plane to a three-dimensional light field by using multiple sub-lenses at different spatial positions. Each sub-lens captures light from a slightly different perspective, creating a multi-view image set that encodes depth and spatial information. This dimensional transformation enables resolution enhancement through computational processing of the multi-view data.
2Volume of moving object
If array lens divides sensor and lens into smaller sub-units to reduce size, then device compactness is improved, but image quality deteriorates due to artifacts in merged images
Solution Approach 1:
The patent introduces an intermediate processing stage that generates temporary restored images from the multi-view sub-images before final merging. This intermediate restoration process uses gradient information and neural network-based matching to create high-quality temporary images that serve as intermediaries between the raw sub-images and the final merged output, thereby reducing artifacts and improving overall image quality.
Solution Approach 2:
The patent replaces traditional optical merging mechanisms with computational image processing methods. Instead of relying on simple optical superposition or mechanical alignment of sub-images, the system uses neural network models to determine matching information between views and employs sophisticated algorithms for merging temporary restored images. This substitution of mechanical/optical merging with computational processing significantly reduces artifacts and improves image quality.
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
A method and apparatus for array image processing are provided. The method includes receiving sub images corresponding to different views of an input array image generated through an array lens, generating temporary restored images based on the sub images using a gradient between neighboring pixels of each of the sub images, determining matching information based on a view difference between pixels of the sub images using a neural network model, based on a pixel distance between matching pairs of the pixels of the sub images using the matching information, extracting refinement targets from the matching pairs, refining the matching information by replacing at least some of target pixels included in the refinement targets based on a local search of a region based on pixel locations of the refinement targets, and generating an output image of a single view by merging the temporary restored images based on the refined matching information.