Heterogeneous Camera Array Super-Resolution With Parallax Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional image sensors, particularly in mobile devices, face limitations in dynamic range, signal-to-noise ratio, and low light sensitivity due to small pixel size, limited light gathering capability, and the constraints imposed by Bayer filters and IR filters, leading to poor performance in capturing images, especially in low-light conditions.
Innovation Solution
A camera array comprising multiple imagers with varying imaging characteristics, including different pixel sizes, filters, and optical elements, fabricated using wafer-level optics technology, which allows for spatial shifting and independent operation of imagers to capture sub-pixel phase-shifted images, enabling super-resolution processing and fusion to enhance image quality and dynamic range.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If Bayer filter mosaic is used to reduce manufacturing complexity, then ease of manufacture is improved, but measurement precision deteriorates due to low light sensitivity and limited dynamic range
Solution Approach 1:
The image sensor is divided into multiple independent photodetector arrays, each with its own color filter array. This segmentation allows each array to capture full-color information independently, eliminating the need for demosaicing interpolation while maintaining manufacturing simplicity through standardized array structures.
Solution Approach 2:
The patent transitions from a single-plane Bayer filter structure to a multi-layered architecture with multiple photodetector arrays stacked vertically. Each array captures the complete color spectrum, and the vertical stacking dimension enables full-color capture at every pixel location without requiring horizontal color filter patterns.
2Measurement precision
If pixel size is reduced to increase resolution, then measurement precision is improved, but light gathering capability deteriorates leading to poor low-light performance
Solution Approach 1:
The patent utilizes the vertical stacking dimension to place multiple photodetector arrays one above another. This allows each pixel location to have multiple photodetectors capturing light simultaneously, effectively increasing the light gathering area without expanding the horizontal pixel footprint, thus maintaining high resolution while improving low-light performance.
Solution Approach 2:
The image sensor employs a composite structure combining multiple photodetector arrays with different color filter configurations stacked vertically. This composite architecture enables each pixel position to aggregate light across multiple detection layers, enhancing light gathering capability while preserving spatial resolution through the vertical integration of sensing elements.
3Measurement precision
If color filters are added to capture full color information, then measurement precision is improved, but light sensitivity deteriorates due to wavelength filtering
Solution Approach 1:
The patent segments the color filtering function across multiple independent photodetector arrays, each equipped with its own color filter array. This segmentation allows each array to capture complete color information at every pixel location, eliminating the need for demosaicing and improving light sensitivity by ensuring every pixel receives full-spectrum light for color capture.
4Measurement precision
If multiple photodetector arrays with different characteristics are used to improve image quality, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent resolves the complexity issue by organizing multiple photodetector arrays in a vertical stack rather than spreading them horizontally. This vertical integration consolidates multiple sensing functions within a compact footprint, reducing overall device complexity while maintaining the ability to capture diverse spectral and spatial information through the stacked architecture.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The solution significantly improves image resolution, dynamic range, and low-light sensitivity by combining images from multiple imagers with different characteristics, resulting in higher quality images with increased signal-to-noise ratio and the ability to capture high dynamic range, panoramic, and hyper-spectral images.
Implementation Method 1
a first substrate having a convex refractive element, a second substrate having a concave refractive element... the convex refractive element may include a plurality of convex refractive elements, the concave refractive element may include a plurality of concave refractive elements
Implementation Method 2
a separation between the first to second substrates, the separation including an air gap between convex refractive element and the concave refractive element
Implementation Method 3
a third substrate having a detector array thereon... The image sensor consists of pixels that generate signals upon receiving light via the optical element
Implementation Method 4
Filters are often employed in the image sensor to selectively transmit lights of certain wavelengths onto pixels. A Bayer filter mosaic is often formed on the image sensor. The Bayer filter is a color filter array that arranges one of the RGB color filters on each of the color pixels
Data Source
Figure 1~2A
Figure 2B~3B
Figure 3C~4
AI summary
A method of synthesizing a higher resolution image from a plurality of images captured from different viewpoints comprises capturing and processing (510, 710) a plurality of images using a plurality of imagers in a camera array, wherein a subset of imagers in the camera array includes telephoto lenses and the subset of imagers includes non-telephoto lenses. Parallax compensation is performed (522, 720) to resolve differences in the fields-of-views of the imagers due to spatial separations between the imagers. Portions of the plurality of images are aligned (514, 734) to compensate for parallax. Super-resolution processing is performed (526, 724) to obtain a higher resolution image, wherein the higher resolution image has a resolution that is greater than the resolutions of the images captured by the plurality of imagers.