Image Sensor Color Filter Arrays With Neural Readout Remapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image generation technology is not well-compatible with different types of colour filter arrays (CFAs), leading to time-consuming and computationally-intensive processing, and is inefficient in generating high-quality images with high resolution, wide field of view, and high frame rate, limiting its suitability for applications like extended-reality devices.
Innovation Solution
An imaging system and method utilizing a neural network to generate output image data according to a colour pattern different from the physical colour filter array of an image sensor chip, enabling efficient image signal processing without hardware or software changes, and supporting high-quality, realistic images at high framerate.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing image generation technology is used with standard Bayer CFA, then image processing can be performed, but it is not compatible with other types of CFAs (quad-Bayer, nona-Bayer, hexadeca-Bayer, RGBIR, RGBW, RCCB) requiring time-consuming and computationally intensive changes to software/firmware algorithms
Solution Approach 1:
The patent creates a virtual copy of the physical CFA pattern through neural network training. The neural network learns the mapping between the physical CFA pattern and the corresponding color information, allowing the system to process images from various CFA types (Bayer, quad-Bayer, nona-Bayer, hexadeca-Bayer, RGBIR, RGBW, RCCB) using a unified processing pipeline without requiring hardware or software changes for each CFA type.
Solution Approach 2:
The patent changes the parameter of color pattern representation by using a neural network to transform the physical CFA pattern into a virtual color pattern. This allows the same image signal processing pipeline to handle different physical CFA configurations by converting them into a standardized virtual representation that the pipeline can process efficiently.
2Measurement precision
If existing image generation technology processes image signals from all pixels, then high resolution can be achieved, but it requires considerable processing resources, involves long processing time, and limits the total number of pixels that can be arranged on the image sensor for full pixel readout at a given frame rate
Solution Approach 1:
The patent uses a neural network to create a virtual copy of the color filter array pattern that enables efficient image signal processing. This virtual CFA allows the system to process images from a subset of pixels (e.g., every other pixel or a reduced grid) while maintaining high resolution through the neural network's upscaling and color reconstruction capabilities, thereby increasing the effective number of processable pixels and improving frame rate.
Solution Approach 2:
The patent adds a computational dimension by using neural networks to transform spatial sampling patterns. Instead of being limited by the physical pixel arrangement, the system can effectively work with subsampled data and reconstruct high-resolution images through the neural network's transformation from the subsampled domain to the full-resolution domain, decoupling the frame rate limitation from the physical pixel count.
3Measurement precision
If image signals are processed with full pixel readout to generate high quality images, then high resolution and accurate color reproduction are achieved, but computational resources and processing time increase significantly
Solution Approach 1:
The patent trains a neural network to copy and reconstruct color information from reduced sampling. The network learns to accurately reproduce color data from subsampled inputs, allowing the system to process fewer physical pixels while maintaining high color accuracy and image quality, thereby reducing computational power consumption and energy usage.
Solution Approach 2:
The patent applies partial action by processing only a subset of pixels (e.g., every other pixel or a reduced grid) instead of all pixels. The neural network then reconstructs the missing information, achieving the same or better image quality with significantly reduced computational resources and energy consumption compared to full pixel processing.
Data Source
AI summary
An imaging system has an image sensor chip and processor(s). The image sensor chip has a photo-sensitive surface with photo-sensitive cells; a colour filter array with physical smallest repeating units, wherein a given physical smallest repeating unit has array(s) of a first type of colour filters, array(s) of a second type of colour filters, and array(s) of a third type of colour filters; and a controller configured to employ neural network(s) during read out of image data. The processor(s) is/are configured to: receive output image data, wherein a smallest repeating unit in a colour pattern of at least a part of the output image data is different from the given physical smallest repeating unit; and generate an output image.