Sparse RGB Image Sensors for AR Passthrough
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current artificial reality systems, such as AR/VR devices, provide limited color passthrough capabilities due to the use of monochrome pixel sensors, which are costly to replace and have lower quantum efficiency compared to panchromatic sensors, necessitating a cost-effective solution for enabling color passthrough while maintaining sufficient quantum efficiency.
Innovation Solution
Implementing a sparse RGB pattern within global shutter sensors, where the RGGB pattern accounts for a percentage of the total pixel array, and using an image signal processor to de-mosaic and interpolate color information from panchromatic pixels, allowing for accurate color representation by determining mean or median values based on brightness gradients.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If monochrome pixel sensors are used in artificial reality systems, then quantum efficiency is improved and cost is reduced, but color passthrough capability is lost
Solution Approach 1:
The pixel array is segmented into multiple types of pixel sensors including color pixel sensors (with RGB filters) and panchromatic pixel sensors (without color filters). This segmentation allows the system to capture both color information and high-efficiency luminance information simultaneously, enabling color passthrough while maintaining overall quantum efficiency through the combined data from different pixel types.
2Adaptability or versatility
If color pixel sensors are used to enable color passthrough, then color representation is improved, but quantum efficiency decreases and cost increases
Solution Approach 1:
Different regions of the pixel array have different local qualities - some pixels have color filters (RGB) while others have no filters (panchromatic). This local differentiation allows the system to optimize for color capture in specific regions while maintaining high quantum efficiency in other regions, achieving color passthrough without sacrificing overall system quantum efficiency.
3Adaptability or versatility
If all pixel sensors are replaced with color sensors, then color passthrough is enabled, but manufacturing cost increases significantly
Solution Approach 1:
Instead of converting the entire pixel array to color sensors, the system implements color pixel sensors in only a portion of the array (e.g., 25-50% coverage). This partial action approach enables color passthrough functionality while minimizing the number of expensive color filters required, thereby reducing manufacturing costs compared to a full color sensor array.
Data Source
AI summary
In one embodiment, a computing system may receive sensor data from an image sensor having a pixel array including color pixel sensors and panchromatic pixel sensors in a first pattern. Each of the color pixel sensors is associated with one of several color channels. The computing system may generate, based on the sensor data, a filtered monochrome image including monochrome values corresponding to the pixel array of the image sensor. The computing system may generate a filtered color image having a second pattern of color channels. A first pixel of a particular color channel at a first pixel location in the filtered color image is determined based on the monochrome value corresponding to the first pixel location in the filtered monochrome image, the sensor data measured by a color pixel sensor at a second pixel location, and the monochrome value at the second pixel location in the filtered monochrome image.