On-Chip Compute Circuit for Image Sensor
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional sensor devices in artificial reality systems, such as head-mounted displays, face challenges with high power consumption and latency due to processing large amounts of data for tasks like eye tracking, hand tracking, and wide field-of-view scanning, which limits their performance and user experience.
Innovation Solution
A sensor assembly with stacked sensor layers, including a photodetector layer and semiconductor layers with on-chip compute circuits, such as a machine learning model accelerator, to process pixel data efficiently, reducing power consumption and latency by performing computations within the chip and dynamically adjusting sensor operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional sensor devices process large amounts of data for tasks like eye tracking, hand tracking, and wide field-of-view scanning, then measurement precision and functionality are improved, but power consumption and latency increase significantly
Solution Approach 1:
The patent segments the sensor system into multiple specialized layers: photodetector layer for light capture, ADC layer for analog-to-digital conversion, feature extraction layer for processing, and neural network layer for complex pattern recognition. Each layer handles specific processing tasks, enabling efficient data reduction before transmission while maintaining tracking precision.
Solution Approach 2:
The patent transitions from conventional 2D planar sensor architecture to a 3D stacked sensor architecture with multiple functional layers vertically arranged. This dimensional change enables parallel processing across layers, reducing power consumption and latency while maintaining high measurement precision for tracking applications.
2Measurement precision
If conventional sensor devices perform computationally intensive operations to capture and process large amounts of data, then functionality and measurement precision are improved, but processing speed deteriorates due to saturation
Solution Approach 1:
The patent implements preliminary action by performing analog-to-digital conversion and feature extraction operations within the sensor chip itself before data leaves the sensor. The ADC layer converts analog signals to digital format, and the feature extraction layer pre-processes data to identify relevant features, reducing the computational burden on external processors and improving overall processing speed.
Solution Approach 2:
The patent introduces an intermediary processing layer (feature extraction layer and neural network layer) between the photodetector layer and external processors. This intermediary performs computationally intensive operations locally, filtering and extracting essential features before transmitting reduced data sets, thereby maintaining feature detection accuracy while improving processing speed.
3Loss of information
If conventional sensor devices capture and process large amounts of data from the surrounding area, then comprehensive information is obtained, but latency increases due to processing requirements
Solution Approach 1:
The patent extracts essential information from large data sets through specialized processing layers. The feature extraction layer identifies and extracts relevant features from captured images, and the neural network layer further processes this extracted data to identify objects, gestures, or patterns. This extraction approach maintains information completeness for tracking purposes while significantly reducing data volume and processing latency.
4Adaptability or versatility
If conventional sensor devices are designed to handle complex tracking and scanning tasks, then adaptability and functionality are improved, but device complexity increases
Solution Approach 1:
The patent implements multi-functionality by designing a universal stacked sensor architecture that can perform multiple tracking tasks (eye tracking, hand tracking, body tracking) and scanning operations through a single integrated device. The neural network layer can be configured to recognize various patterns and objects, enabling the same hardware to adapt to different tracking requirements without increasing physical complexity.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The solution enhances the performance of sensor devices by reducing power consumption and latency, enabling more efficient data processing and improved user experience in artificial reality systems by processing data locally and dynamically managing sensor operations.
Implementation Method 1
A first sensor layer of the plurality of stacked sensor layers located on top of the sensor assembly can be implemented as a photodetector layer and includes an array of pixels. The top sensor layer can be configured to capture one or more images of light reflected from one or more objects in the local area.
Data Source
AI summary
In one example, an apparatus comprises: a first sensor layer, including an array of pixel cells configured to generate pixel data; and one or more semiconductor layers located beneath the first sensor layer with the one or more semiconductor layers being electrically connected to the first sensor layer via interconnects. The one or more semiconductor layers comprises on-chip compute circuits configured to receive the pixel data via the interconnects and process the pixel data, the on-chip compute circuits comprising: a machine learning (ML) model accelerator configured to implement a convolutional neural network (CNN) model to process the pixel data; a first memory to store coefficients of the CNN model and instruction codes; a second memory to store the pixel data of a frame; and a controller configured to execute the codes to control operations of the ML model accelerator, the first memory, and the second memory.