Stacked Image Sensor Architecture for Low-Latency Object Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current image sensors rely heavily on external processors for data processing, leading to increased bandwidth requirements and latency, as they transmit raw image data for extensive processing, which is inefficient and slows down object detection in applications like autonomous vehicles.
Innovation Solution
The image sensor incorporates multiple stacked integrated circuit layers, including pixel sensor groups, image processing circuitry groups, and neural network circuitry groups, allowing for distributed processing and reducing the need for external processing by transmitting only processed data and neural network output, thereby enhancing processing efficiency and reducing bandwidth usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If image sensors transmit raw image data for external processing, then object detection can be performed, but bandwidth requirements increase and latency increases
Solution Approach 1:
The image sensor is divided into multiple stacked integrated circuit layers, each performing specific processing functions. The first layer captures raw image data, the second layer performs initial image processing, and the third layer conducts neural network-based object detection. This segmentation allows processing to be distributed across layers, reducing the need to transmit complete raw image data externally while maintaining detection accuracy.
Solution Approach 2:
The image sensor performs preliminary processing operations before data leaves the sensor. Image processing circuitry in the second integrated circuit layer pre-processes the captured data, and neural network circuitry in the third layer performs initial object detection. This preliminary action reduces the amount of data that needs to be transmitted externally, thereby reducing bandwidth requirements while maintaining detection capability.
2Productivity
If image sensors rely on external processors, then processing can be performed, but processing speed decreases due to latency
Solution Approach 1:
Multiple functional layers are merged into a single integrated sensor assembly. The pixel sensor layer, image processing layer, and neural network layer are stacked and integrated together, allowing data to flow seamlessly between layers without external transmission. This merging eliminates communication latency between separate processing units and enables faster end-to-end processing.
Solution Approach 2:
The patent transitions from a two-dimensional planar sensor architecture to a three-dimensional stacked architecture. By adding the vertical dimension with multiple integrated circuit layers, the system enables parallel processing operations to occur simultaneously at different depths, increasing processing throughput without increasing the sensor's footprint area.
3Loss of information
If all image data is transmitted for processing, then complete analysis can be performed, but bandwidth requirements increase
Solution Approach 1:
The neural network circuitry extracts only the essential information - object detection results and relevant features - from the captured image data. Instead of transmitting complete raw image data, the system extracts and transmits only the critical detection outcomes, thereby maintaining information quality while significantly reducing bandwidth requirements.
Solution Approach 2:
Different regions of the image data receive different levels of processing and transmission priority. The neural network circuitry identifies regions containing objects of interest and processes/transmits this data with higher fidelity, while less critical regions are processed with lower resources. This local quality approach ensures important information is preserved while reducing overall data transmission requirements.
Data Source
AI summary
A image sensor includes a first integrated circuit layer including pixel sensors that are grouped based on position into pixel sensor groups, a second integrated circuit layer in electrical communication with the first integrated circuit layer, the second integrated circuit layer including image processing circuitry groups that are configured to each receive pixel information from a corresponding pixel sensor group, the image processing circuitry groups further configured to perform image processing operations on the pixel information to provide processed pixel information during operation of the image sensor, a third integrated circuit layer in electrical communication with the second integrated circuit layer, and the third integrated circuit layer including neural network circuitry groups that are configured to each receive the processed pixel information from a corresponding image processing circuitry group and perform analysis for object detection on the processed pixel information during operation of the image sensor.