3D-Stacked CMOS Image Sensor HARP for Low-Latency Edge Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing imaging systems lack integration of computing and sensing functionalities at the network edge, leading to inefficiencies in power consumption, latency, and data transfer, particularly with gigapixel image sensors.
Innovation Solution
A 3D-stacked complementary metal-oxide-semiconductor (CMOS) image sensor with a hierarchical attention-oriented region-based processing (HARP) module that performs AI-based knowledge inference at the source of image data, integrating sensing and computing functions to reduce power and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If image data is processed at a remote data processor, then comprehensive signal processing can be performed, but data transmission overhead and latency increase
Solution Approach 1:
The processing system is segmented into multiple layers: a first processing circuit layer performing pre-processing operations (noise filtering, region of interest identification) and a second processing circuit layer performing inference-based processing. This segmentation allows critical processing to occur closer to the sensor while maintaining comprehensive analysis capabilities, thereby reducing latency without sacrificing processing completeness.
Solution Approach 2:
The patent introduces a hierarchical processing architecture that adds a spatial dimension to data processing by placing processing circuits in close proximity to the sensor array. This near-sensor processing layer operates in parallel with remote processing, creating a multi-dimensional processing framework that simultaneously reduces transmission latency and maintains comprehensive signal analysis.
2Measurement precision
If all image data is transmitted to the data processor, then complete analysis is possible, but data transmission overhead and power consumption increase
Solution Approach 1:
The first processing circuit layer extracts and filters redundant spatiotemporal data from the full image data stream, identifying and retaining only salient features and regions of interest. This extraction process transmits a compressed subset of data to the second processing layer, significantly reducing transmission overhead and power consumption while preserving the information necessary for complete analysis.
Solution Approach 2:
The patent changes the parameter of data representation by transforming raw image data into a compressed feature space through pre-processing operations. By altering the data format and dimensionality in the near-sensor layer, the system reduces the volume of data requiring transmission and processing while maintaining the essential information content needed for comprehensive analysis.
3Loss of energy
If a hierarchical processing architecture is implemented, then data transmission is reduced, but device complexity increases
Solution Approach 1:
The processing circuits are designed with multi-functionality, where the same hardware components can perform various operations including noise filtering, region identification, feature extraction, and inference processing. This universal design approach reduces the need for specialized dedicated circuits for each function, thereby implementing a hierarchical architecture while limiting the increase in overall device complexity.
Data Source
AI summary
An in/near-sensor artificial intelligence (AI) architecture that facilitates knowledge inference at a source of data, wherein the architecture comprises a three-dimensional (3D) stacked complementary metal-oxide-semiconductor (CMOS) image sensor that comprises a multi-layer computational structure that enables AI-based knowledge inference before readout electronics. The multi-layer computational structure comprises a hierarchical attention-oriented region-based processing (HARP) module that is configured between a processing unit and readout circuitry that is coupled to an image sensor.