Intelligent Image Sensor Stack for On-Sensor AI Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing autonomous driving systems face challenges in efficiently processing high-resolution image data from sensors, leading to bandwidth constraints and processing limitations, which can hinder real-time decision-making and data transmission.
Innovation Solution
An integrated circuit device incorporating a CMOS image sensor, ISP ASIC, and DRAM, along with an AI engine and non-volatile memory, processes images locally to generate inference results, reducing data transmission by analyzing and converting high-resolution images into smaller, more manageable inference results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high-resolution image data is transmitted from sensors to the host system for processing, then the image quality and detail are improved, but the bandwidth consumption and data transmission time increase significantly
Solution Approach 1:
The patent extracts and processes only the essential features and inference results from the complete high-resolution image data at the sensor level. The ISP ASIC and AI engine perform local processing to extract meaningful information (object detection, classification, key features) while discarding redundant pixel data, thereby transmitting only the extracted inference results to the host system rather than the entire high-resolution image stream.
Solution Approach 2:
The patent performs preliminary image processing and AI inference operations directly at the sensor device before data transmission. The ISP ASIC conducts initial image signal processing (noise reduction, enhancement, color correction) and the integrated AI engine performs object detection and classification in advance, so that when data is transmitted to the host system, only the processed inference results are sent, not the raw high-resolution images.
2Productivity
If complex image processing tasks are performed centrally at the host system, then processing capabilities are improved, but the real-time decision-making speed decreases due to data transmission delays
Solution Approach 1:
The patent segments the image processing workflow into two parts: (1) preliminary processing and inference execution at the sensor device using the ISP ASIC and AI engine, and (2) final decision-making and complex analysis at the host system. This segmentation enables parallel processing where time-critical inference operations occur locally without waiting for host system communication, while less time-sensitive tasks remain centralized.
Solution Approach 2:
The patent introduces an intermediary processing layer (ISP ASIC and AI engine) between the image sensor and the host system. This intermediary performs real-time inference operations locally, acting as a buffer that prevents direct dependency between sensor data capture and host system processing, thereby reducing transmission delays and enabling faster real-time decisions.
3Quantity of substance
If an AI engine is integrated at the sensor device for local inference, then data transmission is reduced, but the device complexity increases
Solution Approach 1:
The patent merges the AI engine with the existing ISP ASIC and sensor device architecture, combining multiple functions (image capture, signal processing, and AI inference) into a single integrated device. This consolidation reduces the need for separate processing units and external communication infrastructure, offsetting the added complexity of the AI engine through functional integration.
Solution Approach 2:
The patent designs the sensor device with multi-functional capabilities, where the same hardware platform (sensor device with ISP ASIC and AI engine) performs multiple functions: capturing images, processing signals, executing AI inference, and transmitting results. This universality justifies the increased device complexity by enabling the sensor to independently perform tasks that would otherwise require separate dedicated systems.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach reduces data traffic to the host system, enhances processing efficiency, and allows for real-time decision-making in autonomous vehicles by offloading complex image processing tasks, thereby optimizing bandwidth and computational resources.
Implementation Method 1
a CMOS image sensor configured to generate image signals
Data Source
AI summary
An integrated circuit device in a single integrated circuit package, having an image sensor, memory, and an inference engine configured to convert images generated by the image sensor into inference results for transmitting to a host system for further analysis and/or generate a more efficient image stream provided in the camera interface. The intelligence provided by the inference engine can reduce or eliminate the amount of image data transmitted to the host system for processing during a typical scenario. The inference results can be automatically stored in the memory accessible by the host system as a solid state drive (e.g., using a NVMe protocol). Optionally, the integrated circuit device can have an interface to an external SSD. For example, the integrated circuit device as an SSD and/or the external SSD can be configured as a black box data recorder of an autonomous vehicle.