Image Sensor Inference Partitioning for Lower Host Data Traffic
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image sensing systems in autonomous vehicles face challenges in efficiently processing and transmitting high volumes of image data, leading to bandwidth constraints and increased processing loads on central processing units (CPUs).
Innovation Solution
An integrated image sensing device is proposed, which includes an image sensor, a memory device, and an inference engine. This device processes images and generates inference results, reducing the need to transmit raw pixel data to a host system. The inference engine is optimized for machine learning tasks and can perform analytics such as pixel segmentation, feature extraction, and object detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If raw image data is transmitted from the image sensor to the host system, then the host system can perform comprehensive processing and analysis, but the data traffic and bandwidth requirements increase significantly
Solution Approach 1:
The patent extracts and performs preliminary processing operations (inference, feature extraction, object detection) directly at the image sensor device before data transmission. This removes the burden of transmitting and processing all raw pixel data at the host system, sending only processed inference results instead of complete raw image streams.
Solution Approach 2:
The patent segments the processing workflow into two parts: initial inference processing performed at the image sensor device, and subsequent detailed analysis performed at the host system. This segmentation allows raw data to be processed locally first, then only essential results are transmitted, reducing overall data traffic while maintaining processing accuracy.
2Quantity of substance
If an inference engine is integrated into the image sensor device, then data traffic to the host system is reduced, but the device complexity increases
Solution Approach 1:
The patent merges the inference engine with the image sensor device, integrating processing capabilities directly into the sensing component. This combination allows the device to perform both image capture and initial inference processing, reducing the need to transmit raw data while managing complexity through integrated design.
Solution Approach 2:
The image sensor device is enhanced with multi-functionality by incorporating an inference engine that can perform various processing tasks (feature extraction, object detection, classification). This universal processing capability allows the device to handle multiple functions within a single integrated unit, reducing overall system complexity despite the added capabilities.
3Measurement precision
If all image data is processed by the central processing unit, then comprehensive analysis can be performed, but the processing load and energy consumption increase
Solution Approach 1:
The patent performs preliminary inference processing and feature extraction at the image sensor device before data transmission to the host system. This preliminary action reduces the volume of data requiring intensive processing at the CPU, thereby lowering the processing energy consumption while still enabling comprehensive analysis at the host system for critical tasks.
Data Source
AI summary
Systems, methods and apparatus of integrated image sensing devices. In one example, a system includes a sensor that generates data. A memory device stores the generated data, and further stores a first portion of an artificial neural network (ANN). A host interface of the system is configured to communicate with a host system that stores a second portion of the ANN. The memory device can be stacked with the sensor. The memory device includes an inference engine configured to generate inference results using the stored data as input to the first portion of the ANN. The host interface is further configured to send the inference results to the host system for processing by the host system using the second portion of the ANN.