Stacked Image Sensor Inference for Lower Host Data Traffic
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image sensing systems in autonomous vehicles face challenges in efficiently processing and transmitting large amounts of image data, leading to bandwidth constraints and increased processing loads on central processing units (CPUs).
Innovation Solution
An integrated image sensing device is developed, comprising an image sensor, a memory device, and an inference engine. This device processes images and generates inference results, reducing the need to transmit raw pixel data to a host system. The inference engine is optimized for machine learning tasks, and the memory device provides high bandwidth connections to support neural network computations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If raw image data is transmitted to the host system for processing, then the host system can perform comprehensive analysis, but the data traffic and bandwidth requirements increase significantly
Solution Approach 1:
The system segments the image processing workflow into two parts: preliminary processing (inference engine) and comprehensive analysis (host system). The inference engine performs initial object detection and generates inference results, while the host system performs detailed analysis only on selected images, reducing data traffic while maintaining analysis accuracy.
Solution Approach 2:
The inference engine performs preliminary processing of images before they reach the host system. By executing inference algorithms locally to generate inference results (such as object detection, classification, or segmentation), the system prepares data in advance, allowing the host system to focus on comprehensive analysis without receiving all raw image data.
2Quantity of substance
If an inference engine is integrated into the image sensing device, then data traffic to the host system is reduced, but the device complexity increases
Solution Approach 1:
The inference engine is merged with the image sensing device by integrating it into the sensor controller. This combination allows the system to perform inference processing locally without requiring separate hardware components, reducing data traffic to the host system while managing device complexity through unified architecture.
3Measurement precision
If all image data is processed by the host system, then comprehensive analysis can be performed, but the processing load on the CPU increases
Solution Approach 1:
The processing workload is segmented between the inference engine (local processing) and the host system (comprehensive analysis). The inference engine handles initial processing of all images to generate inference results, while the host system performs comprehensive analysis only on selected images, distributing processing power requirements and reducing the burden on the CPU.
Solution Approach 2:
The inference engine performs preliminary processing of images locally, generating inference results that summarize key information. This preliminary action allows the host system to perform comprehensive analysis on reduced datasets, maintaining analysis completeness while reducing processing power requirements.
4Productivity
If the image sensor operates at high frame rates, then more images are captured, but the bandwidth required to transmit all images increases
Solution Approach 1:
The inference engine performs preliminary processing of images at high frame rates locally, generating inference results that capture essential information from each frame. This allows the system to maintain high productivity (frame rate) while transmitting only the processed inference results to the host system, significantly reducing bandwidth requirements.
Data Source
AI summary
Systems, methods and apparatus of integrated image sensing devices. In one example, a system includes a sensor that generates data. A memory device stores the generated data, and further stores a first portion of an artificial neural network (ANN). A host interface of the system is configured to communicate with a host system that stores a second portion of the ANN. The memory device can be stacked with the sensor. The memory device includes an inference engine configured to generate inference results using the stored data as input to the first portion of the ANN. The host interface is further configured to send the inference results to the host system for processing by the host system using the second portion of the ANN.