Stacked Image Sensor Memory for On-Device Inference Traffic Reduction

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Autonomous driving systems face challenges in processing high volumes of image data from sensors, leading to bandwidth constraints and increased computational loads on central processing units, which can limit frame rates and accuracy, especially under low-light conditions.

Innovation Solution

An integrated image sensing device with a memory device and an inference engine is used to process images locally, converting them into inference results rather than transmitting raw pixel data, thereby reducing data traffic and offloading processing tasks from the host system.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If raw image data is transmitted from sensor to host system, then data transmission bandwidth is utilized, but processing load on central system increases and frame rate decreases

Engineering Contradiction:

Improveframe rateVSAvoidprocessing load on host system

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system segments the processing task by separating image capture (image sensor), local processing (inference engine in memory device), and result transmission (host system). The inference engine processes images locally within the memory device stack, dividing the overall processing load between edge device and central system, thereby increasing frame rate while managing complexity.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent transitions from a two-dimensional architecture (sensor directly connected to host) to a three-dimensional stacked architecture (sensor-memory-inference engine host). By stacking the inference engine and memory device onto the image sensor substrate, the system adds a vertical dimension for local processing, reducing data transmission requirements and improving frame rate.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Measurement precision

If more image data is processed by the host system, then analysis accuracy improves, but power consumption and processing load increase

Engineering Contradiction:

Improveanalysis accuracyVSAvoidpower consumption

Core Design Contradiction:

Measurement precisionVSUse of energy by moving object

Solution Approach 1:

The inference engine performs preliminary processing of image data locally before transmission to the host system. By conducting initial analysis (object detection, feature extraction) at the edge device, the system reduces the amount of data requiring full host processing, thereby maintaining analysis accuracy while reducing power consumption and processing load on the central system.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS20220036164A1Neuromorphic memory and inference engine stacked with image sensor to reduce data traffic to host

Publication Date: 2022.02.03 MICRON TECHNOLOGY INC

AI summary

Systems, methods and apparatus of integrated image sensing devices. In one example, a system includes an image sensor that generates image data. A memory device is stacked with the image sensor and stores the generated image data. A host interface communicates with a host system. The memory device includes an inference engine to generate inference results using the stored image data as input to an artificial neural network. The inference engine includes a neural network accelerator configured to perform matrix arithmetic computations on the data stored in the memory device. The host interface sends the inference results to the host system for processing.