Surveillance Cameras with Analog Inference for On-Chip Image Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transmitting large amounts of image data from image sensors to general-purpose microprocessors for processing is inefficient, especially for tasks like image segmentation and object recognition, due to the intensive computations involved in matrix multiplications and accumulations.
Innovation Solution
Implementing an integrated circuit device with an image sensing pixel array, memory cell array, and inference logic circuit that performs computations such as artificial neural networks using hybrid bonding to connect an image sensor chip and memory chip, enabling on-device multiplication and accumulation operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If image data is transmitted from image sensors to general-purpose microprocessors for processing, then image processing tasks can be performed, but transmission efficiency is low and processing speed is slow due to intensive matrix multiplication and accumulation operations
Solution Approach 1:
The patent merges the image sensor, memory cells, and inference logic circuit into a single integrated circuit device. This integration eliminates the need to transmit image data from the sensor to external processors, as all processing operations are performed on-chip. The memory cells store weight data for neural network operations, while the inference logic circuit performs matrix multiplication and accumulation directly on the captured image data, resolving the contradiction between processing capability and transmission time.
Solution Approach 2:
The patent introduces memory cells as an intermediary between the image sensor and the inference logic circuit. These memory cells store weight data that is used by the inference logic circuit to perform neural network computations. This intermediary structure enables efficient on-chip processing by providing fast access to weight data during matrix multiplication operations, eliminating the need for external memory access and reducing overall processing time.
2Productivity
If specialized circuits are implemented for acceleration of multiplication and accumulation operations, then computational performance is improved, but device complexity increases
Solution Approach 1:
The patent implements memory cells that serve multiple functions: they store weight data for neural network operations, provide fast data access during computations, and enable both matrix multiplication and accumulation operations. The inference logic circuit is designed to perform multiple types of computations including matrix multiplication, accumulation, and activation functions within a unified architecture. This multi-functionality reduces overall device complexity compared to implementing separate specialized circuits for each operation type.
Data Source
AI summary
A surveillance camera having: an image sensing pixel array operable to generate first data representative of an input image; a memory cell array having memory cells, wherein threshold voltages of the memory cells are programmable in a first mode to store weight matrices and programmable in a second mode to store second data representative of an output image generated from the input image; voltage drivers; current digitizers; an inference logic circuit operable to perform operations of multiplication and accumulation using the voltage drivers, the current digitizers, and a first portion of the memory cells programmed in the first mode to store the weight matrices used in generation of the output image; a transceiver; and a microprocessor configured to use the transceiver to communicate, to a computer system, a report identifying the output image stored in the memory cell array.