Neuromorphic Image Sensors With In-Pixel Analog MACs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computer vision tasks using neuromorphic cameras face energy, latency, and throughput bottlenecks due to compute-intensive determinations, particularly in remote image sensors, necessitating energy-efficient computing solutions for on-edge intelligent machine vision applications.
Innovation Solution
An asynchronous processing-in-pixel-in-memory (P2M) paradigm integrates multi-bit multi-channel weights inside the pixel array using analog multiply and accumulate (MAC) blocks, enabling massively parallel spatio-temporal convolution operations and reducing energy consumption by performing computations closer to the sensor.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If compute-intensive determinations are performed remotely from image sensors, then processing power and accuracy are improved, but energy consumption increases and latency increases
Solution Approach 1:
The patent transitions from traditional remote processing architecture to in-pixel processing by embedding computational elements (weighting transistors, accumulation capacitors) directly within the pixel array structure. This spatial reorganization moves computation from a separate remote processor to the sensor plane itself, eliminating data transfer bottlenecks and reducing energy consumption while maintaining processing accuracy through parallel analog MAC operations.
Solution Approach 2:
The patent introduces analog multiply-accumulate (MAC) blocks as intermediary computational units between the photodetector and readout circuitry. These MAC blocks perform weighted summation of pixel signals using transistor-based weighting elements and capacitor-based accumulation, enabling feature extraction at the sensor level without requiring high-energy digital processing remotely.
2Power
If compute-intensive determinations are performed remotely from image sensors, then processing power is improved, but throughput decreases due to bottlenecks
Solution Approach 1:
The patent divides the pixel array into multiple independently operable units, each with its own weighting transistors and accumulation capacitors. This segmentation enables parallel processing of multiple regions or features simultaneously, increasing throughput while maintaining the processing power needed for compute-intensive determinations through distributed analog MAC operations.
3Use of energy by moving object
If in-pixel processing is implemented, then energy efficiency is improved, but device complexity increases
Solution Approach 1:
The patent merges the computational functions (weighting and accumulation) with the sensing function within the same pixel structure. By integrating weighting transistors, accumulation capacitors, and readout circuitry into the pixel array, the design achieves in-pixel processing capability without requiring entirely separate processing circuits, thus improving energy efficiency while controlling the increase in device complexity through functional integration.
Solution Approach 2:
The patent designs the pixel structure to perform multiple functions: photodetection, analog weighting, temporal accumulation, and feature extraction. This multi-functionality is achieved by making the pixel elements serve dual purposes—for example, the same transistor network used for signal readout also performs weighted summation operations, reducing the need for dedicated processing components and managing device complexity.
4Loss of information
If frame-based image processing is used, then complete image information is captured, but memory and bandwidth requirements increase
Solution Approach 1:
The patent extracts and processes only the essential feature information (e.g., motion, contrast changes, temporal derivatives) at the pixel level through analog MAC operations. By performing feature extraction in-pixel before digital conversion and data transfer, the system transmits only processed feature data rather than complete raw frame data, significantly reducing memory and bandwidth requirements while retaining the most salient visual information for downstream processing.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The P2M paradigm achieves 1.95× lower energy consumption with 88.36% test accuracy on the IBM DVS128-Gesture dataset compared to state-of-the-art methods, while maintaining substantial accuracy, and reduces memory and bandwidth requirements by processing event-based rather than frame-based images.
Implementation Method 1
Each pixel element may include a photodetector
Data Source
AI summary
Provided is an integrated circuit comprising: a sensor structure; a set of weighting elements, each configured to weight an output of the sensor structure; and an output accumulation element, the output accumulation element configured to collect weighted outputs of the set of weighting elements over an accumulation time.