PISA In-Sensor Accelerator With Non-Volatile Memory For IoT Imaging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current IoT imaging systems face challenges with high power consumption (>90%) in converting and storing image pixel values, high latency and power consumption in cloud-based computations, and large area-overhead and power consumption in edge processing-in-sensor units, limiting their efficiency and accuracy.
Innovation Solution
A Processing-In-Sensor Accelerator (PISA) design that integrates sensing and processing with non-volatile magnetic memory, enabling low-overhead, dual-mode, and reconfigurable operations for efficient data conversion and transmission, single-cycle in-sensor processing, and highly parallel processing to reduce power consumption and increase throughput, while leveraging non-volatile magnetic memory for reduced standby power and instant wake-up capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If conventional image sensors convert and store image pixel values using traditional memory and processing units, then the imaging system can capture and process images, but the power consumption exceeds 90% of total system power
Solution Approach 1:
The patent merges sensing and computing functions into a single integrated sensor unit. The sensor directly performs neural network computations on captured images using embedded weights and biases stored in non-volatile memory, eliminating the need to transfer raw pixel data to separate processing units. This integration eliminates redundant data movement and reduces power consumption while maintaining processing efficiency.
Solution Approach 2:
The patent introduces non-volatile memory as an intermediary between the sensor and processing units. This memory stores weights and biases locally at the sensor, enabling in-sensor computing without requiring continuous power supply or frequent data transfers. The non-volatile memory acts as a mediator that enables computation while minimizing energy consumption.
2Power
If cloud-based computations are used to process image data, then the system can leverage powerful processing capabilities, but latency and power consumption increase significantly
Solution Approach 1:
The patent segments the neural network computation into two parts: the first convolutional layer is executed locally at the sensor using integrated weights and biases, while remaining layers are processed externally. This segmentation enables immediate local processing that reduces latency for critical operations while maintaining the ability to leverage external processing power for more complex computations.
Solution Approach 2:
The patent performs preliminary processing of image data at the sensor before transmission to external systems. By executing the first convolutional layer locally with pre-stored weights and biases, the system prepares and reduces the data set before cloud processing, thereby reducing transmission time and overall latency while maintaining access to powerful external processing capabilities.
3Speed
If edge processing-in-sensor units are implemented to reduce cloud dependency, then processing speed improves, but area overhead and power consumption increase
Solution Approach 1:
The patent implements a universal sensor architecture that can operate in multiple modes: capturing raw images when external processing is available, and performing neural network computations when operating independently. The sensor includes configurable computational units with stored weights and biases that can be activated based on system needs, providing multi-functionality without requiring dedicated hardware for each mode.
Solution Approach 2:
The patent implements local computational capabilities within the sensor by embedding weights and biases in non-volatile memory at the sensor location. This local quality enables the sensor to perform computations independently when needed, improving processing speed and reducing cloud dependency, while the non-volatile memory ensures these computational resources are available without continuous power supply.
4Speed
If volatile memory is used to store processing data in sensor units, then fast access is achieved, but standby power consumption increases
Solution Approach 1:
The patent changes the state of memory from volatile to non-volatile, allowing weights and biases to be stored without continuous power supply. This parameter change enables the sensor to maintain computational capabilities in standby mode with minimal power consumption, while still providing fast access to stored weights and biases when active through the non-volatile memory interface.
Data Source
AI summary
Disclosed is a Processing-In-Sensor Accelerator (PISA) that provides a flexible, energy-efficient, and high-performance solution for real-time and smart image processing in AI devices. PISA implements a coarse-grained convolution operation in Binarized-Weight Neural Networks (BWNNs) leveraging a novel compute-pixel with non-volatile weight storage at the sensor side. This reduces power consumption of data conversion and transmission to an off-chip processor. A bit-wise near-sensor in-memory computing unit processes the remaining network layers. Once the object is detected, PISA switches to typical sensing mode to capture the image for a fine-grained convolution using only a near-sensor processing unit. The circuit-to-application co-simulation results on a BWNN acceleration demonstrate minor accuracy degradation on various image datasets in coarse-grained evaluation compared to baseline BWNN models. PISA achieves a frame rate of 1000 and efficiency of 1.74 TOp/s/W. PISA reduces data conversion and transmission energy by at least 84% compared to a baseline.