In-Pixel Video Processor With Analog Neural Edge Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video processors for edge computing are expensive, energy inefficient, and experience substantial latency due to remote data processing in centralized data centers, which is not suitable for real-time video processing in smart city applications.
Innovation Solution
A video processor capable of in-pixel processing using an analog neural network integrated with a photosensor array, performing saccade processing and salient feature extraction to reduce data volume, enabling local processing and minimizing bandwidth requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If video processing is performed in centralized data centers using conventional digital systems, then processing capability and accuracy are improved, but energy consumption increases, latency increases, and cost increases
Solution Approach 1:
The patent segments the video processing system into two parts: (1) an analog neural network at the edge device for initial processing and feature extraction, and (2) a digital system in the data center for final analysis. This segmentation allows energy-intensive digital processing to be minimized while maintaining accuracy, as the analog portion handles the bulk of processing locally with very low power consumption.
Solution Approach 2:
The patent introduces an analog neural network as an intermediary between the photosensor array and the digital processing system. This intermediary performs preliminary processing, feature extraction, and data filtering, transforming raw video data into condensed feature representations that require minimal digital processing, thereby reducing the energy burden on the centralized system while preserving processing accuracy.
2Power
If video processing is performed in centralized data centers, then processing power is sufficient, but latency is substantial and real-time processing is not achieved
Solution Approach 1:
The patent implements preliminary action by performing feature extraction, object detection, and data filtering at the edge device before data is transmitted to the data center. The analog neural network pre-processes video frames, identifies salient features, and prepares condensed data representations in advance, so that when data arrives at the centralized system, minimal additional processing is required, dramatically reducing latency while maintaining sufficient processing power.
3Measurement precision
If full video frames are transmitted for processing, then processing accuracy is maintained, but data volume is huge and bandwidth requirements are excessive
Solution Approach 1:
The patent extracts only the essential and salient features from full video frames using the analog neural network. Instead of transmitting complete high-resolution frames, the system extracts key objects, motion vectors, color histograms, and other discriminative features, reducing data volume by more than 1000x while preserving the information necessary for accurate processing and analysis at the data center.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Enables low-latency, energy-efficient, and cost-effective video processing at the network edge, reducing data by 10× and power consumption to less than 30 mW per megapixel, suitable for real-time object tracking and classification.
Implementation Method 1
an analog neural network to select, within the video frame, at least one patch of pixels and process the at least one pixel patch to produce a patch representation
Data Source
AI summary
Method and apparatus of processing a sequence of video frames comprising generating at least one video frame and using an analog neural network to select, within the at least one video frame, at least one patch of pixels and process the at least one pixel patch to produce a patch feature for each of the at least one pixel patches. The method digitizes the patch feature, identifies objects within the digitized patch feature, and tracks the objects to generate control information that is used by the analog neural network to select and process the pixel patches.