Change-Based Neural Network Inference for Low-Power Video Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural networks for computer vision and image processing tasks require significant computational resources, memory, and power, making them infeasible for resource-constrained devices like mobile phones and cameras, and they inefficiently process entire video frames without exploiting temporal and spatial sparsity.
Innovation Solution
Implementing low-power change-based neural network inference techniques that leverage key layers and compressed tensor comparisons to exploit temporal redundancy in videos, reducing computational burden and data transfer bandwidth by identifying key layers and using lossy-compressed projection tensors for in-place computations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional neural networks are used for computer vision and image processing tasks, then processing accuracy can be maintained, but computational resources, memory, and power consumption increase significantly
Solution Approach 1:
The patent segments the video processing into key frame identification and change detection components. By identifying key frames where significant changes occur and only processing those frames through the full neural network pipeline, the system maintains processing accuracy for important moments while reducing overall power consumption by skipping redundant processing of similar frames.
Solution Approach 2:
The patent applies partial action by performing complete neural network processing only on key frames that contain significant changes, rather than processing every frame. For non-key frames, the system uses change detection based on previous processing results, thereby reducing power consumption while maintaining accuracy when it matters most.
2Reliability
If conventional neural networks process entire video frames, then comprehensive analysis is achieved, but computational burden and data transfer bandwidth increase
Solution Approach 1:
The patent extracts and processes only the essential information from video frames by identifying key frames with significant changes. Instead of analyzing every frame in detail, the system extracts temporal redundancy and focuses computational resources on frames that contain meaningful changes, thereby reducing computational burden while maintaining analysis completeness for important events.
Solution Approach 2:
The patent performs preliminary action by pre-identifying key frames based on change detection before applying the full neural network processing. This preliminary filtering step reduces the number of frames that require intensive computational analysis, thereby reducing overall computational burden while ensuring that frames requiring comprehensive analysis are properly identified and processed.
3Manufacturing precision
If full neural network processing is applied to every video frame, then processing thoroughness is maintained, but data transfer bandwidth and memory usage increase
Solution Approach 1:
The patent segments the video data processing by dividing frames into key frames requiring full processing and non-key frames requiring minimal processing. This segmentation reduces the total volume of data that needs to be transferred and processed thoroughly, thereby reducing data transfer bandwidth requirements while maintaining processing thoroughness for frames that require it.
Solution Approach 2:
The patent applies partial processing action by performing complete neural network analysis only on key frames identified through change detection. For non-key frames, the system uses simplified processing or skips analysis entirely, thereby reducing data transfer bandwidth and memory usage while maintaining processing thoroughness for frames where it is necessary.
Data Source
AI summary
One or more aspects of the present disclosure enable high accuracy computer vision and image processing techniques with decreased system resource requirements (e.g., with decreased computational load, shallower neural network designs, etc.). As described in more detail herein, one or more aspects of the described techniques may leverage key layers (e.g., certain key layers of a neural network) and compressed tensor comparisons to efficiently exploit temporal redundancy in videos and other slow changing signals (e.g., to efficiently reduce neural network inference computational burden, with only minor increase in data transfer power consumption). For example, key layers of a neural network may be identified, and temporal/spatial redundancy across frames may be efficiently leveraged such that only a computation region in a subsequent frame n+1 is re-computed in layers between identified key layers, while remaining feature-map calculations may be disabled in the layers between the identified key layers.