Spatial Data Processing With Motion-Translated CNN Activations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer vision systems require high processing power due to performing nearly identical computations on each image frame, which is inefficient for real-time applications like live video analysis, and there is an opportunity to exploit the naturally approximate nature of CNNs to reduce computational requirements.
Innovation Solution
A system and method that divides spatial data into receptive fields, performs zero-gap run length encoding, and uses bilinear interpolation and on-the-fly skip-zero decoding to predict neural network outputs, reducing the need for full CNN computations on non-key frames by using motion estimation to translate activations between frames.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full CNN computations are performed on every image frame, then processing accuracy is maintained, but processing power requirements become excessively high
Solution Approach 1:
The patent segments the CNN computation process into two parts: (1) full CNN computation performed only on key frames, and (2) approximate computation using motion estimation and interpolation performed on non-key frames. This segmentation allows the system to maintain high accuracy on important frames while using low-power approximate methods on intermediate frames, directly resolving the contradiction between accuracy and power consumption.
Solution Approach 2:
The patent applies partial action by performing complete CNN computations only when necessary (on key frames) and using partial/approximate computations (motion estimation + interpolation) for the remaining frames. This selective approach ensures sufficient processing accuracy is maintained while significantly reducing overall power consumption by avoiding excessive full computations on every frame.
2Productivity
If motion estimation and interpolation are used to reduce computations, then processing speed increases, but computational accuracy may deteriorate
Solution Approach 1:
The patent performs motion estimation as a preliminary action before the interpolation step. By first calculating motion vectors that describe how regions move between frames, the system prepares accurate transformation data that guides the subsequent interpolation process. This preliminary motion analysis ensures that the interpolation operates on physically meaningful data, maintaining accuracy while enabling faster processing.
Solution Approach 2:
The patent introduces motion vectors as an intermediary between the key frame features and the non-key frame predictions. These motion vectors act as a mediator that captures the spatial transformation information, allowing the interpolation process to accurately predict non-key frame features without performing full CNN computations, thus balancing speed and accuracy.
3Use of energy by stationary object
If approximate methods are used for non-key frames, then energy consumption decreases, but processing reliability may be compromised
Solution Approach 1:
The patent implements periodic action by systematically alternating between full CNN computations (on key frames at regular intervals) and approximate computations (on non-key frames between key frames). This periodic pattern ensures that the system regularly refreshes its knowledge with accurate full computations while maintaining low energy consumption during intermediate periods, thus preserving reliability without excessive energy use.
Solution Approach 2:
The patent changes the computation parameter dynamically based on frame type: using high-accuracy full CNN computation parameters for key frames and low-power approximate computation parameters for non-key frames. This parameter switching allows the system to adapt its processing reliability and energy consumption characteristics to the specific requirements of each frame type.
Data Source
AI summary
A system for processing spatial data may be designed to receive neural network outputs corresponding to a first spatial data set, and translate the neural network outputs corresponding to the first spatial data set based on the motion between a second spatial data set and the first spatial data set. The system may perform zero-gap run length encoding on the neural network outputs to store the neural network outputs in memory. The system may also perform on-the-fly skip zero decoding and bilinear interpolation to translate the neural network outputs.