Skip Convolutions for Video Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Video processing using artificial neural networks is computationally intensive, leading to significant time and energy consumption, especially on mobile devices with limited computational power.
Innovation Solution
The implementation of skip convolutions in video processing systems, which adaptively adjust computations based on the similarity between consecutive frames, skipping redundant regions and applying convolutional kernels only to salient areas.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If standard convolutional neural networks are used for video processing, then processing accuracy is maintained, but computational complexity and energy consumption increase significantly
Solution Approach 1:
The patent segments the video processing task by dividing it into two parts: (1) processing only the residual frame containing changed regions, and (2) copying unchanged regions from the previous frame. This segmentation allows the system to maintain processing accuracy for changed regions while avoiding redundant computations for unchanged regions, thereby reducing computational complexity without sacrificing overall processing accuracy.
Solution Approach 2:
The patent extracts and processes only the essential information - the residual frame containing changes between consecutive video frames. By taking out only the relevant changed regions and processing them through the convolutional neural network, while copying unchanged regions directly, the system maintains processing accuracy for important changes while significantly reducing the computational load compared to processing entire frames.
2Reliability
If standard convolutional neural networks are used for video processing, then processing completeness is maintained, but energy consumption increases
Solution Approach 1:
The patent segments video processing to apply computationally intensive convolution operations only to residual frames containing changed regions, while copying unchanged regions from previous frames. This segmentation maintains processing completeness for changed regions while dramatically reducing energy consumption by avoiding redundant processing of static regions across consecutive frames.
Solution Approach 2:
The patent applies partial action by processing only the necessary portion of video data - specifically, only the residual frames containing changes are processed through the full convolutional neural network pipeline. Unchanged regions are copied without processing, representing a partial application of the full processing pipeline that maintains completeness for important regions while reducing overall energy consumption.
3Measurement precision
If convolution operations are applied to all video frames, then processing accuracy is maintained, but processing time increases
Solution Approach 1:
The patent segments the video processing workflow into residual frame processing and frame copying operations. By segmenting the task to apply convolution operations only to residual frames (which contain changes) and copying unchanged frames directly, the system maintains processing accuracy for dynamic regions while significantly reducing processing time by avoiding redundant convolution operations on static regions.
Solution Approach 2:
The patent implements skipping by bypassing the computationally intensive convolution operations for unchanged video frames and instead copying them directly from the previous frame. This skipping strategy maintains processing accuracy for changed regions (which undergo full processing) while dramatically reducing processing time by rushing through unchanged regions with minimal operations.
Data Source
AI summary
A method for video processing via an artificial neural network includes receiving a video stream as an input at the artificial neural network. A residual is computed based on a difference between a first feature of a current frame of the video stream and a second feature of a previous frame of the video stream. One or more portions of the current frame of the video stream are processed based on the residual. Additionally, processing is skipped for one or more portions of the current frame of the video based on the residual.