Video CNN Update Blocks for Frame Difference Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for reducing the computational load in video processing using convolutional neural networks (CNN) are complex and inefficient, particularly when dealing with rapid frame changes in videos.
Innovation Solution
An image processing apparatus and method that determines difference areas between frames and adjusts convolution processing blocks accordingly, using a simple configuration to minimize calculations by selectively processing only relevant areas.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If CNN inference processing is performed on each frame of video, then processing accuracy is maintained, but the amount of calculation increases proportionally with the number of frames
Solution Approach 1:
The patent divides the video processing into key frames and non-key frames. Key frames undergo full CNN inference processing while non-key frames use differential processing. This segmentation allows the system to maintain processing accuracy on important frames while reducing calculation load on redundant frames, directly resolving the contradiction between accuracy and computational complexity.
Solution Approach 2:
The patent applies different processing qualities to different parts of the video stream. Key frames receive high-quality full CNN processing, while non-key frames receive lower-quality differential processing. This local differentiation in processing quality allows the system to optimize the balance between overall accuracy and total computational load by concentrating resources where they are most needed.
2Device complexity
If inference processing is limited to key frames only, then calculation amount is reduced, but processing accuracy deteriorates for rapid frame changes
Solution Approach 1:
The patent performs preliminary differential processing on non-key frames to identify change areas before deciding whether full CNN processing is needed. By预先 analyzing frame differences and locating areas of change, the system can determine which non-key frames require full processing to maintain accuracy, preventing accuracy deterioration while still reducing overall calculation compared to processing all frames fully.
Solution Approach 2:
The patent uses differential processing results as feedback to guide subsequent full CNN processing decisions. The change detection mechanism provides feedback about which non-key frames contain significant changes, allowing the system to selectively apply full processing only where necessary to maintain accuracy, thus resolving the contradiction between reduced calculation and maintained precision.
3Device complexity
If differential processing is performed for each partial area, then calculation amount is reduced, but it becomes difficult to perform inference across partial areas
Solution Approach 1:
The patent merges differential processing results with full CNN processing in a hybrid approach. Instead of choosing between differential or full processing, the system combines both methods: using differential processing to identify change areas and then applying full CNN processing to those specific areas. This merging allows the system to maintain cross-area inference capability while reducing overall calculation compared to processing all frames fully.
Solution Approach 2:
The patent adds a temporal dimension to the processing strategy by introducing the key frame concept and differential processing across time. This temporal dimension allows the system to process only changed areas in non-key frames while maintaining the ability to perform comprehensive inference by referencing previous key frames, thus preserving inference capability across partial areas while reducing computational load.
Data Source
AI summary
An image processing apparatus includes: an acquisition unit configured to acquire a moving image to be processed; a difference determination unit configured to determine a difference area from a past frame for frames other than a key frame among a plurality of frames; a block setting unit configured to set, for the frames other than the key frame among the plurality of frames, an update block including an update area according to the difference area among a plurality of blocks obtained by dividing an output feature map for each of a plurality of layers to be subjected to convolution processing of a neural network; and a processing unit configured to process, for the key frame among the plurality of frames, the key frame using the neural network on the update block, and store an output feature map of each layer, and perform, for the frames other than the key frame among the plurality of frames, processing using the neural network, and overwrite an output feature map stored for the update block, and the block setting unit sets the difference area for each layer to be subjected to the convolution processing so that the difference area is expanded to a surrounding area from a previous layer according to parameters of the convolution processing, and sets the update block including the update area according to the difference area.