Video Frame Synthesis With Masked CNNs for Motion-Focused Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video compression technologies, including hybrid codecs and neural network-based approaches, can be improved for more efficient encoding and decoding processes, particularly in terms of memory usage and processing speed, especially when dealing with video frames that have static backgrounds or minimal motion.
Innovation Solution
A convolutional neural network with mask units is employed to generate pyramids of features from sub-portions of video frames, allowing for dynamic/conditional convolution only on relevant areas, reducing the need for processing entire frames and optimizing memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If entire video frames are processed using convolutional neural networks, then comprehensive feature extraction is achieved, but processing time and memory requirements increase significantly
Solution Approach 1:
The patent divides the video frame into multiple patches or regions, processing only the relevant portions through the convolutional neural network while skipping static or redundant areas. This segmentation approach maintains feature extraction completeness for dynamic regions while significantly reducing overall processing time and computational resources required.
Solution Approach 2:
The patent applies different processing strategies to different regions of the video frame based on their content characteristics. Dynamic regions with important motion information are processed in detail, while static background regions are either skipped or processed with reduced computational complexity, achieving local optimization of processing quality.
2Measurement precision
If convolutional operations are applied to all regions of video frames, then complete motion analysis is achieved, but memory consumption increases
Solution Approach 1:
The patent extracts and processes only the essential dynamic regions containing motion information, while discarding or skipping processing of static background regions. This extraction approach maintains complete motion analysis for relevant areas while significantly reducing memory consumption by avoiding unnecessary processing of entire frames.
Solution Approach 2:
The patent applies partial convolutional operations only to regions where motion detection is necessary, rather than applying operations uniformly across the entire frame. This partial action approach achieves sufficient motion analysis for critical regions while reducing overall memory requirements.
3Productivity
If standard hybrid codecs are used for video compression, then established compression efficiency is achieved, but adaptability to neural network-based optimizations is limited
Solution Approach 1:
The patent creates a dynamic video coding system that can adaptively switch between traditional hybrid coding methods and neural network-based processing based on the specific requirements of each video sequence. The system dynamically adjusts the degree of neural network involvement, allowing for flexible integration of AI optimizations while maintaining compatibility with established compression standards.
Solution Approach 2:
The patent designs a universal video processing framework that can handle both traditional compression tasks and neural network-based optimizations within a single system architecture. This multi-functional approach enables the same system to perform standard hybrid coding and advanced AI-enhanced compression, improving adaptability to different optimization needs.
Data Source
AI summary
It is provided a method of video frame synthesis by means of a convolutional neural network comprising an encoder comprising at least one first mask unit and a decoder comprising at least one second mask unit. The method includes: generating by the at least one first mask unit first data corresponding to only first sub-portions of a first video frame taken at a first time instance and second data corresponding to only second sub-portions of a second video frame taken at a second time instance, generating by the encoder a first pyramid of features based on the first data and a second pyramid of features based on the second data and generating by the decoder a synthesized third video frame for a third time instance between the first and second time instances based on the generated pyramids of features and by means of the at least one second mask unit.