Deformable Convolutional DNN for Video Frame Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional video coding methods, such as H.264/AVC, HEVC, and VVC, face inefficiencies in handling complex and long-range motions due to limitations in block-based motion estimation and frame interpolation, which affect compression efficiency and visual quality.
Innovation Solution
The use of a deformable convolutional deep neural network (DNN) for temporal deformable convolution, which generates predicted frames by aligning and synthesizing features from previously reconstructed reference frames, allowing for flexible handling of complex motions without explicit motion estimation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If traditional block-based motion estimation is used, then device complexity is reduced, but manufacturing precision (prediction accuracy) deteriorates due to inability to handle complex and long-range motions
Solution Approach 1:
The patent replaces traditional mechanical block-based motion estimation algorithms with a deep neural network-based deformable convolution system. The DNN automatically learns motion patterns and performs pixel-level alignment, substituting the manual block-matching mechanism with an intelligent system that achieves superior prediction accuracy for complex motions while maintaining computational efficiency through parallel processing.
2Reliability
If explicit motion estimation and image warping are used, then prediction accuracy improves for simple motions, but reliability deteriorates for complex and long-range motions due to error accumulation
Solution Approach 1:
The patent introduces offset maps as an intermediary mechanism that directly guides the deformable convolution process. Instead of relying on multi-step motion estimation and warping that accumulate errors, the offset maps provide direct pixel displacement information learned by the DNN, enabling accurate alignment of reference frames even for complex and long-range motions without error propagation.
3Manufacturing precision
If traditional frame interpolation is used, then processing speed is maintained, but manufacturing precision deteriorates due to inability to handle complex motions
Solution Approach 1:
The patent employs a dynamic deformable convolution approach where the convolution kernels adapt their shapes and positions based on learned offset maps. This dynamic mechanism allows the system to flexibly handle varying motion patterns including complex and long-range motions, achieving high prediction quality while maintaining processing efficiency through the differentiable nature of the operation that enables end-to-end optimization.
Data Source
AI summary
A method of video coding at a video coding device includes performing a deformable convolution through a deformable convolutional deep neural network (DNN) to generate one or more first feature maps based on a set of one or more previously reconstructed reference frames, generating a predicted frame based on the one or more first feature maps, and reconstructing a current frame based on the predicted frame. In an embodiment, a set of one or more second feature maps corresponding to the one or more previously reconstructed reference frames can be generated based on a feature extraction DNN. One or more offset maps corresponding to the one or more second feature maps can be generated, respectively, using an offset generation DNN.