Deep Neural Network Motion Field Refinement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current video encoding and decoding methods face challenges in achieving high compression efficiency due to limitations in motion field refinement, particularly in exploiting complex motion patterns and residual errors between prediction blocks.
Innovation Solution
The use of a deep neural network (DNN) to obtain and refine motion fields between motion-compensated prediction blocks, enabling pixel-based motion field refinement at both the encoder and decoder sides, thereby improving prediction accuracy and compression efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional motion field refinement methods are used, then device complexity is reduced, but prediction accuracy and compression efficiency deteriorate
Solution Approach 1:
The patent replaces traditional mechanical motion estimation algorithms with a neural network-based system. The neural network learns motion patterns from training data and directly predicts motion vectors, substituting the conventional block-matching and optical flow algorithms with a data-driven approach that achieves higher prediction accuracy while maintaining computational efficiency through parallel processing capabilities.
Solution Approach 2:
The patent changes the parameters used in motion estimation by incorporating additional features such as gradient information, multiple reference picture lists, and refined motion vector candidates. The neural network processes these enhanced parameters to generate more accurate motion fields, improving prediction precision without proportionally increasing device complexity through efficient parameter utilization.
2Productivity
If deep neural network is used for motion field refinement, then compression efficiency is improved, but computational complexity increases
Solution Approach 1:
The patent segments the motion estimation process into multiple stages: initial motion vector generation, neural network-based refinement, and final prediction block generation. By dividing the computational task into discrete segments, the system achieves high compression efficiency through accurate motion compensation while managing computational complexity through staged processing and selective application of the neural network.
Solution Approach 2:
The patent performs preliminary motion estimation using traditional methods to generate initial motion vectors and prediction blocks before applying the neural network for refinement. This preliminary action reduces the computational burden on the neural network by providing it with pre-processed data, thereby improving compression efficiency while controlling overall computational complexity through hierarchical processing.
3Measurement precision
If pixel-based motion field refinement is implemented, then prediction accuracy is improved, but processing time increases
Solution Approach 1:
The patent applies pixel-based motion field refinement selectively to regions where it provides the most benefit, such as areas with complex motion patterns or high prediction errors. By applying the computationally intensive neural network refinement locally rather than uniformly across all blocks, the system improves prediction accuracy in critical regions while minimizing the overall processing time through selective processing.
Data Source
AI summary
For a bi-prediction block, the initial motion field can be refined using a DNN. In one implementation, the initial motion field is integer rounded to obtain initial prediction blocks. Based on the initial prediction, the DNN can generate motion refinement information, which is scaled and added to the sub-pel residual motion from the initial motion field to generate a refined motion field. The scaling factor can take a default value, or be based on the motion asymmetry. While the initial motion field is usually block based on sub-block based, the refined motion field is pixel based or sub-block based and can be at an arbitrary accuracy. The same refinement process is performed at both the encoder and decoder, and thus the motion refinement information need not to be signaled. Whether the refinement is enabled can be determined based on the initial motion, the block activities and the block size.