Optical Flow Prediction Refinement for Decoder Motion Vector Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video coding systems, such as HEVC, face inefficiencies in handling bi-directional motion compensated prediction, especially under conditions of rapid illuminance changes, due to the simplistic combination of uni-prediction signals.
Innovation Solution
The implementation of advanced motion prediction techniques, including affine motion models with 4- and 6-parameters, and decoder-side motion vector refinement, along with prediction refinement using optical flow, to enhance the accuracy and adaptability of motion compensation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If advanced motion prediction techniques (affine motion models, decoder-side motion vector refinement, optical flow) are implemented, then motion compensation accuracy is improved, but device complexity increases
Solution Approach 1:
The motion compensation process is segmented into multiple stages: initial motion vector derivation, decoder-side motion vector refinement (DMVR), and prediction refinement with optical flow (PROF). Each stage processes motion data at different granularities and applies different levels of refinement, allowing the system to achieve high accuracy while managing complexity through progressive refinement rather than monolithic processing
Solution Approach 2:
Motion vectors are preliminarily derived using affine motion models before refinement stages. The DMVR process performs preliminary refinement by deriving refined motion vectors for sub-blocks before the final PROF stage. This preliminary action establishes a foundation that reduces the computational burden of subsequent refinement stages while maintaining accuracy
Solution Approach 3:
The system dynamically adapts motion compensation accuracy to scene characteristics. By using optical flow to calculate pixel-level motion offsets and applying them selectively based on gradient analysis, the system increases accuracy for regions with significant motion or illuminance changes while maintaining efficiency for static regions, creating a dynamic rather than static complexity-accuracy tradeoff
2Productivity
If decoder-side motion vector refinement is applied, then compression efficiency is improved, but processing time increases
Solution Approach 1:
The DMVR process applies motion vector refinement selectively rather than uniformly across all blocks. By focusing refinement on blocks where it provides the most benefit (those with significant motion or illuminance changes) and using early termination criteria, the system achieves partial action that improves compression efficiency without proportionally increasing processing time for all blocks
Solution Approach 2:
The system uses feedback mechanisms where refined motion vectors from DMVR are evaluated against prediction accuracy metrics. The PROF stage uses optical flow feedback to identify regions requiring additional refinement. This feedback-driven approach allows the system to allocate processing time adaptively, spending more time on blocks that benefit most from refinement while quickly processing blocks that achieve sufficient accuracy earlier
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
These techniques improve the accuracy of motion compensation, leading to better compression efficiency and adaptability to varying illuminance conditions, thereby enhancing the overall performance of video coding systems.
Implementation Method 1
prediction refinement using optical flow
Implementation Method 2
affine motion models with 4- and 6-parameters
Data Source
AI summary
Methods, devices, apparatus, systems, architectures and interfaces to improve motion vector (MV) refinement based sub-block (SB) level motion compensated prediction are provided. A decoding method includes receiving a bitstream of encoded video data, the bitstream including at least one block of video data including a plurality of SBs; performing a MV derivation, including a decoder based MV (DMVR) process, for at least one SB in the block to generate a refined MV for each SB; performing SB based motion compensation on the at least one sub-block to generate a SB based prediction within each SB; obtaining a spatial gradient for the prediction within each SB; determining a MV offset for each pixel in each SB; obtaining an intensity change in each SB based on the spatial gradients and MV offsets via an optical flow equation; and refining the prediction within each SB based on the obtained intensity changes.