ML Video Compression With Multi-Hypothesis Motion Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing end-to-end learned video compression methods suffer from inaccurate motion estimation and limited temporal alignment due to reliance on a single motion hypothesis, particularly in complex scenes with complex movements, leading to inefficiencies in removing temporal redundancies.
Innovation Solution
A multiple hypotheses based motion compensation framework that generates multiple motion vectors and warped features using a neural network, incorporating an attention mechanism to weight the contributions of different hypotheses, and a context combination module to fuse these features for improved temporal alignment and compression efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single motion hypothesis is used for temporal alignment, then the method complexity is reduced, but the alignment accuracy deteriorates in complex scenes
Solution Approach 1:
The patent segments the motion estimation process into multiple independent hypotheses (first motion hypothesis, second motion hypothesis, etc.), where each hypothesis provides a different motion compensation approach. This segmentation allows the system to explore multiple motion possibilities simultaneously without exponentially increasing complexity, as each hypothesis can be processed independently and then combined.
Solution Approach 2:
The patent implements dynamic motion hypothesis selection through an attention mechanism that adaptively weights different motion hypotheses based on the specific scene content. The system dynamically adjusts which hypotheses contribute more to the final result, allowing flexible adaptation to complex motions while maintaining computational efficiency through selective processing.
2Measurement precision
If multiple motion hypotheses are generated and processed, then the temporal alignment accuracy is improved, but the computational complexity increases
Solution Approach 1:
The patent merges multiple motion hypotheses into a unified motion compensation result through an attention mechanism that combines the outputs of different hypotheses. Instead of processing each hypothesis completely independently to completion, the system merges their contributions in a weighted manner, reducing redundant computations while maintaining the benefits of multiple hypotheses.
Solution Approach 2:
The attention mechanism serves as an intermediary that mediates between multiple motion hypotheses and the final motion compensation result. It selectively weights and combines the hypotheses without requiring full processing of each hypothesis to the same level of detail, thus reducing overall computational complexity while maintaining accuracy.
3Loss of information
If optical flow based motion estimation is used, then the temporal information mining is improved, but the bit rate for motion data increases
Solution Approach 1:
The patent extracts only the essential motion information from multiple hypotheses rather than transmitting complete optical flow fields. The attention mechanism identifies and extracts the most relevant motion components, discarding redundant information, thus maintaining temporal information quality while reducing the quantity of motion data that needs to be encoded and transmitted.
Solution Approach 2:
The patent changes the representation parameters of motion data by using attention weights to selectively emphasize important motion hypotheses and suppress less relevant ones. This parameter transformation allows the system to convey the same temporal information with fewer bits by focusing on the most significant motion components rather than transmitting all motion details equally.
Data Source
AI summary
A computer-implemented method for facilitating machine-learning based media (e.g., video) compression. The method includes receiving a motion data set associated with motion-related difference between a first image and a second image, and processing the motion data set using a neural network to determine a plurality of motion data subsets. The method also includes processing the plurality of motion data subsets using one or more features associated with the first image to obtain a plurality of motion-warped feature data sets each associated with a respective motion data subset; and processing the plurality of motion-warped feature data sets to facilitate generation of context data for facilitating conditional coding based compression of the second image.