U-Net Transformer In-Loop Filter Video Coding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network-based in-loop filtering architectures for video coding, such as the JVET HOP architecture, suffer from high computation complexity, limited receptive fields, and static weights that cannot adapt to input content.
Innovation Solution
The proposed solution combines a U-Net architecture with self-attention mechanisms, specifically using Transformers, to enhance the neural network-based in-loop filtering. This approach reduces computation complexity and memory bandwidth requirements while improving performance compared to existing architectures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a large number of sequential residual blocks are used in the neural network architecture, then the filtering performance is improved, but the processing delay increases
Solution Approach 1:
The patent divides the neural network into two distinct parts: a shallow U-Net encoder with residual blocks for feature extraction, and a separate transformer decoder for refinement. This segmentation allows the heavy computational tasks to be distributed across different architectural components with different depth characteristics, reducing overall processing delay while maintaining filtering performance.
Solution Approach 2:
The patent transitions from a purely sequential deep residual block architecture to a hybrid architecture that incorporates transformer mechanisms. The transformer component introduces a different computational dimension based on self-attention mechanisms, allowing the model to capture long-range dependencies without increasing the sequential depth, thus reducing processing delay while maintaining or improving filtering performance.
2Device complexity
If two-dimensional convolutions are used in the neural network, then the computation is simplified, but the receptive field is limited and long-range dependencies cannot be modeled
Solution Approach 1:
The patent merges two different architectural paradigms: the U-Net encoder with 2D convolutions for local feature extraction and the transformer decoder with self-attention mechanisms for global dependency modeling. This combination allows the system to benefit from both the computational simplicity of convolutions for local processing and the long-range dependency modeling capability of transformers, effectively resolving the contradiction between computation complexity and long-range dependency modeling.
Solution Approach 2:
The patent introduces the transformer component that operates in a different computational dimension compared to traditional 2D convolutions. The self-attention mechanism in the transformer allows for modeling long-range dependencies across the entire image or video block, providing a new dimension of computational capability that complements the local feature extraction from convolutions, thereby enabling effective long-range dependency modeling without excessive computational complexity.
3Productivity
If static weights are used in the neural network, then the inference is faster, but the filter cannot adapt to input content
Solution Approach 1:
The patent incorporates dynamic elements through the transformer component with self-attention mechanisms that can adaptively weigh different regions and features based on the input content. While the U-Net encoder uses static convolutional weights for efficient local feature extraction, the transformer decoder introduces dynamic adaptability through its self-attention mechanism, allowing the filter to adjust its processing based on the specific characteristics of the input video or image data, thus resolving the contradiction between inference speed and adaptability.
Data Source
AI summary
Methods and devices for decoding video data are described. An example method includes in-loop filtering a current block of the video data using a neural network-based in-loop filter to generate an in-loop filtered current block, wherein the neural network-based in-loop filter is trained using an architecture comprising a U-Net architecture comprising one or more residual blocks and one or more transform blocks; and outputting the in-loop filtered current block.