Residual Transformer In-Loop Filter for Video Compression Artifacts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video compression methods like HEVC and VVC suffer from irreversible information loss and compression artifacts such as blocking, blurring, and banding, especially at high compression ratios, requiring significant computing resources for inefficient artifact reduction.
Innovation Solution
A neural network-based in-loop filter using a residual transformer neural network (RTNN) with feature extraction, backbone, and reconstruction modules to enhance image quality by incorporating quantization parameter maps and employing multi-stage progressive training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional video compression methods (HEVC, VVC) are used with high compression ratios, then compression efficiency is improved, but compression artifacts (blocking, blurring, banding) increase significantly
Solution Approach 1:
The patent applies this principle by using the quantization parameter map (which indicates regions of high compression and potential artifacts) as an input feature to guide the neural network's filtering process. The harmful compression artifacts are converted into useful information that helps the network identify and correct problematic regions.
Solution Approach 2:
The patent changes the parameter representation by introducing a quantization parameter map that spatially varies compression information across different regions of the video frame. This allows the filtering process to adapt locally rather than applying uniform compression, thereby reducing artifacts while maintaining overall compression efficiency.
2Object-affected harmful factors
If traditional filtering methods are applied to reduce compression artifacts, then artifact reduction is achieved, but computing resources and processing complexity increase significantly
Solution Approach 1:
The patent segments the filtering process into distinct functional modules: a feature extraction module that processes multiple inputs (reconstructed picture, prediction picture, partition picture, QP map), a backbone module with residual blocks and transformer blocks for feature processing, and a reconstruction module. This modular segmentation allows efficient computation by distributing processing tasks across specialized sub-units.
Solution Approach 2:
The patent introduces an intermediary quantization parameter map that mediates between the compression process and the filtering process. This intermediary provides spatially-varying guidance information to the neural network, enabling more efficient artifact reduction without requiring the network to process all raw pixel data equally.
3Ease of operation
If uniform filtering is applied across the entire video frame, then processing simplicity is maintained, but regional variations in artifact types and severity cannot be addressed
Solution Approach 1:
The patent applies local quality by using the quantization parameter map to provide region-specific filtering guidance. Different regions of the video frame receive tailored filtering based on their local compression characteristics, allowing the system to address regional variations in artifact types and severity while maintaining a unified neural network architecture.
Data Source
AI summary
Methods and systems for video processing are provided. The method includes that: (i) a video sequence is received by a neural network (NN) based in-loop filter, the NN based in-loop filter includes a residual transformer NN (RTNN) filter having a feature extraction module, a backbone module, and a reconstruction module; (ii) features are extracted by the feature extraction module from input information, the input information includes a quantization parameter (QP) map, a reconstruction picture, a prediction picture and a partition picture; (iii) a feature map is generated by the backbone module based an output from the feature extraction module, the backbone module includes multiple residual blocks and a transformer block (TB); and (iv) a dimension-reduced feature map is generated by the reconstruction module based on the feature map via a convolution process.