Mode-Aware CNN In-Loop Filtering for Video Compression Artifacts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video encoding and decoding technologies struggle to effectively reduce artifacts while maintaining high compression efficiency, as conventional in-loop filters do not adequately account for block boundary information and coding modes.
Innovation Solution
A mode-aware neural network is employed to filter reconstructed video images, utilizing block boundary information and additional data such as quantization parameters and prediction modes to enhance the filtering process, thereby improving artifact reduction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Object-affected harmful factors
If conventional in-loop filters are used for video encoding and decoding, then compression efficiency is maintained, but artifact reduction is insufficient because block boundary information and coding modes are not adequately considered
Solution Approach 1:
The patent segments the filtering process by introducing a mode-aware neural network that processes different block types (intra-blocks, inter-blocks, skip-blocks) separately with specialized filtering operations. Each block type receives tailored filtering based on its coding mode, allowing artifact reduction to be improved without uniformly increasing complexity across all blocks.
Solution Approach 2:
The patent applies local quality by adapting the filtering operation to each block's specific coding mode and boundary characteristics. The neural network analyzes local block boundary information and coding modes to determine appropriate filtering strength and type for each region, rather than applying a uniform filter across the entire image, thus improving artifact reduction where needed while maintaining efficiency elsewhere.
2Object-affected harmful factors
If block boundary information and coding modes are incorporated into the filtering process, then artifact reduction is improved, but computational complexity increases
Solution Approach 1:
The patent performs preliminary action by pre-processing the block boundary information and coding modes before the main filtering operation. The mode-aware neural network prepares filtered versions of these inputs in advance, organizing the information in a format that facilitates efficient processing during the actual filtering stage, thereby reducing the computational energy required during real-time encoding/decoding.
Solution Approach 2:
The patent introduces an intermediary mode-aware neural network that mediates between the raw block boundary information and the final filtering operation. This neural network acts as a bridge that processes and transforms the input information into a form that is more efficient for the subsequent filtering stage, reducing the overall computational energy required by the system.
3Manufacturing precision
If a mode-aware neural network is used to filter reconstructed video images, then video quality and compression efficiency are improved, but the complexity of the encoding and decoding apparatus increases
Solution Approach 1:
The patent achieves universality by designing a mode-aware neural network that handles multiple block types and coding modes within a single unified architecture. The neural network is trained to recognize and process different block characteristics (intra, inter, skip modes) and boundary conditions, allowing one system to perform multiple filtering functions that would otherwise require separate specialized filters, thus improving video quality without proportionally increasing apparatus complexity.
Data Source
Figure 1~2
Figure 3~5
Figure 6A~6C
AI summary
Deep learning may be used in video compression for in-loop filtering in order to reduce artifacts. To improve the performance of a convolutional neural network (CNN) used for filtering, information available from the encoder or decoder, in addition to the initial reconstructed image, can also be used as input to the convolutional neural network. In one embodiment, QP, block boundary information and prediction image can be used as additional channels of the input. The boundary information may help the CNN to understand where the blocking artifacts are, and thus, may improve the CNN since the network does not need to spending parameters looking for blocking artifacts. QP or prediction block also provide more information to the CNN. Such a convolutional neural network may replace all in-loop filters, or work together with other in-loop filters to more effectively remove compression artifacts.