CNN In-Loop Filter for Video Coding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current video coding technologies face challenges in ensuring the integrity of encoding and decoding operations for digital media, particularly in reducing distortion and optimizing filtering processes, which can lead to information loss and sub-optimal performance.
Innovation Solution
The implementation of a convolutional neural network (CNN) filter as an in-loop filter or post-processing method, which selectively applies filtering based on rules that control its interaction with traditional non-deep learning-based filters, and is specifically used for I slices and B slices with temporal layer id equal to 0, avoiding over-filtering and optimizing the receptive field.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional in-loop filters are applied to all reconstructed frames, then filtering coverage is maximized, but over-filtering occurs causing information loss and sub-optimal performance
Solution Approach 1:
The patent applies different filtering strategies to different regions of the video data based on slice type and temporal layer characteristics. Specifically, CNN filtering is selectively applied only to I slices and B slices with temporal layer id equal to 0, while other regions use traditional filtering or no filtering. This local differentiation prevents over-filtering in regions where it would be harmful while maintaining filtering benefits in regions where it is advantageous.
Solution Approach 2:
The filtering approach dynamically adapts to different video content characteristics by making filtering decisions based on slice type and temporal layer information. The system transitions from static uniform filtering to dynamic selective filtering, adjusting the filtering intensity and type according to the specific characteristics of each region, thereby optimizing the balance between filtering effectiveness and information preservation.
2Loss of information
If CNN filter is applied selectively based on slice type and temporal layer, then over-filtering is avoided, but filtering coverage is reduced
Solution Approach 1:
The patent changes the parameters controlling filter application from uniform application to selective application based on slice type (I slice, B slice) and temporal layer id. By modifying these control parameters, the system achieves optimal filtering coverage that prevents information loss while maintaining sufficient filtering effectiveness through targeted application to specific video regions.
3Manufacturing precision
If CNN filtering is applied to improve reconstruction quality, then bitrate reduction is achieved, but computational complexity increases
Solution Approach 1:
Instead of applying CNN filtering to all reconstructed frames (excessive action), the patent applies it only to specific I slices and B slices with temporal layer id equal to 0 (partial action). This partial application achieves the majority of the reconstruction quality improvement and bitrate reduction benefits while significantly reducing the computational complexity compared to universal application.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, apparatus for media processing are described. One example method of digital media processing includes performing a conversion between visual media data and a bitstream of the visual media data, wherein the performing of the conversion includes selectively applying a convolutional neural network filter during the conversion based on a rule, and wherein the rule specifies whether and/or how the convolutional neural network filter is applied.