Neural-Network Post-Filter Control for Multi-Picture Video Coding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video coding standards, such as VVC, do not effectively utilize neural-network post-processing filters (NNPFs) for optimizing video quality, leading to inefficiencies in bandwidth usage and video processing.
Innovation Solution
The implementation of neural-network post-processing filters (NNPFs) is signaled and specified through SEI messages, allowing for the use of neural networks to enhance video quality by adjusting parameters like resolution, bit depth, and color format, with support for multiple input pictures and interpolation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If neural-network post-processing filters are implemented for video processing, then video quality is improved, but device complexity increases
Solution Approach 1:
The video processing system is segmented into distinct functional modules: neural-network post-processing filter (NNPF) for quality enhancement, conversion between visual media data and bitstream for encoding/decoding, and SEI message handling for control signaling. This modular architecture allows the NNPF to operate as an independent quality enhancement component without entangling the entire processing system, thereby managing complexity while improving video quality.
Solution Approach 2:
SEI messages serve as intermediaries that carry control information between different processing stages. These messages enable the NNPF to be activated or deactivated without requiring complex direct control wiring, simplifying the interaction between the quality enhancement module and the rest of the video processing system.
2Manufacturing precision
If multiple input pictures are used for neural-network post-processing, then visual fidelity is improved, but processing time increases
Solution Approach 1:
The system determines in advance whether the required number of input pictures are available in the coded layer video sequence before initiating the neural-network post-processing filter operation. This preliminary check prevents unnecessary processing when sufficient input pictures are not present, reducing wasted processing time while maintaining the ability to process multiple pictures when available for improved visual fidelity.
Solution Approach 2:
The processing system dynamically adjusts its behavior based on picture availability. When multiple input pictures are present, the NNPF processes them to enhance visual fidelity. When pictures are insufficient, the system adapts by using available pictures or skipping the filter, thereby optimizing processing time according to actual data availability rather than following a fixed time-consuming process.
3Stability of the object's composition
If neural-network post-processing filter is activated for every picture, then video quality consistency is improved, but processing efficiency decreases
Solution Approach 1:
The neural-network post-processing filter operation is made dynamic through conditional activation based on picture availability and quality assessment. The system determines whether to apply the filter to each picture based on whether sufficient input pictures exist in the coded layer video sequence, allowing quality consistency to be maintained where possible while improving processing efficiency by skipping unnecessary filter applications.
Solution Approach 2:
The filter activation parameter is changed from a static 'always on' mode to a dynamic mode that varies based on input conditions. The system modifies the filter activation state according to the presence and quality of input pictures, thereby optimizing the balance between maintaining video quality consistency and improving overall processing efficiency.
Data Source
AI summary
A mechanism for processing media data is disclosed. The mechanism includes determining that an indicated number of input pictures for a neural-network post-filter (NNPF) are present in a coded layer video sequence (CLVS) and precede a current picture in output order when the NNPF is indicated to have multiple input pictures and activated for the current picture. A conversion is performed between a visual media data and a bitstream based on the NNPF.