Masked Neural Loop Filter Sharing for Multi-QP Video Enhancement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video coding standards face challenges in accommodating multiple Quantization Parameter (QP) values, leading to increased storage and computation costs due to the need for multiple neural network model instances for each QP value, and existing NN-based quality enhancement methods struggle to adapt to varying QP settings.
Innovation Solution
A Multi-Quality Loop Filter (MQLF) mechanism using a Multi-Task Neural Network (MTNN) with micro-structured parameter sharing, where a single MTNN model instance accommodates multiple QP settings, guided by a binary Micro-Structured Mask (MSM) for each QP value, to enhance video quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If one neural network model instance is trained and deployed for each QP value, then the video quality enhancement is optimized for each specific QP setting, but the storage and computation costs increase significantly
Solution Approach 1:
The patent combines multiple QP-specific neural network models into a single unified model instance that processes multiple QP values. The model accepts QP value as an additional input channel and uses it to adapt its processing to different quantization settings, thereby reducing the number of model instances from multiple (one per QP) to a single shared model that handles all QP values.
Solution Approach 2:
The neural network model is designed to be universal across different QP settings by incorporating the QP value as an input feature. This allows a single model instance to perform quality enhancement for multiple QP values (0-63), making the model multi-functional rather than dedicated to a single QP setting, thus reducing storage and computation overhead.
2Manufacturing precision
If multiple neural network model instances are deployed for different input channels with different QP values, then each channel is optimized for its specific QP setting, but the device complexity and storage requirements increase due to combinatorial number of models
Solution Approach 1:
The patent merges multiple channel-specific QP-optimized models into a single unified model that processes all input channels and QP combinations. By feeding the QP value as an additional input channel, the model learns to adapt its processing to different QP settings for different channels (luma and chroma) without requiring separate model instances for each combination.
Solution Approach 2:
The model is designed to be universally applicable across different input channels (luma, chroma) and QP values by incorporating channel identity and QP value as input features. This universal design eliminates the need for managing combinatorial numbers of model instances while maintaining channel-specific optimization through learned feature interactions.
3Quantity of substance
If QP values are treated as additional input to a single neural network model, then storage costs are reduced, but the model must learn to organize parameters to implicitly model distribution of sub-groups of data samples
Solution Approach 1:
The patent changes the parameter representation by incorporating QP values as additional input channels rather than as separate model instances. This parameter change allows the model to dynamically adapt its internal parameter organization based on the input QP value, learning to implicitly model the distribution of sub-groups of data samples corresponding to different QP settings through the shared parameter space.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Video processing with a multi-quality loop filter using a multi-task neural network is performed by at least one processor and includes generating model IDs, based on quantization parameters in an input, selecting a first set of masks, each mask in the first set of masks corresponding to one of the generated model IDs, performing convolution of first weights of a first set of neural network layers and the selected first set of masks to obtain first masked weights, and selecting a second set of neural network layers and second weights, based on the quantization parameters, generating a quantization parameter value, based on the quantization parameters, and computing an inference output, based on the first masked weights and the second weights, using the generated quantization parameter value.