Multi-Model Neural Filtering for Rate-Distortion Video Coding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video coding techniques using neural network-based filters are not optimal for specific video sequences, leading to suboptimal bitrate and distortion performance.
Innovation Solution
Implementing multi-model neural network-based filtering techniques that select the most suitable neural network model for a portion of the decoded picture based on rate distortion optimization or quantization parameters, allowing for improved filtering of decoded video data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single neural network model is used for filtering decoded video data, then the filtering process is simple and fast, but the bitrate and distortion performance is suboptimal for specific video sequences
Solution Approach 1:
The patent implements dynamic model selection where the system transitions from a static single-model approach to a dynamic multi-model selection approach. The video encoder selects different neural network models from a set of pre-trained models based on rate distortion optimization or quantization parameter mapping, allowing the filtering process to adapt to different video sequences and conditions while maintaining manageable complexity through pre-computed model sets
Solution Approach 2:
The patent changes the parameter of model selection by introducing multiple pre-trained neural network models with different characteristics. Instead of using a fixed model, the system varies the selected model based on video sequence properties, QP values, or RDO results, thereby improving performance across diverse video content without requiring a single overly complex model
2Reliability
If multiple neural network models are selected and signaled in the bitstream, then better bitrate and distortion results are achieved, but the signaling overhead and processing complexity increases
Solution Approach 1:
The patent extracts only the essential model selection information into the bitstream rather than transmitting full model data. By using syntax elements to signal indices into pre-defined model sets, the system transmits minimal overhead information that enables the decoder to select the same model as the encoder, significantly reducing bitstream overhead while maintaining coding efficiency
Solution Approach 2:
The patent creates a universal set of pre-trained neural network models that can serve multiple video sequences and conditions. These models are designed to handle various video characteristics, allowing a single model set to provide good performance across diverse content types, thereby reducing the need to transmit extensive model-specific information in the bitstream
Data Source
AI summary
An example device for filtering decoded video data includes a memory configured to store video data; and one or more processors implemented in circuitry and configured to: decode a picture of video data; code a value for a syntax element representing a neural network model to be used to filter a portion of the decoded picture, the value representing an index into a set of pre-defined neural network models, the index corresponding to the neural network model in the set of pre-defined neural network models; and filter the portion of the decoded picture using the neural network model corresponding to the index.