Machine Learning Video Codec Filtering Without Partition Inputs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The high complexity of neural network-based in-loop filters in video codecs, such as those described in JVET-Y0143, poses a challenge for practical hardware implementations, necessitating a reduction in complexity while maintaining or improving performance.
Innovation Solution
A method involving a machine learning model that processes reconstructed and predicted sample values, along with quantization parameters and block boundary strength information, to generate encoded or decoded videos, without requiring partition information about coding tree units, thereby reducing complexity and improving efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If neural network-based in-loop filters are used in video codecs, then compression efficiency is improved, but device complexity increases
Solution Approach 1:
The patent extracts and removes the partition information input from the neural network model inputs. By taking out this complex input requirement, the model can be implemented with simpler hardware structures while maintaining the core filtering functionality through remaining inputs like reconstructed samples, predicted samples, and block boundary strength information.
Solution Approach 2:
The patent changes the parameter set of the neural network model by replacing partition information with alternative parameters such as block boundary strength information and quantization parameters. This parameter substitution maintains the model's ability to process video data while reducing hardware complexity requirements.
2Reliability
If partition information is included in neural network inputs, then filtering performance is improved, but processing complexity increases
Solution Approach 1:
The patent extracts the partition information from the input list of the neural network model. This removal simplifies the processing requirements while the model maintains effective filtering performance through the remaining inputs that capture essential spatial and boundary information without requiring detailed partition data.
Solution Approach 2:
The patent uses alternative inputs such as block boundary strength information and quantization parameters as substitutes for partition information. These copying alternatives provide sufficient contextual data for the model to perform filtering without needing the original partition information, thus reducing processing complexity.
Data Source
AI summary
A method for generating an encoded video or a decoded video is provided. The method comprises obtaining values of reconstructed samples, and obtaining input information comprising any one or a combination of: i) information about filtered samples, ii) information about predicted samples, or iii) information about skipped samples. The method further comprises providing the values of reconstructed samples and the input information to a machine learning, ML, model, thereby generating at least one ML output data, and based at least on said at least one ML output data, generating the encoded video or the decoded video.