Convolutional In-Loop Video Filtering for Lower Codec Complexity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The high complexity of neural network-based in-loop filters in video codecs, such as those described in JVET-Y0143, poses a challenge for practical hardware implementations, necessitating a reduction in complexity while maintaining or improving performance.
Innovation Solution
A method involving a machine learning model that processes input data such as reconstructed and predicted sample values, block boundary strength, and quantization parameters, without requiring partition information, to generate encoded or decoded video data, utilizing a series of attention residual blocks and convolutional layers to optimize filter complexity and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If neural network-based in-loop filters are used in video codecs, then filtering performance and compression efficiency are improved, but model complexity and computational requirements increase significantly
Solution Approach 1:
The patent extracts and removes partition information from the neural network input, keeping only the most essential features (reconstructed samples, predicted samples, boundary strength, quantization parameters). This extraction principle reduces input data complexity while preserving the filtering performance by focusing on the most relevant features for boundary artifact reduction.
Solution Approach 2:
The patent applies different filtering strengths based on local boundary characteristics by utilizing block boundary strength information. The neural network processes different regions with varying complexity - applying stronger filtering where boundaries are detected and maintaining simpler processing in uniform regions, thereby optimizing the trade-off between filtering performance and computational complexity locally.
2Productivity
If complex neural network models are used for video filtering, then compression efficiency improves, but hardware implementation becomes more difficult
Solution Approach 1:
The patent extracts only the most critical input features needed for effective filtering, removing unnecessary partition information. This reduction in input complexity makes the neural network more suitable for hardware implementation while maintaining compression efficiency by focusing computational resources on the most important filtering decisions.
Solution Approach 2:
Instead of reducing filtering performance to simplify hardware implementation, the patent inverts the approach by maintaining high filtering performance through selective feature extraction and letting the simplified input structure enable easier hardware implementation. The inversion lies in achieving simplicity through intelligent selection rather than through reduction of functionality.
3Measurement precision
If more input information is provided to the neural network, then filtering accuracy improves, but computational complexity increases
Solution Approach 1:
The patent extracts and retains only the four most essential input features: reconstructed samples, predicted samples, block boundary strength, and quantization parameters. By removing redundant partition information and focusing on these critical features, the patent achieves high filtering accuracy through quality over quantity of input data, reducing computational complexity while maintaining precision.
Solution Approach 2:
The patent changes the parameter set by selecting specific features (boundary strength, quantization parameters) that have the highest impact on filtering accuracy. This parameter selection strategy ensures that the neural network receives the most relevant information for accurate filtering without the computational burden of processing all possible input parameters.
Data Source
AI summary
There is provided a method of generating encoded video data or decoded video data. The method comprises providing input data to a first convolution layer, CL, thereby generating first convoluted data, generating residual data based on the first convoluted data, and generating the encoded video data or the decoded video data based on a combination of the input data and the residual data.