NN In-Loop Filter Using Transform-Domain Features to Cut Decoding Complexity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video decoding technologies face challenges in reducing computational complexity and memory bandwidth requirements while maintaining the coding gains achieved by neural network-based filtering techniques.
Innovation Solution
The proposed techniques process a first input channel in a transform domain and a second input channel in a non-transform domain, reducing complexity while maintaining coding gains through convolutional neural network-based filtering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If neural network-based filtering is applied to improve decoded video quality, then video quality is improved, but computational complexity increases
Solution Approach 1:
The patent segments the input data into two distinct channels: transform domain data (quantized transform coefficients) and non-transform domain data (context information, motion vectors, prediction modes). This segmentation allows the neural network to process different types of information separately and combine them, reducing the overall computational complexity while maintaining decoding quality.
Solution Approach 2:
The patent extracts and utilizes transform domain data (quantized transform coefficients) as an additional input channel to the neural network. By taking out this previously unused information and feeding it directly to the network, the system achieves better reconstruction quality without requiring full inverse transformation, thereby reducing computational complexity.
2Manufacturing precision
If neural network-based filtering is applied to improve decoded video quality, then video quality is improved, but memory bandwidth requirements increase
Solution Approach 1:
The patent segments the data flow into separate channels that can be processed independently with different memory access patterns. Transform domain data requires minimal memory access since it's already available from the decoding pipeline, while context data is processed separately. This segmentation reduces peak memory bandwidth requirements compared to processing all data uniformly.
Solution Approach 2:
The patent extracts transform domain coefficients and feeds them directly to the neural network without requiring full reconstruction of the video block first. This extraction approach avoids the memory-intensive inverse transform and reconstruction steps, significantly reducing memory bandwidth requirements while maintaining quality.
3Device complexity
If transform domain processing is applied to reduce computational complexity, then computational complexity is reduced, but filtering precision may be compromised
Solution Approach 1:
The patent merges transform domain data and non-transform domain data within the neural network framework. By combining these different types of information in the feature extraction layers, the network achieves accurate filtering decisions without requiring full inverse transformation, thus maintaining precision while reducing computational complexity.
Solution Approach 2:
The patent replaces the traditional mechanical approach of inverse transforming coefficients to spatial domain before filtering with a neural network-based system that operates directly on transform coefficients. This substitution enables filtering in the transform domain, reducing computational complexity while maintaining or even improving filtering precision through learned patterns.
Data Source
AI summary
A device for decoding encoded video data is configured to determine, from the encoded video data, a block of a picture; apply a neural network (NN)-based filter process to the block to generate a filtered block, wherein to apply the NN-based filter process, the processing circuitry is configured to process a first input channel comprising sample data in a transform domain and process a second input channel comprising context data in a non-transform domain; determine a decoded version of the block based on the filtered block; and output a decoded version of the picture comprising the decoded version of the block.