NN-Based In-Loop Filters With Alternating Separable Convolutions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
NN-based filtering techniques in video coding can prioritize vertical or horizontal features, leading to visual distortion and reduced quality in decoded video, while also increasing computational complexity and signaling overhead.
Innovation Solution
Implement a neural network-based filter process with a first backbone block process comprising a M×N convolution followed by a N×M convolution, and a second backbone block process comprising a second N×M convolution followed by a second M×N convolution, where N and M are different integer values, to improve decoded video quality without additional signaling overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single directional convolution (M×N or N×M) is used in the backbone block, then computational complexity is reduced, but visual distortion occurs and decoded video quality deteriorates due to prioritization of one direction over the other
Solution Approach 1:
The single convolution operation is segmented into two sequential convolutions: first an M×N convolution to capture horizontal features, then an N×M convolution to capture vertical features. This segmentation allows the system to process both directional features separately rather than prioritizing one direction, thereby improving decoded video quality without significantly increasing overall computational complexity.
Solution Approach 2:
The patent applies asymmetric convolution kernels with different dimensions (M×N followed by N×M) in sequence. The first convolution uses an M×N kernel to extract horizontal features, while the second uses an N×M kernel to extract vertical features. This asymmetric approach ensures that both horizontal and vertical features are captured equally, preventing the visual distortion that occurs when only one direction is prioritized.
2Productivity
If simplified NN-based filtering is applied, then computational complexity is reduced, but coding quality deteriorates due to loss of directional feature balance
Solution Approach 1:
The filtering process is segmented into two distinct backbone block processes, each handling a different directional convolution. This segmentation maintains processing efficiency by using simple sequential operations while improving coding quality by ensuring both horizontal and vertical features are captured through the alternating M×N and N×M convolutions.
3Manufacturing precision
If dual directional convolutions (M×N followed by N×M) are implemented in both backbone blocks, then decoded video quality is improved and rate-distortion tradeoff is optimized, but the system requires increased computational resources
Solution Approach 1:
The patent implements asymmetric convolutions (M×N followed by N×M) in both backbone blocks to capture both horizontal and vertical features. This asymmetric approach improves decoded video quality by ensuring balanced directional feature extraction. The computational resource increase is managed by using separable convolutions that process dimensions independently rather than requiring a full M×M convolution.
Data Source
AI summary
A device for decoding video data receives a picture of video data; reconstructs a block of the picture of video data to generate a reconstructed block; applies a neural network (NN)-based filter process to the reconstructed block to generate a filtered block, wherein the NN-based filter process includes a first backbone block process followed by a second backbone block process, wherein the first backbone block process comprises a first M×N convolution followed by a first N×M convolution, and the second backbone block process comprises a second N×M convolution followed by a second M×N convolution, wherein N and M are different integer values; determines a decoded block of video data based on the filtered block; and outputs a decoded version of the picture, wherein the decoded version of the picture comprises the decoded block of video data.