Neural Network Video Filter Complexity Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network (NN)-based video coding tools face high computational complexity and memory bandwidth requirements, particularly in filtering processes due to the need for extensive computations across all input data types, which can limit their efficiency and scalability in video encoding and decoding processes.
Innovation Solution
Implementing techniques that reduce computational complexity by performing fewer computations for supplementary data types, such as using 1×1 convolutions instead of 3×3 convolutions and applying feature map derivation or value replication, specifically for data with sparse representations like quantization parameters and boundary strength, to streamline the initial processing stage in NN-based filtering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If NN-based filter performs extensive computations on all input data types using 3×3 convolutions, then filtering effectiveness is improved, but computational complexity and memory bandwidth requirements increase
Solution Approach 1:
The patent applies different convolution operations to different input data types based on their specific characteristics. Reconstruction samples and prediction samples receive 3×3 convolutions to preserve spatial relationships and filtering effectiveness, while supplementary data (QP, partitioning information, boundary strength) receive 1×1 convolutions or feature map derivation to reduce computational complexity. This localized differentiation resolves the contradiction by maintaining quality where needed and reducing complexity where sufficient.
Solution Approach 2:
The patent applies partial action by performing fewer computations on supplementary data types that have sparse representations. Instead of uniformly applying 3×3 convolutions to all input data, the system selectively applies computational operations only where necessary, using 1×1 convolutions or value replication for supplementary data. This partial approach reduces overall computational complexity while maintaining adequate filtering effectiveness through the combined processing of all data types.
2Stability of the object's composition
If NN-based filter processes all input data with the same computational operations, then processing consistency is improved, but computational load and memory bandwidth requirements increase
Solution Approach 1:
The patent implements local quality by tailoring computational operations to the specific nature of each input data type. Reconstruction and prediction samples use 3×3 convolutions to maintain spatial consistency, while supplementary data use 1×1 convolutions or feature map derivation. This differentiated approach maintains processing consistency for each data type while reducing overall computational load and energy consumption.
Solution Approach 2:
The patent changes the convolution parameters (kernel size) based on data type characteristics. By switching from 3×3 to 1×1 convolutions for supplementary data, the system adjusts computational parameters to match the sparse representation nature of this data, thereby reducing computational load and energy consumption while maintaining adequate processing consistency.
3Measurement precision
If 3×3 convolutions are applied to supplementary data with sparse representations, then feature extraction quality is improved, but computational complexity increases significantly
Solution Approach 1:
The patent applies local quality by matching the computational operation to the data characteristics. For supplementary data with sparse representations (single QP value per block, partitioning information), 1×1 convolutions or feature map derivation are used instead of 3×3 convolutions. This localized approach maintains adequate feature extraction quality for these data types while significantly reducing computational complexity compared to uniform 3×3 convolution application.
Solution Approach 2:
The patent uses value replication techniques for supplementary data with sparse representations. Instead of performing computationally intensive 3×3 convolutions on data that has constant values across spatial dimensions (like single QP values per block), the system replicates these values through feature map derivation, achieving the necessary feature extraction with minimal computational complexity.
Data Source
AI summary
A video encoder and video decoder are configured to perform a neural network (NN)-based filter process on reconstructed blocks of video data. In one example, the NN-based filter process uses reconstruction samples of the block, prediction samples of the block, and supplementary data related to the block as inputs. The NN-based filter process includes an initial processing of one or more types of the supplementary data with fewer computations relative to the initial processing of the reconstruction samples and the prediction samples.