NN Video Coding Filters Using Separable Convolutions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network-based video coding tools are computationally complex and require significant memory bandwidth, limiting their widespread adoption and efficiency in video encoding and decoding processes.
Innovation Solution
Implement separable convolutions in place of multi-dimensional convolutions within neural network-based filters to reduce computational complexity and memory bandwidth requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multi-dimensional convolutions are used in NN-based filters, then filtering accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent applies segmentation by decomposing a multi-dimensional convolution operation into multiple separable one-dimensional convolution operations. Instead of performing a single complex 3×3 convolution, the filter divides the operation into sequential 1D convolutions along different dimensions, reducing the computational burden while maintaining filtering accuracy.
2Measurement precision
If multi-dimensional convolutions are used in NN-based filters, then filtering accuracy is improved, but memory bandwidth requirements increase
Solution Approach 1:
The patent segments the multi-dimensional convolution into multiple 1D convolution operations. This segmentation reduces the amount of data that needs to be loaded from memory at any given time, as each 1D operation processes fewer elements simultaneously compared to a full multi-dimensional convolution, thereby reducing memory bandwidth requirements.
Solution Approach 2:
The patent transforms the problem from a multi-dimensional operation to a series of one-dimensional operations. By changing the dimensionality of the convolution operations, the patent reduces memory access patterns and bandwidth requirements while preserving the essential filtering functionality through sequential application of 1D kernels.
3Device complexity
If separable convolutions are used instead of multi-dimensional convolutions, then computational complexity is reduced, but filtering accuracy may deteriorate
Solution Approach 1:
The patent merges multiple separable 1D convolution operations to approximate the effect of a multi-dimensional convolution. By combining the results of several 1D convolutions applied sequentially along different dimensions, the patent reconstructs the filtering effect of the original multi-dimensional operation, maintaining accuracy while reducing complexity.
Solution Approach 2:
The patent uses multiple 1D convolution operations (excessive action) to approximate a single multi-dimensional convolution. By applying more operations than the minimal required for basic filtering, the patent achieves an approximation that closely matches the accuracy of the full multi-dimensional convolution while keeping each individual operation computationally simple.
Data Source
AI summary
A video coder is configured to perform a neural network (NN)-based filter process on reconstructed blocks of vide data. In one example, a video coder may receive a picture of video data, and reconstruct a block of the picture of video data to generate a reconstructed block. The video coder may perform the NN-based filter process on the reconstructed block to generate a filtered block, wherein the NN-based filter process includes performing a plurality separable convolutions to approximate a multi-dimensional convolution.