Neural Network Video Filter Multi-Scale Feature Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current video coding technologies face challenges in reducing computational complexity and memory bandwidth requirements for neural network-based filtering techniques, which are complex and resource-intensive, limiting their widespread adoption in video encoding and decoding devices.
Innovation Solution
The implementation of a neural network-based filter process that performs multi-scale feature extraction, where a first feature extraction is done at one scale and a second at a different scale, reducing overall complexity and enabling more devices to perform NN-based filtering while maintaining coding quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If neural network-based filtering techniques are implemented in video coding devices, then coding quality is improved, but computational complexity and memory bandwidth requirements increase
Solution Approach 1:
The neural network filter is divided into a headblock and a backbone structure. The headblock performs initial multi-scale feature extraction at reduced resolution, while the backbone processes features at full resolution. This segmentation allows the system to maintain coding quality while reducing overall computational complexity by performing heavy processing at lower resolutions.
Solution Approach 2:
The patent introduces multi-scale processing by operating on pixel data at different resolutions (scales). Feature extraction is performed at both a first scale (lower resolution) and a second scale (higher resolution), allowing the system to capture both global and local features efficiently, thereby reducing computational burden while maintaining quality.
2Manufacturing precision
If neural network-based filtering techniques are implemented in video coding devices, then coding quality is improved, but memory bandwidth requirements increase
Solution Approach 1:
The filter architecture segments processing across multiple scales and blocks. By performing feature extraction at reduced resolution in the headblock before upsampling and processing in the backbone, the system reduces the volume of data that needs to be transferred through memory, thereby reducing memory bandwidth requirements while maintaining coding quality.
Solution Approach 2:
Multi-scale feature extraction operates at different resolution levels, allowing the system to process fewer pixels at high resolution. This dimensional approach reduces the total number of operations and memory accesses required, decreasing memory bandwidth requirements while preserving coding quality through selective high-resolution processing.
3Manufacturing precision
If multi-scale feature extraction is performed throughout the entire filter, then coding quality is improved, but computational complexity increases
Solution Approach 1:
Multi-scale feature extraction is segmented and performed only in the headblock at reduced resolution, rather than throughout the entire filter. The backbone then processes the upsampled features at full resolution with a single scale. This segmentation maintains coding quality by capturing multi-scale features where most beneficial while reducing computational complexity by avoiding redundant multi-scale processing in the backbone.
Data Source
AI summary
A device for decoding video data determines a block of a picture; applies a neural network (NN)-based filter process to the block to generate a filtered block, wherein to apply the NN-based filter process, the device performs a first feature extraction on pixel data of the block at a first scale to generate a first set of extracted features for the block; and performs a second feature extraction on the pixel data of the block at a second scale to generate a second set of extracted features for the block, wherein the first scale is different than the second scale; and generates the filtered block based on the first set of extracted features and the second set of extracted features.