ResNet In-Loop Video Filters with Hardware-Friendly Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current implementations of transformer blocks in video coding include operators that are not hardware friendly, such as exponent and square root operators, leading to inefficiencies in hardware implementation due to the need for approximations during quantization.
Innovation Solution
A neural network-based filter with backbone blocks, including at least one attention block that processes non-normalized data, eliminating normalization layers and performing only multiplication and addition operations, thereby simplifying hardware implementation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If transformer blocks with normalization layers are used in video coding, then filtering performance is improved, but hardware implementation complexity increases due to non-hardware-friendly operators
Solution Approach 1:
The patent removes normalization layers from the transformer block architecture, extracting the problematic components (exponent operators, square root operators) that cause hardware implementation difficulties. This allows the model to maintain filtering performance while eliminating operations that are not hardware-friendly.
Solution Approach 2:
The patent replaces complex normalization operations with simpler, more computationally efficient operations that can be implemented using basic hardware components. The design accepts that some precision may be sacrificed in favor of operations that can be efficiently implemented in hardware without requiring complex approximation circuits.
2Adaptability or versatility
If non-linear operators like exponent and square root are used, then model expressiveness is improved, but quantization efficiency deteriorates due to approximation requirements
Solution Approach 1:
The patent removes non-linear operators (exponent, square root) from the transformer block architecture. By extracting these problematic operations, the model maintains its expressive capabilities through alternative means while eliminating the need for complex quantization approximations.
Solution Approach 2:
The patent replaces complex mathematical operations (exponent, square root) with simpler arithmetic operations (multiplication, addition) that are more amenable to efficient quantization and hardware implementation. This substitution maintains the essential functionality while improving computational efficiency.
3Measurement precision
If normalization layers are included in the attention block, then processing accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent removes normalization layers from the attention block, extracting the computational overhead associated with these operations. This allows the model to maintain processing accuracy through alternative mechanisms while significantly reducing computational complexity.
Solution Approach 2:
The patent replaces expensive normalization operations with cheaper arithmetic operations (multiplication and addition only) that can be performed efficiently without requiring complex computational resources. This substitution maintains adequate processing accuracy while reducing the computational burden.
Data Source
AI summary
A device for decoding encoded video data is configured to determine, from the encoded video data, a block of a picture; apply a neural network (NN)-based filter to the block to generate a filtered block, wherein the NN-based filter comprises a plurality of backbone blocks and at least one of the backbone blocks comprises an attention block configured to process non-normalized data; determine a decoded version of the block based on the filtered block; and output a decoded version of the picture comprising the decoded version of the block.