Hardware-Friendly Attention Map Normalization for Video In-Loop Filters

Solution Overview

Problem

Existing video coding techniques using neural network-based in-loop filters (NN-ILFs) face inefficiencies due to non-hardware-friendly normalization operations in attention maps, leading to increased processing complexity and power consumption, especially when dealing with varying block sizes.

Innovation Solution

Normalize attention maps in a hardware-friendly manner by modifying them based on the size of blocks used for training, utilizing linear operations such as scaling and averaging, to maintain effective filtering across different block sizes.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If non-linear operations (square roots and exponential functions) are used to normalize the attention map, then the attention map can be properly normalized, but the processing complexity and power consumption increase

Engineering Contradiction:

Improveattention map normalization accuracyVSAvoidprocessing complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent changes the normalization approach by using linear operations (scaling and averaging) instead of non-linear operations (square roots and exponential functions). This parameter change in the normalization method reduces processing complexity and power consumption while maintaining effective filtering performance across different block sizes

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent employs a simplified normalization technique that uses computationally inexpensive linear operations rather than expensive non-linear operations. This allows the system to achieve adequate normalization without requiring complex computational resources, effectively trading off some normalization precision for significantly reduced processing complexity

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

2Measurement precision

If non-linear operations are used to normalize the attention map, then the attention map can be properly normalized, but the power consumption increases

Engineering Contradiction:

Improveattention map normalization accuracyVSAvoidpower consumption

Core Design Contradiction:

Measurement precisionVSUse of energy by moving object

Solution Approach 1:

The patent changes the normalization approach by using linear operations (scaling and averaging) instead of non-linear operations (square roots and exponential functions). This parameter change in the normalization method reduces processing complexity and power consumption while maintaining effective filtering performance across different block sizes

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent employs a simplified normalization technique that uses computationally inexpensive linear operations rather than expensive non-linear operations. This allows the system to achieve adequate normalization without requiring complex computational resources, effectively trading off some normalization precision for significantly reduced processing complexity

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

3Ease of manufacture

If the attention map normalization method is fixed, then the implementation is simple, but it cannot adapt to varying block sizes

Engineering Contradiction:

Improveimplementation simplicityVSAvoidadaptability to block sizes

Core Design Contradiction:

Ease of manufactureVSAdaptability or versatility

Solution Approach 1:

The patent makes the normalization method dynamic by adjusting scaling factors based on the actual block size being processed. This allows the same normalization framework to adapt to varying block sizes (e.g., 16x16, 32x32, 64x64) without requiring separate fixed normalization methods for each size, thereby maintaining implementation simplicity while achieving adaptability

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS20250301132A1Attention map normalization for in-loop filtering for video coding

Publication Date: 2025.09.25 QUALCOMM INC

AI summary

A method of processing video data includes receiving, with a neural network in-loop filter (NN-ILF), a current block of video data of a current picture; and filtering, with the NN-ILF, the current block of video data to generate a filtered current block of video data, wherein filtering the current block of video data comprises: generating, with an attention block of the NN-ILF, an attention map indicative of a correlation between elements of features of the current block of video data; modifying, with the attention block of the NN-ILF, the attention map based on a size of blocks used for training the NN-ILF to generate a modified attention map; generating, with the attention block of NN-ILF, feature data based on the modified attention map; and filtering, with the NN-ILF, the current block of video data based on the feature data to generate the filtered current block of video data.