Content Adaptive Attention Model for Neural Network Encoders
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deep neural networks (DNNs) for image and video encoding do not consider the sensitivities of the human visual system (HVS), leading to omission of relevant details and suboptimal encoding quality, as they do not differentiate between different textures and patterns.
Innovation Solution
Incorporating a content adaptive attention layer into neural network-based encoders to learn which image regions are more important from the HVS perspective, allowing for more bits to be allocated to sensitive regions, thereby enhancing visual quality at the same bitrate.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If conventional DNN solutions are used for encoding images and videos, then the encoding process is simple and fast, but relevant details are omitted because the DNNs are not aware of HVS sensitivities
Solution Approach 1:
The encoder is segmented into multiple functional components: a base DNN encoder for general encoding, a separate HVS sensitivity analysis module for detecting important regions, and a detail enhancement module for selectively processing sensitive areas. This segmentation allows the system to maintain simple fast encoding paths while adding targeted complexity only where HVS sensitivity requires it.
Solution Approach 2:
The encoder applies different processing quality to different regions of the image/video based on HVS sensitivity analysis. High-frequency texture regions and important semantic areas receive enhanced detail processing, while less sensitive regions use standard encoding. This local quality differentiation improves overall encoding quality without uniformly increasing complexity across the entire encoding pipeline.
2Manufacturing precision
If uniform encoding is applied to all image regions, then the encoding process is simple, but sensitive regions lose important visual details
Solution Approach 1:
The encoder dynamically changes encoding parameters (such as quantization step size, transform block size, and prediction mode) based on local HVS sensitivity measurements. Regions with high HVS sensitivity receive parameter adjustments that preserve more detail, while insensitive regions use aggressive compression parameters. This parameter adaptation improves visual quality in sensitive regions without uniformly reducing encoding efficiency across the entire content.
Solution Approach 2:
Instead of applying full-detail encoding to the entire image, the system applies enhanced encoding actions only partially to regions where HVS sensitivity indicates it is necessary. This selective application of excessive encoding action to specific regions maintains overall encoding efficiency while ensuring that visually critical areas receive the additional processing they need.
3Manufacturing precision
If more bits are allocated to all regions, then visual quality improves, but bitrate consumption increases
Solution Approach 1:
The bitrate is distributed non-uniformly across different image regions based on HVS sensitivity analysis. Sensitive regions such as high-frequency textures and important semantic areas receive higher bit allocation, while insensitive regions receive lower bit allocation. This local quality differentiation ensures that visual quality improves in critical regions without proportionally increasing overall bitrate consumption.
Solution Approach 2:
The encoder adjusts local bitrate allocation parameters dynamically based on HVS sensitivity measurements and content characteristics. By changing parameters such as quantization precision and transform resolution locally rather than globally, the system achieves improved visual quality in sensitive regions while maintaining efficient bitrate utilization across the entire encoded content.
Data Source
AI summary
Various embodiments are generally directed to using attention models in neural network-based image and video encoders and/or decoders. A first feature map of a first image may be generated by a first layer of a neural network, the neural network executing on a computer processor to encode the first image. An attention layer of the neural network may compute an adaptive spatial saliency map for the first feature map of the first image based on the first feature map of the first image. The neural network may then perform an element-wise multiplication of the first feature map and the adaptive spatial saliency map for the first feature map to generate a modulated feature map to encode the first image.