Neural Codec Proxy for Perceptual Image Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current perceptual optimization approaches in image and video encoders require multiple encoding passes, are limited to single quality scoring methods, fail to encapsulate advanced quality scoring methods like VMAF or AVQT, offer minor BD-rate improvements, and are not compatible with standards-based encoders.
Innovation Solution
A neural codec model is used as a proxy between pre- and post-processing ANNs to emulate rate and distortion characteristics of standards-based codecs, enabling end-to-end training and improving coding efficiency and perceptual quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple encoding passes or in-loop implementation are used for perceptual optimization, then quality scoring improvement is achieved, but device complexity and processing time increase
Solution Approach 1:
The patent applies preliminary action by performing perceptual optimization before the standard encoding process. A neural network processes the input image to generate optimized representations that are then fed into the standard codec, allowing quality improvement without requiring multiple encoding passes or in-loop implementation
Solution Approach 2:
The patent introduces a neural network as an intermediary component between the input image and the standard codec. This intermediary learns to embed information in a codec-aware manner, acting as a bridge that improves quality without increasing the complexity of the standard encoding process itself
2Measurement precision
If optimization is performed for a single quality scoring method like VMAF or SSIM, then that specific quality metric improves, but performance on other quality scores deteriorates
Solution Approach 1:
The patent applies universality by training the neural network to optimize for multiple quality scoring methods simultaneously. The network learns to embed information in a way that benefits various quality metrics (VMAF, SSIM, AVQT) rather than being tuned for a single specific metric, making the solution universally applicable across different quality evaluation standards
3Ease of manufacture
If hand-crafted models of low-level human perception are used, then implementation simplicity is maintained, but they fail to encapsulate advanced quality scoring methods like VMAF or AVQT
Solution Approach 1:
The patent replaces hand-crafted mechanical models of human perception with a data-driven neural network. Instead of using fixed mathematical models that are difficult to train, the system uses a neural network that learns from data to capture complex perceptual characteristics, enabling accurate modeling of advanced quality scoring methods like VMAF and AVQT
4Productivity
If entirely neural approaches are used for image processing, then coding efficiency improves, but compatibility with standards-based encoders is lost
Solution Approach 1:
The patent applies segmentation by dividing the image processing pipeline into two distinct segments: a neural network-based pre-processing stage for perceptual optimization, and a standards-based codec stage for compression. This segmentation allows the system to leverage the efficiency of neural networks while maintaining compatibility with standard encoders like HEVC or AV1
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method of processing image data, comprising receiving, at a pre-processing artificial neural network, ANN, image data of one or more images, pre-processing the received image data at the pre-processing ANN to generate pre-processed image data of the one or more images, encoding and decoding, in accordance with an image or video codec, the pre-processed image data to generate decoded image data of the one or more images, and post-processing the decoded image data at a post-processing ANN to generate post-processed image data of the one or more images. The pre-processing ANN and the post-processing ANN are jointly trained in an end-to-end manner using a neural codec model arranged between the pre-processing ANN and the post-processing ANN, the neural codec model acting as a proxy for the image or video codec and comprising an ANN configured to emulate rate and distortion characteristics of the image or video codec.