Neural Image Pre- and Post-Processing for Codec-Aware Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current perceptual optimization approaches in image and video encoders require multiple encoding passes, are limited to single quality scoring methods, fail to encapsulate advanced quality scoring methods like VMAF or AVQT, offer minor BD-rate improvements, and are not compatible with standards-based encoders.
Innovation Solution
Employing pre- and post-processing artificial neural networks (ANNs) jointly trained in an end-to-end manner with a neural codec model that emulates the characteristics of a standards-based codec, allowing for improved coding efficiency and perceptual quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple encoding passes or in-loop implementation are used for perceptual optimization, then quality scoring improvement is achieved, but device complexity and processing time increase
Solution Approach 1:
The patent applies preliminary action by performing perceptual optimization before the encoding process. A pre-processing neural network analyzes the source image and generates optimized representations that guide subsequent encoding operations, eliminating the need for multiple encoding passes or in-loop adjustments while achieving improved quality scores
Solution Approach 2:
The patent introduces an intermediary neural network model that bridges the source image and the encoding process. This intermediary system processes the source image through a pre-processing neural network that outputs optimized representations, serving as a mediator that guides the encoding without requiring complex iterative adjustments
2Ease of manufacture
If hand-crafted models of low-level human perception are used, then implementation simplicity is maintained, but they fail to encapsulate advanced quality scoring methods like VMAF or AVQT
Solution Approach 1:
The patent replaces hand-crafted mechanical models of human perception with data-driven neural network models. The pre-processing neural network is trained using advanced quality scoring methods like VMAF and AVQT as loss functions, allowing the system to learn complex perceptual characteristics automatically rather than relying on simplified hand-crafted rules
Solution Approach 2:
The patent changes the fundamental parameters of the perception model from hand-crafted rules to learned neural network parameters. The neural network's weights and biases are optimized through training with advanced quality metrics, enabling the system to capture complex perceptual characteristics that hand-crafted models cannot represent
3Measurement precision
If entirely neural approaches are used for optimization, then advanced quality scoring methods can be encapsulated, but compatibility with standards-based encoders is lost
Solution Approach 1:
The patent segments the image processing pipeline into distinct functional components: a pre-processing neural network for perceptual optimization, a standards-based encoder for compression, and a post-processing neural network for quality enhancement. This segmentation allows each component to specialize in its function while maintaining compatibility with standards-based encoders
Solution Approach 2:
The patent uses neural networks as intermediary systems that bridge the gap between advanced perceptual optimization and standards-based encoding. The pre-processing neural network converts source images into optimized representations that are compatible with standard encoder formats, while the post-processing network enhances the final output without requiring changes to the encoder itself
4Productivity
If pre-processing and post-processing ANNs are used with joint end-to-end training, then coding efficiency and perceptual quality are improved, but training complexity and computational resources increase
Solution Approach 1:
The patent merges the pre-processing and post-processing neural networks into a single end-to-end trained system. Both networks are trained simultaneously using shared loss functions and optimized together, allowing the system to learn coordinated transformations that improve coding efficiency while sharing the training burden rather than requiring separate training processes
Data Source
AI summary
A method of processing image data, comprising receiving, at a pre-processing artificial neural network, ANN, image data of one or more images, pre-processing the received image data at the pre-processing ANN to generate pre-processed image data of the one or more images, encoding and decoding, in accordance with an image or video codec, the pre-processed image data to generate decoded image data of the one or more images, and post-processing the decoded image data at a post-processing ANN to generate post-processed image data of the one or more images. The pre-processing ANN and the post-processing ANN are jointly trained in an end-to-end manner using a neural codec model arranged between the pre-processing ANN and the post-processing ANN, the neural codec model acting as a proxy for the image or video codec and comprising an ANN configured to emulate rate and distortion characteristics of the image or video codec.