Video Coding Rate-Distortion Optimization Using Neural Network Distortion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video coding technologies face challenges in optimizing the encoding process for both human perceptual quality and machine analysis, particularly when dealing with computationally-heavy processes involving large and deep neural networks.
Innovation Solution
The proposed solution involves a method for rate-distortion optimization that incorporates both human-perception distortion and neural network distortion metrics. This is achieved by encoding multiple versions of an input image block with different coding parameter values, reconstructing each version, and then using a neural network to analyze the content and derive a distortion metric. The optimal coding parameters are selected based on a function of these distortions and encoded into a bitstream.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If video encoding is optimized for human perceptual quality, then human observation quality is improved, but machine analysis performance deteriorates
Solution Approach 1:
The patent segments the distortion metric into two separate components: human-perception distortion (measured by PSNR/SSIM) and neural network distortion (measured by feature extraction error). This allows independent optimization for both human observation and machine analysis by calculating both metrics and selecting coding parameters that balance both requirements.
Solution Approach 2:
The patent changes the optimization parameter from a single distortion metric to a composite parameter that includes both human-perception distortion and neural network distortion. By modifying the rate-distortion optimization function to consider both metrics, the system can select coding parameters that simultaneously satisfy both human observation quality and machine analysis performance requirements.
2Manufacturing precision
If multiple coding parameter versions are encoded and reconstructed, then optimal parameter selection is improved, but computational complexity increases
Solution Approach 1:
The patent applies partial action by evaluating multiple coding parameter versions only to the extent necessary for rate-distortion optimization. Instead of exhaustively testing all possible parameter combinations, the system evaluates a limited set of candidate parameters, reconstructs them, calculates distortion metrics, and selects the optimal one. This partial evaluation approach balances parameter selection accuracy with computational feasibility.
3Adaptability or versatility
If neural network distortion metric is incorporated into rate-distortion optimization, then machine analysis support is improved, but encoding complexity increases
Solution Approach 1:
The patent applies preliminary action by pre-training the neural network for distortion metric calculation before the actual encoding process. The neural network is trained offline to extract features and calculate distortion metrics efficiently. During encoding, the pre-trained network can quickly evaluate distortion for different coding parameters without requiring complex real-time training, thus reducing encoding complexity while maintaining machine analysis support.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
A method comprising encoding, in an encoder, several versions from an input image block, the encoding of the several versions being adjusted by different values of at least one coding parameter; reconstructing, per each of the several versions, a reconstructed image block from the encoded image block; deriving a first distortion between the reconstructed image block and the input image block; inputting the reconstructed image block into a first neural network trained to analyze content of decoded image content; obtaining a second distortion on the basis of loss of the first neural network; selecting values of the at least one coding parameter among the several versions using a function of said first and second distortions; and encoding the selected values into a bitstream.