Decoder-Side Neural Network Fine-Tuning for Adaptive Video Coding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video coding technologies lack efficient methods for decoder-side fine-tuning of neural networks to improve video quality and compression efficiency.
Innovation Solution
A method and apparatus for decoder-side fine-tuning of neural networks through iterative processes involving finetuning driving content, computing outputs, losses, and updates using approximated ground truth data, and applying these updates to decoder-side neural networks, utilizing techniques like backpropagation and stochastic gradient descent.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If video content is encoded in high quality, then video quality is improved, but compression efficiency deteriorates (larger file size)
Solution Approach 1:
The patent applies different quality levels to different regions of the video content. Important regions (e.g., regions containing objects of interest) are encoded at high quality, while less important regions are encoded at lower quality. This resolves the contradiction by making video quality subjective and region-dependent rather than uniformly high, thereby improving compression efficiency while maintaining perceived quality.
Solution Approach 2:
The patent dynamically adjusts quality allocation based on content analysis. The system identifies important regions in each frame and adapts the encoding quality accordingly, rather than using a static quality setting for the entire video. This dynamic approach optimizes the balance between video quality and compression efficiency for each specific region and frame.
2Ease of manufacture
If neural networks are trained offline, then training flexibility is improved, but adaptability to specific video content deteriorates
Solution Approach 1:
The patent performs preliminary offline training to establish a base neural network model with general capabilities. This preliminary action provides training flexibility and establishes a foundation that can be quickly adapted to specific video content through fine-tuning or parameter adjustment during encoding, resolving the contradiction between offline training benefits and content-specific adaptability.
Solution Approach 2:
The system enables the neural network to adapt to specific video content through self-service mechanisms during the encoding process. The network automatically adjusts its parameters or selects appropriate pre-trained models based on the characteristics of the input video content, eliminating the need for manual retraining while maintaining adaptability.
Data Source
AI summary
Various embodiments provide an apparatus, a method, and a computer program product. An example apparatus includes at least one processor; and at least one non-transitory memory comprising computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to iteratively perform following until a stopping criterion is met: provide a finetuning driving content (FDC) or a content derived from FDC to a decoder side neural network (DSNN); compute an output of the DSNN as a processed FDC; compute a loss based on the processed FDC and an approximated ground truth data (AGT) associated with the FDC; compute an update to the DSNN; and apply the computed update to the DSNN.


