Neural Network Intra Prediction Iterative Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video compression standards face challenges in training neural networks for intra prediction, as they either become too generic and provide blurry predictions or overly specialize in the codec, failing to strike a balance between genericity and specialization.
Innovation Solution
An iterative training method for neural networks is proposed, where the networks are initially trained on partitioned video block and context pairs, then retrained iteratively using these pairs extracted with the neural network as an additional intra coding mode, to learn an intra prediction diverging from the codec while maintaining rate-distortion performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If neural networks are trained on generic video data, then they provide broad applicability, but prediction accuracy deteriorates (blurry predictions)
Solution Approach 1:
The patent applies preliminary action by first training neural networks on generic video data to establish broad applicability, then iteratively retraining them on codec-specific data extracted using the networks themselves. This preliminary generic training provides a foundation that is later refined through iterative specialization, allowing the networks to maintain both adaptability and accuracy.
Solution Approach 2:
The patent implements dynamics through iterative retraining where the training data is dynamically extracted using the current state of the neural networks. The process alternates between encoding with the codec and retraining the networks, creating a dynamic feedback loop that continuously adapts the networks to the specific codec characteristics while maintaining their generic capabilities.
2Measurement precision
If neural networks are trained to specialize in the codec, then prediction accuracy improves, but they fail to balance genericity and specialization
Solution Approach 1:
The patent applies feedback by using the neural networks themselves to extract training data during the iterative retraining process. The networks encode video blocks, and the resulting encoded data feeds back into the training set for the next iteration. This feedback mechanism ensures the networks specialize in the codec while the iterative process maintains their ability to generalize.
Solution Approach 2:
The patent implements the nested doll principle by embedding the neural networks within the codec framework during training. The networks are trained using data that is generated by the codec itself, creating a nested structure where the networks learn from the codec's internal representations. This nesting allows the networks to achieve specialization while maintaining compatibility with the broader codec system.
3Productivity
If standard video compression schemes are used, then compression efficiency is achieved, but they cannot provide accurate intra predictions for neural network integration
Solution Approach 1:
The patent introduces an intermediary approach by using the neural networks as a bridge between standard video compression and accurate intra prediction. The networks are trained to work within the existing codec framework, acting as an intermediary layer that enhances prediction accuracy without disrupting the overall compression pipeline. This allows standard compression schemes to maintain efficiency while incorporating improved neural network-based prediction.
Data Source
AI summary
An iterative training of neural networks for video coding and decoding using intra prediction is provided that finds a tradeoff between an extreme genericity and an extreme specialization to a codec for the trained neural networks. At the first iteration, the set of neural networks is trained following a partitioning approach. Then, for several iterations, the set of neural networks is inserted into the codec, and pairs of a block and its context are extracted from the partitioning of images via the codec with a single additional neural network-based mode then, the neural networks are retrained on these pairs. This way, from the second iteration, the neural networks learn an intra prediction diverging from that in the codec while still being valuable for the codec in terms of rate-distortion performance.