Neural Net Video Coding Adaptation via On-Line Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network-based video/image encoding methods focus on enhancing encoding tools but do not leverage on-line learning approaches for adaptive prediction, limiting the efficiency of neural net-based prediction in video/image compression.
Innovation Solution
A method that determines if a block or picture is used for on-line learning, reconstructs a coarse version, enhances it using a neural net, and fine-tunes the neural net with a training signal, allowing adaptive encoding and decoding with neural net enhancement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional pre-defined neural net approaches are used for video coding, then encoding enhancement is achieved, but adaptability to different content is limited
Solution Approach 1:
The neural net transitions from a static pre-defined model to a dynamic adaptive model that evolves during the encoding process. The neural net parameters are updated in real-time based on the content being encoded, allowing the system to adapt its prediction capabilities to different video content characteristics while maintaining a relatively simple overall architecture.
Solution Approach 2:
The encoding system uses its own encoded content as training data to continuously improve its neural net model. The reconstructed blocks from the encoding process themselves become the training signals, allowing the system to self-enhance without requiring external training datasets or complex pre-training procedures.
2Productivity
If on-line learning is implemented for neural net-based prediction, then compression efficiency is improved, but computational load increases
Solution Approach 1:
Instead of performing full neural net training on all encoded content, the system selectively applies on-line learning updates only to relevant portions of the neural net model using subsets of the encoded content. This partial action approach provides compression efficiency improvements while limiting the computational energy consumption to manageable levels.
Solution Approach 2:
The on-line learning process operates continuously throughout the encoding process, using every reconstructed block as a potential training signal. This continuous accumulation of training data over time allows the neural net to progressively improve compression efficiency without requiring intensive batch processing that would consume excessive computational energy at any single moment.
3Measurement precision
If coarse version reconstruction is performed for neural net enhancement, then prediction accuracy is improved, but encoding time increases
Solution Approach 1:
The coarse version of the block is reconstructed and prepared in advance before the neural net enhancement is applied. This preliminary reconstruction provides a simplified input that speeds up the neural net processing while still capturing the essential features needed for accurate prediction, thereby reducing the overall encoding time compared to processing full-resolution blocks.
Solution Approach 2:
The approach transitions from operating directly on full-resolution blocks to operating on a downsampled coarse version in a different resolution dimension. This dimensional transformation reduces the computational complexity of the neural net processing while maintaining prediction accuracy, as the neural net learns to recover fine details from the coarse representation.
Data Source
AI summary
A method comprising: obtaining a block of a picture or a picture in an encoder; determining if the block/picture is used for on-line learning; if affirmative, encoding the block/picture; reconstructing a coarse version of the block/picture or the respective prediction error block/picture; enhancing the coarse version using a neural net; fine-tuning the neural net with a training signal based on the coarse version; determining if the block/picture is enhanced using the neural net; and if affirmative, encoding the block/picture with enhancing using the neural net.