Neural Network Video Codec Parameter Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video encoding technologies face challenges with compression delay and bit-rate jitter due to the high bit requirements of I-frames, leading to larger and uneven bit streams, and require additional parameters for proper decoding, which increases the bit stream size.
Innovation Solution
The use of trained neural networks to predict and correct encoding parameters, reducing the need to encode all parameters by predicting them from previous frames and including prediction errors in the bit stream, thereby reducing the bit stream size and smoothing the bit rate.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If I-frames are used for random access points, then decoding capability is improved, but bit stream size increases and bit rate becomes uneven
Solution Approach 1:
The patent extracts only the essential information needed for decoding by using neural networks to predict and encode only the differences (residuals) between frames rather than encoding complete frame data. This allows random access through I-frames while significantly reducing the bit stream size by removing redundant information.
Solution Approach 2:
The patent changes the encoding parameters from traditional complete frame encoding to difference-based encoding using neural network predictions. By transforming the data representation from full frame to residual information, the system maintains decoding capability while reducing bit stream volume and smoothing bit rate.
2Quantity of substance
If neural networks predict encoding parameters, then bit stream size is reduced, but prediction accuracy must be maintained for proper decoding
Solution Approach 1:
The patent implements feedback mechanisms where the neural network predictions are continuously refined and corrected based on actual encoded parameters. The system uses feedback loops to adjust prediction models, ensuring high accuracy while maintaining reduced bit stream size through iterative optimization.
Solution Approach 2:
The patent performs preliminary training of neural networks on large datasets of video encoding parameters before actual encoding. This preliminary action enables the networks to make highly accurate predictions during operation, reducing the need for transmitting complete parameter sets while maintaining decoding precision.
3Reliability
If all encoding parameters are encoded explicitly, then decoding reliability is ensured, but bit rate becomes uneven and memory footprint increases
Solution Approach 1:
The patent applies partial encoding by transmitting only the necessary difference information predicted by neural networks rather than all encoding parameters. This partial action reduces bit stream size and memory footprint while maintaining decoding reliability through the use of prediction models that capture essential frame differences.
Solution Approach 2:
The patent introduces neural network predictions as an intermediary layer between the original video frames and the encoded bit stream. This intermediary compresses the information by predicting and encoding only the essential differences, acting as a mediator that reduces data volume while preserving decoding accuracy and reliability.
Data Source
Figure 1A~1C
Figure 2
Figure 3
AI summary
Training a video decoder system comprising, generating at least two sets of video encoding parameters wherein the at least two sets of video encoding parameters are valid, masking a set of the at least two sets of encoding parameters with invalid values to generate an invalid set of video encoding parameters, providing a set of the at least two sets of video encoding parameters to a neural network, training the neural network to predict valid video encoding parameter values for the invalid set using an iterative training algorithm, determining which encoding parameters need to be encoded based on analysis of a prediction error of the trained recurrent neural network, dropping the encoding parameters from the encoded data which are determined to be accurately predicted by the trained recurrent neural network, encoding a new video stream without the dropped encoding parameters. A coder and decoder system with neural network is also disclosed.