ML-Based Coded Size Estimation for Video Rate Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video encoding methods face challenges in accurately estimating coded size, leading to buffer underflow or overflow issues due to the complex relationship between coded size and picture complexity, which affects encoding quality and performance.
Innovation Solution
The implementation of machine learning-based coded size estimation using neural networks, such as multilayer perceptrons (MLPs) and convolutional neural networks (CNNs), that learn the relationship between coded size and picture characteristics, enabling more accurate estimation and integration into the encoder for real-time rate control.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If traditional estimation methods assume a nonlinear relationship between coded size and QP, then the estimation process is simplified, but the accuracy of coded size estimation deteriorates due to the complex and variable relationship between coded size and picture complexity
Solution Approach 1:
The patent replaces traditional mathematical modeling approaches (mechanical/system-based estimation formulas) with machine learning models that automatically learn the complex nonlinear relationships between picture characteristics and coded size. The ML model is trained on historical encoding data to capture the variable relationship between picture complexity and coded size, eliminating the need for manual formulation of estimation equations while significantly improving accuracy.
Solution Approach 2:
The patent transforms the estimation approach by changing from fixed parametric models to adaptive parameter learning. The ML model dynamically adjusts its internal parameters (weights and biases) based on training data, allowing it to adapt to different video content characteristics, resolutions, and encoding conditions. This enables accurate estimation without requiring explicit mathematical formulation of the complex relationships.
2Measurement precision
If machine learning models are used for coded size estimation, then the accuracy of estimation improves, but the computational overhead increases
Solution Approach 1:
The patent performs the computationally intensive model training process in advance, before real-time encoding operations. The trained ML model is then deployed for inference during encoding, where it requires minimal computational resources. This preliminary action separates the heavy computational workload (training) from the real-time operation (inference), enabling accurate estimation without significant overhead during actual encoding.
Solution Approach 2:
The patent uses lightweight ML model architectures that can be efficiently deployed on encoding hardware. The models are designed to be computationally inexpensive during inference, using simplified network structures, quantization, and pruning techniques. This allows the system to use sophisticated ML models for accurate estimation while keeping the runtime computational overhead low enough for practical deployment.
3Reliability
If coded size estimation accuracy is improved to prevent buffer underflow and overflow, then encoding quality improves, but the device complexity increases due to integration of machine learning models
Solution Approach 1:
The patent integrates the ML-based estimation model directly into the existing rate control architecture of the video encoder. The model becomes a component within the rate control loop, working together with traditional rate control mechanisms. This merging approach allows the system to benefit from both the adaptive capabilities of ML and the proven reliability of traditional rate control, improving buffer control stability without creating a completely new complex system.
Solution Approach 2:
The patent introduces the ML model as an intermediary component that provides refined coded size predictions to the rate control mechanism. Rather than replacing the entire rate control system, the ML model acts as a mediator that enhances the accuracy of size predictions, which then feed into the existing buffer management and QP adjustment logic. This intermediary approach improves reliability while maintaining compatibility with established encoder architectures.
Data Source
AI summary
Techniques for machine learning based coded size estimation in rate control of video encoding are described. An encoder in accordance with various embodiments uses one or more machine learning approaches to facilitate the rate control of video encoding. When training one or more neural network models, the relationship between the coded size and unit characteristics (e.g., picture pixels and/or picture features) is learned from past encoding. The encoder then uses the trained model(s) to estimate the coded size through model inference with improved accuracy. In some embodiments, the trained model(s) are integrated into the encoder for direct model inference. The direct model inference reduces the overhead of referencing application programming interfaces (APIs) provided by a separate machine learning platform, thus making the rate control methods and systems described herein useful in real time video encoding.