Perceptually-Aware Online Per-Title Encoding for Live Video
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current per-title encoding for live video streaming is computationally expensive and introduces latency, making it unsuitable for real-time applications, despite its effectiveness in Video on Demand systems.
Innovation Solution
A low-latency perceptually aware online per-title encoding method that extracts spatial and temporal energy features using Discrete Cosine Transform (DCT) and predicts optimal bitrate-resolution pairs based on a just noticeable difference (JND) quality model, reducing the number of required pairs and leveraging machine learning for efficient convex-hull prediction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If per-title encoding is used to optimize bitrate ladders for each video content, then video quality is improved, but computational complexity and latency increase
Solution Approach 1:
The system pre-trains machine learning models offline to learn the relationship between video content characteristics and optimal bitrate-resolution pairs. During live streaming, these pre-trained models are used to make rapid predictions without requiring complex real-time computations, thus maintaining low latency while achieving per-title optimization
Solution Approach 2:
The patent replaces traditional brute-force convex-hull computation with machine learning-based prediction. Instead of computationally expensive optimization algorithms, a trained neural network model predicts optimal encoding parameters directly from extracted video features, dramatically reducing computational complexity and latency
2Manufacturing precision
If convex-hull determination is performed for all bitrate-resolution pairs, then optimal encoding is achieved, but computational cost increases
Solution Approach 1:
The system extracts only the most relevant features from video content (spatial energy, temporal energy, motion complexity) rather than analyzing all possible bitrate-resolution pairs. This selective extraction of critical characteristics enables the machine learning model to make accurate predictions with minimal computational input
Solution Approach 2:
The patent uses a trained machine learning model that has learned optimal encoding patterns from training data. Instead of performing complex optimization computations during live streaming, the system copies the knowledge gained during offline training to make rapid predictions, avoiding repeated computational complexity
3Productivity
If machine learning methods are used for bitrate prediction, then encoding efficiency is improved, but latency remains too high for live streaming
Solution Approach 1:
The machine learning models are pre-trained offline using training datasets with various video characteristics and ground-truth optimal bitrate-resolution pairs. This preliminary training phase transfers the computational burden to offline processing, enabling rapid inference during live streaming with minimal latency
Solution Approach 2:
The system dynamically adapts the complexity of feature extraction and model selection based on video content characteristics and available processing resources. For highly dynamic content, the system may use simplified models with fewer features, while for static content, more comprehensive analysis can be performed, optimizing the balance between efficiency and accuracy in real-time
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach significantly reduces latency in live video streaming while maintaining high video quality by optimizing bitrate ladders in real-time, ensuring efficient encoding without disrupting the streaming process.
Implementation Method 1
extracting two or more features for each of a plurality of segments of the input video, the two or more features comprising a spatial energy feature and an average temporal energy
Data Source
AI summary
Techniques for implementing perceptually aware per-title encoding may include receiving an input video, a set of resolutions, a maximum target bitrate and a minimum target bitrate, extracting content aware features for each segment of the input video, predicting a perceptually aware bitrate-resolution pair for each segment using a model configured to optimize for a quality metric using constants trained for each of the set of resolutions, generating a target encoding set including a set of perceptually aware bitrate-resolution pairs, and encoding the target encoding set. The content aware features may include a spatial energy feature and an average temporal energy. According to these methods only a subset of bitrates and resolutions, less than a full set of bitrates and resolutions, are encoded to provide high quality video content for streaming.