ML-Based Video Downsampler for Rate Distortion Compensation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing downsampling methods in content streaming systems, such as bilinear, cubic, or Lanczos filters, are not perceptually optimal and do not account for the encoding process, leading to suboptimal quality in downsampling operations.
Innovation Solution
A machine learning (ML) model-based codec rate distortion compensating downsampler that includes a learned downsampler, a proxy video codec, a temporally-aware perceptual loss function, and a mechanism for inference-time interpolation, optimized to replicate the rate distortion characteristics of standard codecs, enabling perceptually-aware quality improvements without requiring client-side changes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If traditional filters (bilinear, cubic, or Lanczos) are used for downsampling, then the downsampling operation is simple and fast, but the perceptual quality is suboptimal and does not account for encoding artifacts
Solution Approach 1:
The patent introduces a machine learning model as an intermediary component between the downsampling operation and the final encoded output. This ML model acts as a mediator that learns the complex relationship between input video frames and encoding artifacts, enabling perceptually optimal downsampling without requiring direct modification of traditional filters or encoders.
Solution Approach 2:
The patent transforms the downsampling problem from a fixed filter-based approach to a learned parameter-based approach. By training the ML model on large datasets with various encoding conditions, the system dynamically adjusts downsampling parameters to optimize perceptual quality for different content types and encoding scenarios.
2Manufacturing precision
If perceptually optimal downsampling is implemented using machine learning models, then rate distortion performance is improved, but computational complexity and processing time increase
Solution Approach 1:
The patent performs the computationally intensive work of learning optimal downsampling strategies in advance during an offline training phase. The ML model is pre-trained on large datasets containing various video content and encoding conditions, so that during actual video processing, the model can quickly apply learned downsampling parameters without requiring complex real-time computations.
Solution Approach 2:
The ML model is designed to be self-adapting to different content types and encoding scenarios. Once trained, the model automatically selects appropriate downsampling parameters for each input video based on its characteristics, eliminating the need for manual parameter tuning or complex real-time optimization algorithms.
3Productivity
If downsampling is performed before encoding, then the encoding process becomes more efficient, but the quality is degraded due to non-perceptual filtering
Solution Approach 1:
The patent incorporates feedback mechanisms where the ML model learns from the interaction between downsampling operations and encoding artifacts. By analyzing how different downsampling approaches affect the subsequent encoding process, the model adjusts its parameters to minimize perceptual quality loss while maintaining encoding efficiency.
Solution Approach 2:
The patent replaces traditional mechanical filter-based downsampling systems with a learned neural network-based system. This substitution allows the downsampling operation to adapt to the specific characteristics of the encoding process and video content, achieving better quality outcomes while maintaining the efficiency benefits of pre-encoding downsampling.
Data Source
AI summary
A system includes a machine learning (ML) model-based video downsampler configured to receive an input video sequence having a first display resolution, and to map the input video sequence to a lower resolution video sequence having a second display resolution lower than the first display resolution. The system also includes a neural network-based (NN-based) proxy video codec configured to transform the lower resolution video sequence into a decoded proxy bitstream. In addition, the system includes an upsampler configured to produce an output video sequence using the decoded proxy bitstream.