Joint Video Upscaling for Low-Bitrate Reconstruction Quality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video streaming systems face issues with sub-optimal visual quality and interoperability of downscalers and upscalers, leading to artifacts and increased bit usage for achieving target visual quality.
Innovation Solution
Jointly train a downscaling convolutional neural network (CNN) and an upscaling CNN to minimize reconstruction errors, using metadata to identify the appropriate upscaler for improved visual quality and interoperability across different client devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If downscalers and encoders eliminate information from source video to generate different video encodings, then bitrate is reduced for efficient streaming, but visual quality of reconstructed video deteriorates with more artifacts
Solution Approach 1:
The downscaler is trained in advance with knowledge of the specific upscaler that will be used for reconstruction. This preliminary training allows the downscaler to pre-compensate for the upscaler's characteristics, optimizing the downscaling process to minimize reconstruction errors when the specific upscaler is applied, thereby maintaining visual quality at reduced bitrates
Solution Approach 2:
The training process uses feedback from the upscaler's reconstruction output to adjust the downscaler's parameters. By measuring reconstruction errors and using them to refine the downscaler's training, the system continuously optimizes the trade-off between bitrate reduction and visual quality preservation
2Adaptability or versatility
If general-purpose upscalers are used for upscaling decoded video, then interoperability across different devices is improved, but visual quality of reconstructed video deteriorates due to lack of optimization
Solution Approach 1:
The system segments the upscaling function into device-specific trained upscalers tailored to different device types (mobile, desktop, TV). Each upscaler is optimized for its target device's characteristics, and the endpoint application selects the appropriate upscaler based on device identification, thereby achieving both high visual quality and broad interoperability
Solution Approach 2:
Different upscalers with specialized characteristics are applied to different device types. Each upscaler has locally optimized parameters and processing characteristics matched to its target device's display capabilities and processing strengths, ensuring optimal visual quality for each device category while maintaining overall system versatility
3Adaptability or versatility
If trained downscalers are used with multiple different upscalers, then versatility is improved, but visual quality deteriorates due to lack of joint optimization
Solution Approach 1:
Each downscaler is preliminarily trained with knowledge of multiple specific upscalers that it will work with. During training, the downscaler learns to compensate for the characteristics of each paired upscaler, enabling it to maintain high visual quality when used with any of its trained partners while preserving compatibility across multiple device types
Data Source
AI summary
In various embodiments an endpoint application reconstructs downscaled videos. The endpoint application accesses metadata associated with a portion of a downscaled video that has a first resolution and was generated using a trained downscaling convolutional neural network (CNN). The endpoint application determines, based on the metadata, an upscaler that should be used when upscaling the portion of the downscaled video. The endpoint application executes the upscaler on the portion of the downscaled video to generate a portion of a reconstructed video that is accessible for playback and has a second resolution that is greater than the first resolution.