Multi-Scale Video Super-Resolution With Lower Decoder Complexity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video compression technologies face challenges in reducing compression artifacts and improving visual quality while managing complexity and computational resources efficiently.
Innovation Solution
Implementing a multi-scale machine learning model, such as a multi-scale convolutional neural network (MSCNN), within or outside the video decoder's prediction loop, to enhance compressed video by reducing complexity and improving visual quality through super-resolution restoration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If a machine learning model is used for super-resolution enhancement of compressed video, then visual quality is improved, but device complexity increases
Solution Approach 1:
The patent divides the video processing into multiple scales (e.g., full resolution and half resolution paths) and processes different components at different resolutions. The multi-scale architecture segments the computational workload to reduce overall complexity while maintaining enhancement quality.
Solution Approach 2:
The patent implements dynamic model construction where the neural network structure can be adaptively configured based on processing requirements. The system can dynamically select between different model configurations (e.g., using half resolution path when computational resources are constrained) to balance quality and complexity.
2Reliability
If a machine learning model is used for super-resolution enhancement, then compression artifacts are reduced, but computational demands increase
Solution Approach 1:
The patent segments the computational workload by processing video at multiple resolution scales simultaneously. The half resolution path handles computational-intensive operations at lower precision, reducing overall energy consumption while the full resolution path maintains quality where needed.
Solution Approach 2:
The patent applies different processing qualities to different parts of the video data. Important regions or frames can be processed at higher resolution while less critical regions use lower resolution processing, optimizing the balance between artifact reduction and computational energy consumption.
Data Source
AI summary
The present disclosure relates to methods, apparatus, systems, and non-transitory computer-readable storage media for training and using a multi-scale machine learning model for the super-resolution enhancement of compressed video. According to some examples, a computer-implemented method includes receiving a video at a content delivery service; downsampling a source frame of the video to generate a frame; performing an encode on a the frame of the video by the content delivery service that coverts the frame from a pixel domain to a transform domain and back to the pixel domain to generate first pixel values and a first residual for a block of the frame at a first resolution; generating a first set of features at the first resolution, by a machine learning model of the content delivery service, for a first input at the first resolution, of the first pixel values and the first residual of the block; upsampling the first set of features to a target resolution to generate an upsampled first set of features; generating a second set of features at a second lower resolution than the first resolution, by the machine learning model of the content delivery service, for a second input based on the first pixel values and the first residual of the block; upsampling the second set of features to the first target resolution to generate an upsampled second set of features; generating a modified version of the frame based on the upsampled first set of features and the upsampled second set of features; and transmitting the modified version of the frame to a frame buffer or from the content delivery service to a viewer device.