Multi-Scale Machine Learning for Compressed Video Enhancement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video compression technologies face challenges in reducing compression artifacts and improving visual quality while managing computational complexity and memory consumption.
Innovation Solution
A multi-scale machine learning model, such as a multi-scale convolutional neural network (MSCNN), is integrated into the video decoder's prediction loop or used as a post-processing algorithm, employing methods to signal information through a bit-stream and utilizing group and one-dimensional convolution processes to reduce complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If a machine learning model is integrated into the video decoder's prediction loop or used as post-processing algorithm, then compression artifacts are reduced and visual quality is enhanced, but computational complexity and memory consumption increase
Solution Approach 1:
The patent divides the video processing into multiple scales (e.g., different resolution levels or block sizes). The machine learning model processes video data at multiple scales rather than a single scale, allowing complex enhancement tasks to be broken down into manageable segments that can be processed efficiently at each scale level.
Solution Approach 2:
The patent introduces multi-scale processing as an additional dimension to the traditional single-scale video decoding approach. By processing video data at multiple resolution levels or block sizes simultaneously, the system enhances visual quality without linearly increasing computational complexity, as each scale can be processed independently and then combined.
2Manufacturing precision
If a machine learning model is integrated into the video decoder's prediction loop or used as post-processing algorithm, then compression artifacts are reduced and visual quality is enhanced, but memory consumption increases
Solution Approach 1:
The patent segments the video processing into multiple scales, allowing the system to allocate memory resources efficiently at each scale level. By processing different portions of the video data at different scales rather than loading all data into memory simultaneously, the system reduces peak memory consumption while maintaining enhancement quality.
Solution Approach 2:
By adding the scale dimension to processing, the patent enables memory-efficient enhancement. The multi-scale approach allows the system to process and discard intermediate results at each scale level, avoiding the need to store all intermediate data in memory, thus reducing overall memory consumption compared to single-scale processing.
3Device complexity
If group and one-dimensional convolution processes are used, then computational complexity is reduced, but processing capability may be limited
Solution Approach 1:
The patent segments convolution operations into group convolutions (processing different feature channels in separate groups) and one-dimensional convolutions (processing along specific dimensions). This segmentation reduces computational complexity by avoiding full two-dimensional convolutions while maintaining essential processing capabilities through the combination of multiple simpler convolution operations.
Solution Approach 2:
The patent employs one-dimensional convolutions instead of traditional two-dimensional convolutions in certain stages of processing. This dimensional reduction significantly decreases computational complexity while the multi-scale framework compensates for any loss in processing capability by applying these simplified convolutions at multiple resolution levels, achieving a balance between efficiency and effectiveness.
Data Source
AI summary
The present disclosure relates to methods, apparatus, systems, and non-transitory computer-readable storage media for training and using a multi-scale machine learning model for the enhancement of compressed video. According to some examples, a computer-implemented method includes receiving a video at a content delivery service; performing an encode on a frame of the video by the content delivery service that converts the frame from a pixel domain to a transform domain and back to the pixel domain to generate first pixel values and a first residual for a block of the frame at a first resolution; generating a first set of features, by a machine learning model of the content delivery service, for an input, at a first resolution, of the first pixel values and the first residual of the block; generating a second set of features, by the machine learning model of the content delivery service, for an input, at a second lower resolution, of second pixel values and a second residual of the block; upsampling the second set of features to the first resolution to generate an upsampled second set of features; generating a modified version of the frame based on the first set of features and the upsampled second set of features; and transmitting the modified version of the frame to a frame buffer or from the content delivery service to a viewer device.