Neural Network Video Downsampling for Quality Retention

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional video encoding pipelines suffer from suboptimal visual quality in reconstructed videos due to inadequate downsampling algorithms that do not minimize the diminution in resolution, leading to higher bitrates and unnecessary visual quality impairments.

Innovation Solution

A method involving training a neural network with residual blocks to downsample images in a video encoding pipeline, allowing for the reduction of reconstruction errors and improvement in visual quality by learning to mitigate the effects of resolution reduction, enabling the use of non-integer downsampling factors and reducing the number of bits required for a given target visual quality.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Manufacturing precision

If conventional downsampling algorithms are used in video encoding pipelines, then the encoding process is simple and robust across different signal types, but the visual quality of reconstructed videos deteriorates due to inadequate resolution reduction minimization

Engineering Contradiction:

Improvevisual quality of reconstructed videoVSAvoidcomplexity of downsampling algorithm

Core Design Contradiction:

Manufacturing precisionVSDevice complexity

Solution Approach 1:

The patent replaces conventional mechanical downsampling algorithms with a neural network-based system. The neural network learns optimal downsampling operations during training by minimizing reconstruction error, substituting traditional signal processing methods with a data-driven approach that adapts to minimize visual quality degradation.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The neural network is trained in advance on a large dataset of image patches to learn optimal downsampling operations. This preliminary training phase allows the network to capture patterns and relationships that minimize reconstruction error, so that during actual video encoding, the pre-trained network can perform high-quality downsampling without requiring complex real-time computations.

Inventive Principle:

Principle #10Preliminary action

2Manufacturing precision

If conventional downsampling algorithms are used, then the encoding pipeline is computationally efficient, but the bitrate required to achieve a target visual quality increases unnecessarily

Engineering Contradiction:

Improvetarget visual quality levelVSAvoidbitrate required for encoding

Core Design Contradiction:

Manufacturing precisionVSQuantity of substance

Solution Approach 1:

The patent replaces conventional downsampling algorithms with a neural network-based system. The neural network learns optimal downsampling operations during training by minimizing reconstruction error, substituting traditional signal processing methods with a data-driven approach that adapts to minimize visual quality degradation.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The neural network is trained using feedback from reconstruction error calculations. During training, downsampled images are upsampled back to the original resolution and compared with the original images. The reconstruction error serves as feedback that guides the optimization of network parameters, enabling the network to learn downsampling operations that minimize quality loss and reduce the bitrate needed to achieve target visual quality.

Inventive Principle:

Principle #23Feedback

3Manufacturing precision

If standard downsampling algorithms are used, then the encoding process is straightforward and works robustly across different signal types, but visual quality impairments and artifacts increase in the reconstructed video

Engineering Contradiction:

Improvevisual quality impairmentsVSAvoidcomplexity of encoding pipeline

Core Design Contradiction:

Manufacturing precisionVSDevice complexity

Solution Approach 1:

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent changes the fundamental parameter of how downsampling is performed by transitioning from fixed algorithmic operations to learned neural network operations. The neural network parameters are optimized during training to minimize reconstruction error, enabling adaptive downsampling that reduces visual quality impairments while managing computational complexity through efficient network architectures and training approaches.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS20240233076A1Machine learning techniques for video downsampling

Publication Date: 2024.07.11 NETFLIX INC

AI summary

In various embodiments, a training application trains a convolutional neural network to downsample images in a video encoding pipeline. The convolution neural network includes at least two residual blocks and is associated with a downsampling factor. The training application executes the convolutional neural network on a source image to generate a downsampled image. The training application then executes an upsampling algorithm on the downsampled image to generate a reconstructed image having the same resolution as the source image. The training application computes a reconstruction error based on the reconstructed image and the source image. The training application updates at least one parameter of the convolutional neural network based on the reconstruction error to generate a trained convolutional neural network. Advantageously, the trained convolution neural network can be implemented in a video encoding pipeline to mitigate visual quality reductions typically experienced with conventional video encoding pipelines that implement conventional downsampling techniques.