Learned Video Transforms for Nonlinear Residual Coding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video coding techniques using linear transforms are sub-optimal for capturing higher-order dependencies and non-linearities in real-world images and videos, leading to inefficient compression.
Innovation Solution
Employing machine-learning models, specifically convolutional neural networks, to perform transform and inverse transforms that learn linear or non-linear models to better capture higher-order statistics in residual blocks, replacing traditional transform, quantization, and entropy coding stages in image or video codecs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If machine-learning models are used to perform transform and inverse transforms, then compression efficiency is improved, but device complexity increases
Solution Approach 1:
The patent replaces traditional mechanical linear transform systems with machine-learning-based neural network transforms. The neural networks learn non-linear transformations from data that better capture higher-order dependencies in video content, substituting the rigid mechanical transform process with an adaptive learning-based approach that achieves superior compression efficiency.
Solution Approach 2:
The patent changes the transformation parameters from fixed linear transform matrices to learnable neural network parameters. By training neural networks on large datasets of video content, the system adapts transform parameters to capture non-linear patterns and higher-order statistics, enabling the system to adjust compression characteristics dynamically based on content characteristics.
2Device complexity
If traditional linear transforms are used, then device complexity is reduced, but compression efficiency deteriorates
Solution Approach 1:
The patent replaces simple linear transform mechanisms with complex machine-learning-based neural network transforms. This substitution enables the system to capture higher-order dependencies and non-linear relationships in video data that linear transforms cannot represent, thereby improving compression efficiency despite increased algorithmic complexity.
Solution Approach 2:
The patent employs learned transform models that are trained copies of optimal transformations derived from large datasets. Instead of using simple fixed transforms, the system uses neural network models that have learned optimal transformation patterns from training data, effectively copying the complexities of optimal transforms without requiring manual design of complex algorithms.
3Productivity
If machine-learning models are used to capture higher-order dependencies, then compression efficiency is improved, but computing resources are increased
Solution Approach 1:
The patent performs preliminary training of neural network transform models using large datasets of video content. During this preliminary training phase, the system learns optimal transformations and stores them in the neural network parameters. This preliminary action enables the system to achieve high compression efficiency during actual operation without requiring computationally intensive real-time learning, thus reducing ongoing computing resource requirements.
Solution Approach 2:
The patent segments the compression system into separate functional components: a trained neural network transform module for capturing higher-order dependencies, and a standard entropy coding module for efficient bitstream generation. By segmenting the system this way, the computationally intensive neural network processing is separated from the efficient entropy coding, allowing the system to leverage the pre-computed neural network transformations to reduce overall computing requirements during operation.
Data Source
AI summary
Decoding a current block includes receiving a compressed bitstream. A transform block of transform coefficients is decoded from the compressed bitstream. The transform coefficients are in a transform domain. The transform block is input to a machine-learning model to obtain a residual block that is in a pixel domain. The residual block is used to reconstruct the current block. Encoding a current block includes receiving a current residual block. The current residual block and a specified rate-distortion parameter are input to a machine-learning model to obtain a quantized transform block. The quantized transform block is entropy encoded into a compressed bitstream.