Video Compression Using Latent Residuals and ML Image Codecs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video codecs face challenges in meeting future demands due to increased spatial resolution, frame rate, and color depth of videos, and training machine learning models like GANs for video compression is complex and challenging.
Innovation Solution
A framework using knowledge distillation and latent space residuals with adversarially trained image compression ML models to build a video compression codec that leverages hallucination capacity, enabling visually pleasing images at low bit-rates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing engineered codecs are used to meet future video demands, then current video compression performance is maintained, but the ability to handle increased spatial resolution, frame rate, and color depth is limited
Solution Approach 1:
The patent replaces traditional engineered video coding mechanisms with machine learning models. Specifically, it uses trained ML models to perform video compression tasks that were previously handled by conventional codecs, enabling the system to adapt to future video demands while maintaining or improving compression performance.
Solution Approach 2:
The patent changes the fundamental parameters of video compression by transitioning from fixed engineered algorithms to flexible machine learning models with learnable parameters. This allows the system to optimize compression for varying spatial resolutions, frame rates, and color depths through training on diverse video data.
2Productivity
If machine learning models like GANs are trained for video compression, then improved compression performance and flexibility are achieved, but training complexity increases due to alternating minimization and maximization steps
Solution Approach 1:
The patent segments the video compression task into separate processing stages handled by different components of the ML model architecture. This segmentation allows the training process to break down the complex alternating minimization-maximization steps into more manageable sub-tasks, reducing overall training complexity.
Solution Approach 2:
The patent introduces intermediary components in the ML model architecture that facilitate the training process. These intermediaries help bridge the minimization and maximization steps, making the training process more stable and less complex by providing intermediate representations and gradients.
3Loss of substance
If machine learning models infer visual details to reduce data transmission, then data transmission costs are reduced, but training challenges increase due to temporal domain complexity
Solution Approach 1:
The patent performs preliminary actions during the training phase by pre-training ML models on large datasets that include temporal information. This preliminary training allows the models to learn temporal patterns and relationships beforehand, reducing the complexity of training when temporal domain considerations are introduced during deployment.
Solution Approach 2:
The patent addresses temporal domain complexity by introducing additional dimensional considerations in the ML model architecture. It processes video data in multiple dimensions (spatial and temporal) simultaneously, allowing the model to infer visual details across time while managing training complexity through dimensional decomposition and specialized temporal processing layers.
Data Source
AI summary
A system processing hardware executes a machine learning (ML) model-based video compression encoder to receive uncompressed video content and corresponding motion compensated video content, compare the uncompressed and motion compensated video content to identify an image space residual, transform the image space residual to a latent space representation of the uncompressed video content, and transform, using a trained image compression ML model, the motion compensated video content to a latent space representation of the motion compensated video content. The ML model-based video compression encoder further encodes the latent space representation of the image space residual to produce an encoded latent residual, encodes, using the trained image compression ML model, the latent space representation of the motion compensated video content to produce an encoded latent video content, and generates, using the encoded latent residual and the encoded latent video content, a compressed video content corresponding to the uncompressed video content.