Transformer Discriminator for Low-Bitrate Generative Video Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional video codecs are handcrafted and not end-to-end optimized, leading to decoded frames suffering from blur at low bit rates due to the use of mean-squared error as the loss function, and existing GAN-based video compression methods lack the ability to model long-distance dependencies and extract global features effectively.
Innovation Solution
Employ a transformer-based discriminator in a generative adversarial network (GAN) for video compression, incorporating a new generator loss function that constrains collective pixel distortion, entropy, and perceptual quality, while using a discriminator-dependent feature loss to improve decoded frame quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If traditional video codecs use mean-squared error as the loss function, then decoding stability is improved, but decoded frame quality deteriorates with blur at low bit rates
Solution Approach 1:
The patent changes the loss function parameter from mean-squared error to a combination of perceptual loss and adversarial loss. This parameter change allows the system to prioritize perceptual quality over mathematical stability, resolving the contradiction by accepting controlled instability in training convergence to achieve superior decoded frame quality that aligns with human perception.
2Manufacturing precision
If GAN-based video compression is used, then decoded frame quality is improved, but the ability to model long-distance dependencies deteriorates
Solution Approach 1:
The patent merges transformer architecture with GAN-based video compression by integrating the transformer discriminator into the adversarial training framework. This combination allows the system to leverage the transformer's global attention mechanism for modeling long-distance dependencies while maintaining the GAN's ability to generate high-quality decoded frames, thus resolving the contradiction between quality improvement and modeling capability.
3Ease of manufacture
If handcrafted video codecs are used, then implementation simplicity is improved, but end-to-end optimization capability deteriorates
Solution Approach 1:
The patent replaces handcrafted mechanical video coding operations with a learned neural network system. Instead of manually designed algorithms, the system uses end-to-end trainable deep learning models that automatically optimize compression parameters through adversarial training, achieving superior adaptability while maintaining implementation feasibility through modular architecture design.
Data Source
AI summary
A method, an apparatus, and a non-transitory computer-readable storage medium for video compression using a generative adversarial network (GAN) are provided. The method includes obtaining, by a generator of the GAN, a reconstructed target frame based on a reference frame and a raw target frame to be reconstructed; concatenating, by a transformer-based discriminator of the GAN, the reference frame, the raw target frame and the reconstructed target frame to obtain a paired data; determining, by the transformer-based discriminator of the GAN, whether the paired data is real or fake to guide reconstruction of the raw target frame; and determining a generator loss and a transformer-based discriminator loss, and performing gradient back propagation and updating network parameters of the GAN based on the generator loss and the transformer-based discriminator loss.