Neural Super-Resolution Using DDGAN and Historical Frame Conditioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing super-resolution models face tradeoffs between image quality, inference speed, and training stability, with diffusion models offering high quality and stability but slow inference, and GANs providing fast inference but reduced stability, limiting their suitability for applications like video rendering.
Innovation Solution
A temporal super-resolution model combining deep learning super sampling (DLSS) and denoising diffusion generative adversarial networks (DDGANs) is developed, incorporating a reference image and historical frames to condition the generator and discriminator, reducing training instability and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If diffusion models are used for super-resolution, then image quality and training stability are improved, but inference speed deteriorates
Solution Approach 1:
The patent combines diffusion models and GANs into a unified framework where the diffusion model generates high-quality super-resolved images and the GAN component accelerates inference through adversarial training, resolving the contradiction between image quality and inference speed
Solution Approach 2:
The system dynamically switches between different processing modes: using diffusion steps when high quality is prioritized and GAN-based shortcuts when speed is prioritized, allowing adaptive balance between image quality and inference speed based on application requirements
2Speed
If GANs are used for super-resolution, then inference speed is improved, but training stability deteriorates
Solution Approach 1:
The patent merges diffusion models (known for training stability) with GANs (known for fast inference) into a hybrid architecture, where the diffusion component provides stable training signals and the GAN component delivers fast inference, simultaneously addressing both concerns
3Manufacturing precision
If existing super-resolution models are used, then image quality may be improved, but latency increases
Solution Approach 1:
The system performs preliminary processing by generating low-resolution intermediate images that are then upsampled using the hybrid model, reducing the computational burden and latency while maintaining high image quality in the final output
Data Source
AI summary
Approaches presented herein provide systems and methods for content generation systems that incorporate a denoising diffusion generative adversarial network (DDGAN) into a content generation pipeline. A generator associated with the DDGAN may be conditioned on a set of input images that include at least a noisy image from a diffusion engine, an upsampled low resolution image, and a historical image. The generator may be used to generate an output image having one or more properties that are different from a content engine. Weights for the generator may be determined during a training process that includes a discriminator that evaluates at least the noisy image, the upsampled low resolution image, and a noised image produced from the output image.