Corruption Mimicking Network for Robust Image Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning techniques for projecting images onto an image manifold, such as Generative Adversarial Networks (GANs) and projected gradient descent (PGD), fail when faced with small distribution shifts or corruptions like missing pixels, rotation, or scale, especially when only a single corrupted sample is available at test time.
Innovation Solution
The Corruption Mimicking (CM) system iteratively trains a corruption mimicking network (CMN) to generate corrupted images, updates latent vectors based on differences between observed and generated images, and applies a generator to produce uncorrupted images, effectively mimicking the corruption process to estimate the uncorrupted version of the image.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If PGD or GAN with explicit encoder is used for image projection, then projection effectiveness is improved, but reliability deteriorates under distribution shifts and corruptions
Solution Approach 1:
The system pre-trains a generator on clean images to learn the image manifold, then at test time uses a corruption mimicking network to generate corrupted versions of clean images. This preliminary training allows the system to handle corruptions and distribution shifts without requiring retraining, resolving the contradiction between projection effectiveness and robustness to corruptions
Solution Approach 2:
The corruption mimicking network creates synthetic corrupted versions of clean images that replicate the corruption patterns observed in the input. By copying and generating these corrupted versions, the system can project onto the image manifold while accounting for corruptions, maintaining both projection effectiveness and robustness
2Adaptability or versatility
If unsupervised image-to-image translation is used to model distributional shifts, then adaptability is improved, but device complexity increases due to requirement for large paired samples
Solution Approach 1:
The system extracts only the essential corruption patterns from the input images using a corruption mimicking network, rather than requiring large paired datasets. By taking out and modeling only the necessary corruption information, the system achieves adaptability to distributional shifts without the complexity of processing large paired samples
Solution Approach 2:
The system uses the corrupted input images themselves to train the corruption mimicking network, eliminating the need for external paired training data. The network learns corruption patterns by processing the actual corrupted images, providing self-service adaptation without requiring complex paired sample collections
Data Source
AI summary
Methods and systems are provided to generate an uncorrupted version of an image given an observed image that is a corrupted version of the image. In some embodiments, a corruption mimicking (“CM”) system iteratively trains a corruption mimicking network (“CMN”) to generate corrupted images given modeled images, updates latent vectors based on differences between the corrupted images and observed images, and applies a generator to the latent vectors to generate modeled images. The training, updating, and applying are performed until modeled images that are input to the CMN result in corrupted images that approximate the observed images. Because the CMN is trained to mimic the corruption of the observed images, the final modeled images represented the uncorrupted version of the observed images.