Neural Network Degradation Map Extraction for Spatially Varying Noise
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep Neural Network-based Super-Resolution methods suffer from overfitting and decreased performance when applied to images with different degradations, particularly due to the assumption of uniformly distributed noise, which does not account for spatially varying noise present in real-world images acquired in photon-limited situations.
Innovation Solution
A method is developed to train a neural network to extract a degradation map from degraded images by generating paired LR/HR images with spatially varying noise, using a degradation pipeline that synthesizes degraded images with noise strength varying across the image, and a super-resolution model that performs pixel-wise feature modulation using the extracted degradation map.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If uniform noise degradation is assumed in training, then training data generation is simple, but performance degrades on real-world images with spatially varying noise
Solution Approach 1:
The patent applies local quality by introducing spatially varying noise patterns in the degradation pipeline, where different regions of the image have different noise characteristics. This is achieved by modulating the noise strength according to local image properties such as brightness, allowing the model to learn region-specific degradation patterns that match real-world conditions.
Solution Approach 2:
The patent changes the noise parameter from uniform to spatially varying by introducing position-dependent noise standard deviations. The degradation pipeline dynamically adjusts noise parameters based on local image characteristics, transforming the degradation model from a global uniform assumption to a local adaptive one, thereby improving generalization to real-world images.
2Productivity
If deep neural networks are trained on synthetically degraded images, then training is efficient, but overfitting occurs leading to decreased performance on images with different degradations
Solution Approach 1:
The patent enhances adaptability by dynamically changing noise parameters during the degradation process. Instead of using fixed uniform noise, the system varies noise intensity and characteristics based on local image properties, creating a more diverse and realistic training distribution that improves generalization to unseen degradation patterns.
Solution Approach 2:
The patent introduces dynamics into the degradation pipeline by making noise characteristics adaptive rather than static. The degradation model dynamically adjusts noise parameters based on local image conditions, allowing the training data to reflect the variability and complexity of real-world degradations, thereby reducing overfitting and improving versatility.
3Reliability
If spatially varying noise is incorporated in the degradation pipeline, then image quality improves on real-world images, but device complexity increases
Solution Approach 1:
The patent applies segmentation by dividing the image processing into distinct stages: feature extraction, local noise characterization, and adaptive degradation application. The degradation pipeline is segmented into modules that independently handle different aspects of spatially varying noise, making the complex process more manageable and computationally efficient.
Solution Approach 2:
The patent introduces an intermediary degradation pipeline that acts as a bridge between the simple uniform noise model and the complex real-world degradation. This intermediate model uses adaptive noise parameters modulated by local image features, providing a computationally efficient approximation of real-world degradations without requiring full complexity of physical imaging processes.
Data Source
AI summary
Training a neural network to extract a degradation map from a degraded image comprises generating training data comprising pairs of images, each pair of images comprising a clean source image and a degraded source image by, for each clean source image, generating a corresponding noisy image by adding spatially invariant noise to the clean source image, and blending the noisy image with the clean source image according to varying intensity levels defined by a spatially variant mask to obtain the degraded image. The training data is used to train the neural network by inputting each degraded source image to the neural network and extracting a degradation map from the degraded source image such that when the degradation map is applied to its corresponding clean source image the loss between the degraded source image and its corresponding clean source image after the degradation map is applied is minimised.