Infrared Image Colorization Using Segmented Generative Adversarial Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional techniques for colorizing infrared images often result in blurry or inaccurately colored RGB images due to averaging multiple valid color candidates, leading to a need for improved methods that enhance accuracy and detail in infrared colorization.
Innovation Solution
The use of a conditional generative adversarial network (cGAN) to colorize infrared images by transferring color statistics from overlapping RGB images, modifying encoded representations of synthesized RGB images, and applying segmentation techniques to identify and color corresponding regions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If conventional end-to-end CNN techniques are used to map IR images to RGB images, then the colorization process can be automated, but the output images become blurry and lack detail due to averaging multiple valid color candidates
Solution Approach 1:
The patent segments the colorization task into multiple stages: first generating a preliminary RGB image from IR image using CNN, then applying segmentation algorithms to identify distinct regions, and finally applying different colorization methods to each segment. This segmentation approach allows the system to handle different regions with appropriate colorization techniques rather than applying a uniform averaging approach, thereby improving color accuracy and sharpness while maintaining automation.
Solution Approach 2:
The patent applies local quality by treating different regions of the image differently based on their characteristics. After segmentation, each region is colorized using methods optimized for its specific properties (e.g., skin tone preservation for facial regions, vegetation coloring for natural scenes). This localized approach prevents the blurring effect of global averaging while maintaining automated processing, thus resolving the contradiction between automation and precision.
2Productivity
If conventional CNN colorization techniques are used, then processing can be performed in a single pass, but color consistency deteriorates as different segments may receive different colors that do not match ground truth
Solution Approach 1:
The patent divides the image into semantic segments (e.g., skin, hair, eyes, vegetation, water) using segmentation algorithms. Each segment is then colorized independently using domain-specific knowledge and constraints, ensuring color consistency within and across segments. This segmented approach maintains processing efficiency while significantly improving color consistency and accuracy compared to single-pass conventional methods.
Solution Approach 2:
The patent incorporates feedback mechanisms where the colorization results are evaluated against ground truth data or consistency constraints, and adjustments are made iteratively. This feedback loop ensures that color assignments remain consistent across different segments and match expected ground truth, thereby improving reliability without substantially increasing processing time.
3Adaptability or versatility
If multiple valid color candidates are considered during colorization, then color diversity is increased, but image quality deteriorates due to averaging effects
Solution Approach 1:
The patent applies local quality by selecting and refining color candidates locally for each segmented region based on contextual information and ground truth constraints. Rather than averaging all valid color candidates globally, the system evaluates and selects appropriate colors for each local region, preserving sharpness and detail while maintaining color diversity where appropriate. This localized selection process resolves the contradiction between color diversity and image quality.
Solution Approach 2:
The patent changes parameters such as color temperature, saturation, and brightness locally for different regions based on their semantic identity and contextual constraints. By adjusting these parameters region-specifically rather than applying uniform transformations, the system maintains color diversity and adaptability while preventing the blurring effect of averaging, thus improving image sharpness and detail.
Data Source
AI summary
In various examples, infrared image data (e.g., frames of an infrared video feed) may be colorized by applying the infrared image data and/or a corresponding edge map to a generator of a generative adversarial network (GAN). The GAN may be trained with or without paired ground truth RGB and infrared (and/or edge map) images. In an example of the latter scenario, a first generator G(IR)→RGB and a second generator G(RGB)→IR may be trained in a first chain, their positions may be swapped in a second chain, and the second chain may be trained. In some embodiments, edges may be emphasized by weighting edge pixels (e.g., determined from a corresponding edge map) higher than non-edge pixels when backpropagating loss. After training, G(IR)→RGB may be used to generate RGB image data from infrared image data (and/or a corresponding edge map).