GAN Lighting Transfer with Conditional Discriminator
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing portrait lighting transfer techniques struggle to produce realistic images with drastically different lighting conditions and fail to accurately match multi-colored lighting, often losing facial details and producing unrealistic shadows.
Innovation Solution
A generative adversarial network (GAN) is trained using a conditional discriminative network to transfer lighting from a reference image to a source image, incorporating noisy lighting conditions and compensating weights based on segmentation, landmark localization, and facial identity losses to retain facial details.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Illumination intensity
If a prior GAN model with unconditional discriminator is used to relight an image, then the photorealism of the generated image can be improved, but the correctness of the lighting conditions cannot be ensured simultaneously
Solution Approach 1:
The discriminator is segmented into two distinct components: an unconditional discriminator that evaluates photorealism and a conditional discriminator that evaluates lighting condition correctness. This segmentation allows each discriminator to specialize in one aspect, resolving the contradiction between maintaining photorealism and ensuring lighting accuracy.
Solution Approach 2:
A lighting encoder is introduced as an intermediary component that extracts lighting conditions from reference images and provides them as conditional inputs to the conditional discriminator. This intermediary enables the system to accurately assess and transfer lighting conditions without compromising the photorealism evaluation performed by the unconditional discriminator.
2Illumination intensity
If face normal is used to perform relighting, then the lighting on certain parts of the face can be adjusted, but realistic shadows cannot be produced
Solution Approach 1:
The traditional face normal-based relighting mechanism is replaced with a deep learning-based GAN system. The generative network learns the complex mapping between lighting conditions and image appearance, automatically producing realistic shadows and lighting effects without relying on explicit geometric calculations or face normals.
3Adaptability or versatility
If prior image lighting transfer techniques are used, then the lighting can be transferred, but multi-colored lighting cannot be supported
Solution Approach 1:
The GAN-based lighting transfer system is designed with universal capability to handle various lighting conditions including multi-colored lighting. The lighting encoder and generative network are trained to recognize and reproduce different lighting types (single-color, multi-colored, directional, ambient), making the system versatile while maintaining accurate lighting transfer across all scenarios.
Data Source
AI summary
A method for training a GAN to transfer lighting from a reference image to a source image includes: receiving the source image and the reference image; generating a lighting vector from the reference image; applying features of the source image and the lighting vector to a generative network of the GAN to create a generated image; applying features of the reference image and the lighting vector to a discriminative network of the GAN to update weights of the discriminative network; and applying features of the generated image and the lighting vector to the discriminative network to update weights of the generative network.