Conditional Cycle-Consistent GAN for Scalable Virtual Makeup
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing virtual makeup applications face challenges in scaling to a large number of products while maintaining realism due to limitations in landmark detection accuracy, manual tuning of color and shine, and identity preservation, and require reference images for each product, making them inefficient for online use.
Innovation Solution
A conditional cycle-consistent generative image-to-image translation model using convolutional conditional batch normalization and output projection to learn a continuous makeup space from swatch images, enabling scalable and realistic makeup rendering without reference images.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If classical machine learning or computer vision techniques are used for virtual makeup applications, then the application can detect facial landmarks and render output images, but the solution cannot easily scale to a large amount of products while maintaining realism due to limitations in landmark detection accuracy, manual tuning requirements, and identity preservation
Solution Approach 1:
The patent replaces the mechanical two-step process of landmark detection followed by traditional graphics rendering with a neural network-based image-to-image translation system. This substitution enables automatic learning of makeup application patterns from training data, eliminating the need for manual parameter tuning and landmark detection, thereby achieving both scalability to large product catalogues and high realism in makeup rendering
Solution Approach 2:
The patent transforms the rigid parameter-based rendering approach into a flexible neural network approach where makeup properties (color, shine, texture) are learned as continuous parameters from training data. The system uses conditional generative adversarial networks to map input images to output images with desired makeup properties, allowing smooth interpolation between different makeup styles and products without manual parameter adjustment
2Manufacturing precision
If reference images are required for each product in virtual try-on features, then the model can learn complex lipstick properties and render them accurately, but the solution becomes problematic for realistic use cases such as online store applications due to the need for extensive reference images and manual tuning
Solution Approach 1:
The patent enables the system to automatically learn and encode makeup properties directly from product images without requiring reference images of the makeup applied to faces. The neural network self-learns the mapping between product images and application results through training, eliminating the need for manual collection and tuning of reference images for each new product
Solution Approach 2:
The patent performs preliminary encoding of makeup properties from product images during an offline training phase. The system pre-learns the relationship between product characteristics and application effects, storing this knowledge in the neural network weights. This preliminary action enables rapid, accurate rendering of new products without requiring reference images at the time of use
3Adaptability or versatility
If manual tuning of color, shine, and blending parameters is performed based on estimates of product properties, lighting, and skin tone, then the rendering can be customized for different products, but the process becomes time intensive and inaccurate, scaling poorly as the product catalogue increases
Solution Approach 1:
The patent replaces manual parameter tuning with automatic neural network-based parameter learning. The system learns optimal color, shine, and blending parameters directly from training data, automatically adapting to different product properties, lighting conditions, and skin tones without human intervention. This substitution reduces tuning time from hours to minutes while improving accuracy
Solution Approach 2:
The patent implements feedback mechanisms where the neural network continuously learns from training data, adjusting its internal parameters to minimize the difference between rendered and actual makeup appearances. This feedback loop enables automatic optimization of customization parameters across the entire product catalogue, eliminating repetitive manual tuning
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
Systems and methods relate to a network model to apply an effect to an image such as an augmented reality effect (e.g. makeup, hair, nail, etc.). The network model uses a conditional cycle-consistent generative image-to-image translation model to translate images from a first domain space where the effect is not applied and to a second continuous domain space where the effect is applied. In order to render arbitrary effects (e.g. lipsticks) not seen at training time, the effect's space is represented as a continuous domain (e.g. a conditional variable vector) learned by encoding simple swatch images of the effect, such as are available as product swatches, as well as a null effect. The model is trained end-to-end in an unsupervised fashion. To condition a generator of the model, convolutional conditional batch normalization (CCBN) is used to apply the vector encoding the reference swatch images that represent the makeup properties.