Scribble-Guided Makeup Transfer for Pixel-Level Face Editing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional image editing systems using machine learning models, such as GANs, lack control over the generated images, especially in complex makeup scenarios, and fail to provide fine-grained modifications.
Innovation Solution
An image editing system that uses a scribble image as input to specify the type and location of makeup, employing a modified U-Net architecture that concatenates the original and scribble images as different channels, allowing for pixel-level control over the generated image.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If conventional machine learning models (GANs) are used for makeup transfer, then automated image generation is achieved, but user control over specific regions is lost
Solution Approach 1:
A scribble image is introduced as an intermediary control mechanism between the user and the GAN model. The scribble image contains mask information that specifies which regions should receive makeup transfer, allowing users to control the process without directly manipulating complex model parameters. The encoder processes both the original image and scribble image together, using the scribble as a guiding signal for region-specific makeup application.
2Stability of the object's composition
If a fixed reference makeup is provided to the model, then consistent makeup style is achieved, but fine-grained modification capability is reduced
Solution Approach 1:
The control mechanism is segmented into two independent components: the reference makeup image that provides overall style guidance, and the scribble image that provides region-specific control. This segmentation allows users to modify makeup in specific regions (eyes, lips, cheeks) independently while maintaining the overall makeup style defined by the reference image. The encoder processes these segmented inputs separately before combining them for generation.
3Device complexity
If the model processes only the original image and reference makeup, then processing simplicity is maintained, but pixel-level control is insufficient
Solution Approach 1:
The control precision is enhanced by adding a new dimension - the scribble image - which provides spatial mask information at the pixel level. Instead of relying solely on the reference makeup image for all control decisions, the system now operates in an expanded input space that includes both the reference makeup and the scribble mask. This additional dimensional input allows the encoder to distinguish between regions that should receive makeup and regions that should remain unchanged, achieving pixel-level control without fundamentally complicating the overall model architecture.
Data Source
AI summary
Systems and methods for image processing are described. Embodiments of the present disclosure identify an original image depicting a face, identify a scribble image including a mask that indicates a portion of the original image for adding makeup to the face, and generate a target image depicting the face using a machine learning model based on the original image and the scribble image, where the target image includes the makeup in the portion indicated by the scribble image.