Multi-Encoder Face Swapping for Facial Expression and Gaze Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning face swapping techniques lack control over the swapped image, particularly in maintaining accurate eye gaze direction and facial expressions, making manual corrections time-consuming and costly.
Innovation Solution
A progressively trained, multi-way neural network with multiple encoders that encode different parts of an image into separate latent space vectors, allowing for high-resolution output images and enabling precise control over facial expressions and gaze direction through a latent space editor.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single encoder is used to encode the entire input image, then the device complexity is reduced, but the manufacturing precision of facial expressions and gaze direction deteriorates
Solution Approach 1:
The patent divides the face image into multiple regions (eyes, eyebrows, nose, mouth, cheeks) and assigns separate encoders to each region. This segmentation allows each encoder to specialize in capturing specific facial features, thereby improving the precision of facial expression and gaze direction representation while managing complexity through modular architecture.
2Manufacturing precision
If multiple encoders are used to encode different parts of the image, then the manufacturing precision of facial expressions improves, but the device complexity increases
Solution Approach 1:
The patent merges the outputs of multiple region-specific encoders into a unified latent space representation. By combining the encoded features from eyes, eyebrows, nose, mouth, and cheeks encoders, the system achieves comprehensive facial expression capture while managing complexity through integrated processing in the latent space.
Solution Approach 2:
The patent designs encoders that can serve multiple functions: each region-specific encoder not only captures local facial features but also contributes to the overall facial expression representation. The modular encoder architecture allows for flexible configuration and can be adapted to different face swapping requirements, providing multi-functionality that balances precision and complexity.
3Ease of operation
If deep learning face swapping is performed without control mechanisms, then the ease of operation is improved, but the measurement precision of eye gaze direction deteriorates
Solution Approach 1:
The patent implements feedback mechanisms by encoding eye region features separately and using this information to adjust and control the gaze direction in the output image. The system provides feedback loops that allow operators to control eye gaze direction while maintaining ease of operation through automated processing of the feedback information.
Solution Approach 2:
The patent enables control over eye gaze direction by changing specific parameters in the latent space representation. By manipulating the encoded eye region features and adjusting gaze-related parameters, the system achieves precise control over eye direction while keeping the overall face swapping process easy to operate through parameter-based control rather than complex manual adjustments.
Data Source
AI summary
A computer-implemented method of changing a face within an output image or video frame includes: receiving an input image that includes a face presenting a facial expression in a pose; separately encoding different portions of the image by, for each separately encoded portion, generating a latent space point of the portion, thereby generating a plurality of multi-dimensional vectors where each multi-dimensional vector is an encoded representation of a different portion of the input image; concatenating the plurality of multi-dimensional vectors into a combined latent space vector; and decoding the combined latent space vector to generate the output image in accordance with a desired facial identity but with the facial expression and pose of the face in the input image.