Face Swap Model Training via Expression Identity Feature Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current face swap algorithms struggle to retain diverse expressions effectively, particularly in scenarios with complex expressions like pouting, eyes closing, single eye blinking, and getting angry, resulting in poor face swap effects.
Innovation Solution
A method for training a face swap model involves acquiring a sample triplet consisting of a source face image, a template image, and a reference image, and then using a generator network to perform encoding and decoding operations while fusing expression and identity features to generate a swapped face image.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional face swap algorithms are used, then common expression scenarios (e.g., smiling) can achieve satisfactory effects, but diverse expression scenarios (e.g., pouting, eyes closing, single eye blinking, getting angry) result in poor face swap effects
Solution Approach 1:
The patent segments the face swap process into independent feature extraction modules: expression feature extraction from template image, identity feature extraction from source face image, and attribute feature extraction. This segmentation allows each module to specialize in capturing specific facial characteristics, enabling accurate retention of diverse expressions while maintaining identity similarity, thereby resolving the contradiction between expression retention capability and face swap accuracy.
Solution Approach 2:
The patent merges multiple feature types (expression features, identity features, and attribute features) into a unified feature representation that is fed into the generator network. This combination ensures that the generated face swap image simultaneously preserves diverse expressions and maintains identity accuracy, addressing the contradiction by integrating multiple perspectives into a single solution.
2Adaptability or versatility
If face swap model is trained with limited expression diversity, then training process is simpler, but the model cannot retain complex expressions accurately
Solution Approach 1:
The patent performs preliminary feature extraction and representation learning during the training phase using diverse expression data. By pre-extracting and storing expression features, identity features, and attribute features in a structured manner, the model is prepared to handle complex expressions during inference without requiring complex real-time processing, thus reducing the apparent complexity while maintaining high adaptability.
Solution Approach 2:
The patent utilizes parameter changes in the generator network and discriminator network during training to adapt to different expression types. The networks learn to adjust their internal parameters based on the input features, enabling them to accurately render complex expressions. This parameter adaptation mechanism allows the model to handle diverse expressions without proportionally increasing training data complexity.
3Adaptability or versatility
If expression features are prioritized in face swap, then expression retention improves, but identity similarity may be compromised
Solution Approach 1:
The patent segments the feature extraction process into separate expression feature extraction and identity feature extraction modules. This segmentation allows independent optimization of expression retention and identity similarity, as each module can focus on its specific objective without compromising the other, thereby resolving the contradiction between expression retention and identity similarity.
Solution Approach 2:
The patent combines expression features and identity features into a comprehensive feature representation that is processed together by the generator network. This merging ensures that both expression retention and identity similarity are simultaneously optimized, as the network learns to balance both aspects in the feature space, producing face swap images that satisfy both requirements.
Data Source
AI summary
A method for training a face swap model includes concatenating an expression feature of a template image and an identity feature of a source face image to obtain a combined feature; performing encoding based on the source face image and the template image to obtain an encoding feature required for face swap; fusing the encoding feature and the combined feature to obtain a fused feature; performing decoding based on the fused feature to obtain a swapped face image; respectively predicting image attribute discrimination results of the swapped face image and a reference image by using a discriminator network of the face swap model; and calculating a difference between an expression feature of the swapped face image and the template image, calculating a difference between an identity feature of the swapped face image and the source face image, and updating the generator network and the discriminator network.