Face Swapping Framework for Identity-Preserving Real-Time Conversion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deepfake models fail to maintain the identity of a source image while accurately mimicking the pose and attributes of a target image, requiring expensive data collection and long-term learning.
Innovation Solution
A lightweight one-step face swapping framework that encodes the source image for identity features and the target image for pose features, using a triple adaptive normalization (TAN) decoder to integrate these features and generate a face-conversion image with the target's attributes, employing data augmentation and a switch-test strategy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If conventional deepfake models are used to learn source images and target images, then face conversion can be generated, but the model fails to maintain source identity and target pose/attributes accurately
Solution Approach 1:
The patent segments the face conversion task into separate feature extraction modules: an identity encoder that extracts source image identity features, and a pose network that extracts target image pose features. This segmentation allows each module to specialize in its specific function, improving overall accuracy of both identity maintenance and pose/attribute preservation in the generated face conversion images.
Solution Approach 2:
The patent introduces an intermediary attribute image that contains both identity and pose information. This intermediary serves as a mediator between the source image (providing identity) and target image (providing pose), allowing the system to transfer identity features from the source while preserving pose and attributes from the target through the attribute image's combined feature representation.
2Reliability
If conventional deepfake models perform face conversion after learning large amounts of data, then more comprehensive learning is achieved, but data collection costs and learning time become expensive and lengthy
Solution Approach 1:
The patent performs preliminary action by pre-extracting and storing identity features from source images using the identity encoder, and pre-extracting pose features from target images using the pose network. These pre-computed features can be directly utilized during face conversion without requiring the model to re-learn from large datasets, significantly reducing both data collection costs and learning time while maintaining comprehensive feature representation.
Solution Approach 2:
The patent creates a simplified feature representation system that copies only the essential identity and pose information into the attribute image, rather than copying entire source and target images. This selective copying reduces the amount of data that needs to be processed and stored, lowering data collection costs and accelerating the learning process while preserving the necessary facial characteristics for accurate conversion.
3Manufacturing precision
If existing face swapping frameworks are used, then face conversion can be achieved, but real-time generation capability and computational efficiency are insufficient
Solution Approach 1:
The patent extracts only the essential features needed for face conversion: identity features from the source image and pose features from the target image. By taking out and utilizing only these critical features rather than processing the entire images, the system reduces computational complexity and achieves real-time generation speed while maintaining high conversion fidelity.
Solution Approach 2:
The patent changes the parameter representation by transforming images into feature vectors and attribute images with specific dimensional characteristics. This parameter transformation enables more efficient computation and faster processing, allowing real-time face conversion while preserving the essential visual fidelity through the carefully designed feature extraction and integration mechanisms.
Data Source
AI summary
The present invention relates to an image processing device comprising a processor, which uses, when a source image and a target image are input, the source image and the target image so as to generate a face-conversion image, wherein the processor encodes the source image so as to extract an identity feature, encodes the target image so as to extract a target code, decodes the target code so as to extract a pose feature, and integrates the identify feature, the target code and the pose feature, and includes a face swapping framework, which uses an attribute image in which the size of the target image has been adjusted, so as to generate a face-conversion image in which an attribute feature of the target image is reflected.