High Resolution Face Swapping via Hierarchical Encoding and Segmented Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current face swapping technologies are limited to 256×256 resolution due to information loss, unstable training of Generative Adversarial Networks (GANs), and hardware constraints, resulting in unrealistic and detail-lacking swapped faces.
Innovation Solution
A video generation method and system for high resolution face swapping, which includes a face feature encoder for hierarchical encoding, a face feature exchanger for piecewise nonlinear optimization, a face generator for generating initial swapped faces, and a face fuser for semantic segmentation-based fusion with the target face, using pre-trained models and phased training to achieve megapixel-level resolution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If GAN-based models are used for face swapping, then vivid swapping images can be synthesized, but the resolution is limited to 256×256 due to information loss, unstable training, and hardware constraints
Solution Approach 1:
The patent divides the face swapping process into multiple stages: a first stage for generating low-resolution swapped face images and a second stage for enhancing to high resolution. The face feature encoder is also segmented into hierarchical encoding structures that process features at different resolution levels separately, preventing information loss by preserving details through multi-scale processing
Solution Approach 2:
The patent performs preliminary face feature extraction and encoding at lower resolutions before generating the final high-resolution output. The face feature encoder extracts features from source and target faces, and the face feature exchanger creates swapped features before the actual image generation, allowing for stable training and preventing information loss through progressive processing
2Reliability
If GAN-based models are used for face swapping, then vivid swapping images can be synthesized, but training becomes unstable
Solution Approach 1:
The training process is segmented into multiple phases: first training the face feature encoder and face generator separately, then training the face feature exchanger. This staged approach stabilizes training by reducing the complexity of each individual training step while maintaining the overall model capability for high-resolution face swapping
Solution Approach 2:
The patent performs preliminary training of the face feature encoder and face generator before introducing the face feature exchanger. This preliminary action establishes stable feature extraction and generation capabilities, which then serve as a foundation for training the more complex face feature exchanger, thereby stabilizing the overall training process
3Manufacturing precision
If GAN-based models are used for face swapping, then vivid swapping images can be synthesized, but hardware requirements become insufficient
Solution Approach 1:
The patent segments the image generation process into two stages: first generating at lower resolution with less computational demand, then enhancing to high resolution. This segmentation reduces the hardware requirements for each individual processing step while achieving the final high-resolution output, making the system more feasible for standard hardware configurations
Solution Approach 2:
The patent performs preliminary feature extraction and encoding at lower resolutions before generating high-resolution output images. This preliminary action reduces the computational burden during the main generation process, as the heavy feature extraction is performed at reduced resolution, thereby reducing hardware requirements while maintaining high-resolution output capability
Data Source
AI summary
A video generation method includes: obtaining a target face image and a source face image; extracting a feature of each of the source face image and the target face image through a face feature encoder, to obtain corresponding source feature codes and target feature codes; generating swapped face feature codes through a face feature exchanger according to the source feature codes and the target feature codes; generating an initial swapped face image through a face generator according to the swapped face feature codes; and fusing the initial swapped face image with the target face image through a face fuser, to obtain a final swapped face image. The face feature encoder performs hierarchical encoding on the face feature to reserve semantic details of a face, and the face feature exchanger performs further processing based on the hierarchical encoding, to obtain hierarchical encoding of a swapped face feature with semantic details.