Real-Time Face Reenactment With Parametric Models and DNN Regions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing face reenactment technologies either produce non-photorealistic results or are too computationally intensive for real-time performance on mobile devices, and there is a lack of personalized advertising methods that effectively utilize user data to replace faces in advertisements.
Innovation Solution
A method and system utilizing a parametric face model and deep neural network (DNN) for real-time face reenactment on mobile devices, combined with a personalized video generation module to replace faces in advertisements based on user data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If 3D models and texture maps are used to represent facial geometry and appearance, then facial expression transfer accuracy is improved, but computational complexity and processing time increase
Solution Approach 1:
The facial representation is segmented into distinct components: 3D geometry models for structural information and texture maps for appearance information. This segmentation allows the system to process different facial attributes separately, improving transfer accuracy while managing computational complexity through modular processing of geometric and textural data
Solution Approach 2:
The system performs preliminary actions by pre-processing source and target facial images to extract 3D models and texture maps before the actual expression transfer. This preliminary extraction and registration of facial geometries prepares the data structures in advance, enabling more accurate expression mapping without requiring complex real-time computations during the transfer process
2Manufacturing precision
If detailed 3D facial models are extracted and registered, then expression mapping accuracy is improved, but processing time increases
Solution Approach 1:
The system performs preliminary extraction and registration of 3D facial models and texture maps before the actual expression transfer. By preparing these data structures in advance with proper alignment and coordinate system registration, the system achieves high mapping accuracy while reducing the computational burden during the actual expression transfer operation
Solution Approach 2:
The system creates simplified 3D model representations and texture map copies that capture essential facial geometry and appearance characteristics. These copied representations are used for expression mapping instead of processing full-resolution detailed models, maintaining mapping accuracy while significantly reducing processing time and computational resources required
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Provided are systems and a method for photorealistic real-time face reenactment. An example method includes receiving a target video including a target face and a source video including a source face. The method includes determining, based on the target face, a target facial expression. The method includes determining, based on the source face, a source facial expression. The method includes synthesizing, using the parametric face model, an output face. The output face including the target face wherein the target facial expression is modified to imitate the source facial expression. The method includes generating, based on a deep neural network, mouth and eyes regions, and combining the output face, the mouth, and eyes regions to generate a frame of an output video.