Real-Time Face Reenactment With Localized DNN Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current face reenactment technologies either produce non-photorealistic results or are too computationally intensive for real-time performance on mobile devices, limiting their applicability and effectiveness.
Innovation Solution
A method and system utilizing a parametric face model and deep neural network (DNN) for real-time photorealistic face reenactment on mobile devices, combining a parametric face model for facial identity and texture with a DNN for mouth and eye regions to enhance realism.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If deep learning methods are used for face reenactment, then photorealistic results are obtained, but computation time increases and real-time performance on mobile devices becomes unachievable
Solution Approach 1:
The face reenactment system is segmented into two independent parts: a parametric face model that handles facial identity and texture (computationally lightweight) and a deep neural network that processes only the mouth and eye regions (computationally intensive but localized). This segmentation allows the system to achieve photorealistic results in critical regions while maintaining overall computational efficiency for real-time performance on mobile devices.
2Loss of time
If morphable face models are used for face reenactment, then real-time performance is achieved, but photorealistic quality is compromised
Solution Approach 1:
The system applies different quality levels to different regions of the face. The parametric face model provides efficient base rendering for the entire face, while the deep neural network applies enhanced photorealistic processing specifically to the mouth and eye regions where realism is most critical. This local quality enhancement allows the system to maintain real-time performance while achieving photorealistic results in key areas.
3Manufacturing precision
If deep learning methods are applied to the entire face, then photorealistic results are obtained, but computational complexity increases for mobile devices
Solution Approach 1:
The deep neural network is extracted from processing the entire face and is instead applied only to specific regions (mouth and eyes). This extraction reduces the computational complexity by limiting the intensive deep learning operations to only the necessary regions, making the system feasible for deployment on mobile devices while still achieving photorealistic results where most important.
Data Source
AI summary
Systems and methods for text and audio-based real-time face reenactment are provided. An example method includes receiving a target video that includes a target face, receiving a source video that includes a source face, determining, based on a parametric face model, facial expression parameters of the source face, modifying, in real time, the target face to imitate a face expression of the source face based on the facial expression parameters to generate a sequence of modified video frames, and displaying at least part of the sequence of modified video frames on a computing device during the generation of at least one frame of the sequence of modified video frames.