3D Talking Face Generation with Audio-Driven Geometry and Texture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for generating talking head videos lack geometric information and personalization, fail to reproduce 3D facial articulation and appearance with high fidelity, and are restricted by fixed-viewpoint 2D approaches, while methods requiring a surrogate actor or time-aligned transcripts are limiting.
Innovation Solution
A computing system using machine-learned models to predict face geometry and texture from audio input, trained on video data to decouple 3D geometry and texture, and incorporate auto-regressive techniques for temporal consistency and audio-independent illumination normalization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If 3D facial meshes are predicted from audio, then dynamic viewpoints and lighting changes are enabled, but visual realism is restricted to game-quality results
Solution Approach 1:
The patent transitions from 2D image-based talking heads to 3D mesh-based talking heads, adding the dimension of three-dimensionality. This enables dynamic viewpoints and lighting changes while maintaining photorealism through high-fidelity 3D facial articulation and texture mapping.
Solution Approach 2:
The patent combines multiple data types (audio signals, 3D facial meshes, textures, and lighting information) to create a composite talking head system. This integration allows the system to achieve both photorealism and adaptability by synthesizing multiple visual qualities from a single audio input.
2Manufacturing precision
If a surrogate actor video is used for face transfer, then impressive visual results are achieved, but the process requires additional human actor involvement and time alignment
Solution Approach 1:
The patent extracts the facial motion information directly from the audio signal, eliminating the need for surrogate actor videos. By regressing facial motions from audio alone, the system removes the complexity of actor coordination and time alignment while maintaining visual realism through high-fidelity facial articulation.
Solution Approach 2:
The patent creates a virtual copy of the target subject's face driven by audio input, eliminating the need for actual human actors. The system generates photorealistic talking heads by synthesizing facial motions from audio, effectively copying the appearance and motion characteristics without requiring real actors.
3Adaptability or versatility
If 2D talking heads are generated from audio, then personalization is achieved, but geometric information and 3D facial articulation are lost
Solution Approach 1:
The patent extends 2D personalization to 3D by generating three-dimensional facial meshes that maintain individual characteristics. This adds depth and spatial information to the personalized talking heads, enabling 3D facial articulation while preserving personalization through subject-specific training data.
Data Source
AI summary
Provided is a framework for generating photorealistic 3D talking faces conditioned only on audio input. In addition, the present disclosure provides associated methods to insert generated faces into existing videos or virtual environments. We decompose faces from video into a normalized space that decouples 3D geometry, head pose, and texture. This allows separating the prediction problem into regressions over the 3D face shape and the corresponding 2D texture atlas. To stabilize temporal dynamics, we propose an auto-regressive approach that conditions the model on its previous visual state. We also capture face illumination in our model using audio-independent 3D texture normalization.