3D Mesh Vertex Sequence Generation for Realistic Talking Head Animation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning-based methods for generating talking videos from a single image face challenges in producing realistic and synchronized facial expressions, with noticeable jitter and lack of authenticity in generated animations.
Innovation Solution
The method employs generative adversarial nets (GANs) with a generator and discriminator to train a model using audio features and real 3D mesh vertex sequences, improving the authenticity, continuity, and synchronization of facial expressions by iteratively adjusting network parameters until a training completion condition is met, without increasing generator parameters or calculation time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If existing deep learning-based methods are used to generate talking videos from a single image, then the lip shape can match the provided speech well, but the generated facial expression shows obvious jitter between frames and appears unreal with poor audio synchronization
Solution Approach 1:
The patent transitions from 2D image-based facial animation to 3D mesh vertex sequence-based animation. By extracting 3D mesh vertices from input images and generating 3D mesh vertex sequences that correspond to audio features, the system achieves more realistic and stable facial expressions while maintaining lip-sync accuracy, thereby resolving the contradiction between manufacturing precision and reliability
Solution Approach 2:
The patent introduces an audio feature extraction module and a 3D mesh generation module as intermediaries between the input audio and the final facial animation. The audio feature extraction module converts audio signals into feature representations, which then guide the 3D mesh vertex sequence generation, enabling both accurate lip synchronization and realistic facial expressions without direct pixel manipulation
2Reliability
If generative adversarial nets are trained with more parameters and longer training time, then the authenticity of generated facial expressions may improve, but the calculation time and complexity increase significantly
Solution Approach 1:
The patent performs preliminary action by pre-extracting 3D mesh vertices from input images before the main training process. The generator is pre-configured with a 3D mesh generation architecture that directly outputs 3D mesh vertex sequences. This preliminary preparation reduces the complexity and time required during the actual training phase while maintaining high authenticity in generated expressions
Solution Approach 2:
The patent changes the parameter representation from traditional 2D pixel-based facial landmarks to 3D mesh vertex coordinates. This parameter transformation enables more efficient training convergence while achieving higher authenticity, as 3D mesh parameters naturally encode spatial relationships and facial geometry, reducing the number of training iterations needed compared to conventional approaches
3Reliability
If the generator produces more detailed and realistic facial expressions, then the vividness of 3D animations improves, but the jitter between frames increases
Solution Approach 1:
The patent ensures continuity of useful action by generating 3D mesh vertex sequences that inherently maintain temporal coherence. The generator is trained to produce sequential 3D mesh frames that smoothly transition between states, and the rendering module continuously renders these sequences into video frames. This continuous generation process preserves both vividness and frame-to-frame stability without introducing jitter
Solution Approach 2:
By moving from 2D image generation to 3D mesh vertex sequence generation, the patent introduces an additional spatial dimension that naturally constrains facial expression transitions. The 3D mesh structure provides geometric priors that ensure physically plausible and temporally smooth transformations, reducing jitter while enhancing animation vividness through realistic 3D facial geometry
Data Source
AI summary
Methods and apparatuses for generating a model and generating a 3D animation, devices, and storage mediums are provided. The method for generating a model may include: acquiring a preset sample set; acquiring pre-established generative adversarial nets, the generative adversarial nets including a generator and a discriminator; and performing training steps as follows: selecting a sample from the sample set; extracting a sample audio feature from the sample audio of the sample; inputting the sample audio feature into the generator to obtain a pseudo 3D mesh vertex sequence of the sample; inputting the pseudo 3D mesh vertex sequence and the real 3D mesh vertex sequence of the sample into the discriminator to discriminate authenticity of 3D mesh vertices; and in response to determining that the generative adversarial nets meet a training completion condition, obtaining a trained generator as a model for generating a 3D animation.