Conditional Variational Autoencoder for Audio-Driven Facial Animation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing viseme-based techniques for computer facial animation are complex due to intricate co-articulation rules, making them inefficient for generating realistic facial movements from audio data.
Innovation Solution
A computer-implemented method using a conditional variational autoencoder is trained to generate facial position data from audio data, involving an encoder and decoder to process training data, sample latent vectors, and update parameters based on loss values, enabling the generation of facial position outputs from audio descriptors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If viseme-based techniques are used for facial animation, then facial movements can be generated from audio data, but the system becomes complex due to intricate co-articulation rules
Solution Approach 1:
The patent replaces the mechanical rule-based viseme system with a neural network-based conditional variational autoencoder. The CVAE learns facial animation directly from audio data and facial position descriptors, eliminating the need for complex co-articulation rules while maintaining the ability to generate realistic facial movements.
Solution Approach 2:
The patent transforms the approach by changing from discrete viseme parameters with fixed rules to continuous latent space parameters learned by the neural network. The encoder maps audio descriptors and facial positions to latent space distributions, and the decoder reconstructs facial positions from sampled latent vectors, allowing flexible and realistic facial animation without rigid rule systems.
2Manufacturing precision
If complex co-articulation rules are implemented, then facial animation accuracy may improve, but computational resources increase
Solution Approach 1:
The patent performs preliminary action by pre-training the conditional variational autoencoder on large datasets of audio-facial pairs. Once trained, the model can generate accurate facial animations in real-time with minimal computational resources, as the complex learning has already been accomplished during the offline training phase.
Solution Approach 2:
The patent creates a learned mapping model (the trained CVAE) that copies the relationship between audio data and facial positions from training data. This learned copy allows the system to generate accurate facial animations without repeatedly processing complex rules during runtime, significantly reducing computational requirements.
Data Source
AI summary
A computer-implemented method for generating a machine-learned model to generate facial position data based on audio data comprising training a conditional variational autoencoder having an encoder and decoder. The training comprises receiving a set of training data items, each training data item comprising a facial position descriptor and an audio descriptor; processing one or more of the training data items using the encoder to obtain distribution parameters; sampling a latent vector from a latent space distribution based on the distribution parameters; processing the latent vector and the audio descriptor using the decoder to obtain a facial position output; calculating a loss value based at least in part on a comparison of the facial position output and the facial position descriptor of at least one of the one or more training data items; and updating parameters of the conditional variational autoencoder based at least in part on the calculated loss value.