3D Facial Animation Using Audio-Visual Phoneme Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing facial animation systems require special equipment or large volumes of training data, making them impractical for ordinary users and lacking in accuracy.
Innovation Solution
A joint audio-visual driven facial animation system using Large Vocabulary Continuous Speech Recognition (LVCSR) with 3D face modeling, combining video and acoustic inputs to track 3D facial motion, and applying knowledge-guided 3D blend shapes modeling to synthesize facial animation without training data, utilizing Bi-Directional Long Short-Term Memory (BLSTM) and Time Delayed Neural Network (TDNN) for phoneme alignment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If performance-based facial animation is used to generate realistic character facial animation, then animation quality is improved, but special equipment such as physical markers, structured light, and camera arrays are required
Solution Approach 1:
The patent replaces the mechanical/optical measurement system (physical markers, structured light, camera arrays) with an acoustic field-based system. Audio recordings are processed to extract phoneme information that drives facial animation, substituting complex physical measurement equipment with audio processing and computational modeling.
Solution Approach 2:
The patent creates a virtual 3D facial model that copies and simulates real human facial movements. Instead of directly measuring real faces with complex equipment, the system generates animated facial movements by mapping phoneme sequences to corresponding facial expressions in a virtual model, achieving realistic animation without physical measurement apparatus.
2Ease of operation
If speech-driven facial animation with phoneme mapping is used, then ease of operation is improved, but accuracy depends greatly on the volume of training data available
Solution Approach 1:
The patent changes the approach from requiring large volumes of training data to using a phoneme-based parameter system. By mapping audio phonemes directly to facial animation parameters through a 3D morphable model, the system achieves accurate results with minimal training data, fundamentally changing the data requirements while maintaining operational simplicity.
3Measurement precision
If large volumes of training data are collected for speech-driven animation, then animation accuracy is improved, but data collection time and storage requirements increase
Solution Approach 1:
The patent extracts only the essential phoneme information from audio recordings, removing the need to collect and process large volumes of video training data. By focusing on acoustic phoneme extraction and mapping these to facial parameters through a pre-defined 3D morphable model, the system achieves accurate animation while minimizing data collection requirements.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present invention relates to a joint automatic audio visual driven facial animation system that in some example embodiments includes a full scale state of the art Large Vocabulary Continuous Speech Recognition (LVCSR) with a strong language model for speech recognition and obtained phoneme alignment from the word lattice.