Audio-Driven Animation Model for Lip and Facial Expression Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods fail to efficiently drive lip animation and facial expressions of virtual humans in real-time through speech, leading to dull and less comprehensible human-computer interactions.
Innovation Solution
A method using an audio-driven animation model with an audio encoding layer and one-hot encoding layer to generate blend shape values for synchronized lip animation and facial expressions, incorporating feature extraction and binary vector encoding to produce a blend shape value indicating both lip animation and facial expressions based on target audio data and speaking style identifiers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional speech analysis methods are used for lip animation, then lip synchronization can be achieved, but facial expressions cannot be effectively driven simultaneously
Solution Approach 1:
The patent segments the speech signal into multiple feature dimensions (spectral features, temporal features, prosodic features) and processes them through separate encoding pathways in the neural network. This segmentation allows independent extraction of lip animation cues and facial expression cues from the same speech signal, enabling both to be driven simultaneously without interfering with each other's generation efficiency
Solution Approach 2:
The patent designs a universal speech-driven animation generation system that can simultaneously produce multiple animation outputs (lip animation and facial expressions) from a single speech input. The multi-task neural network architecture processes the speech signal to generate both types of animation in one unified framework, improving versatility while maintaining productivity through shared feature extraction layers
2Reliability
If complex animation models are used to generate both lip animation and facial expressions, then animation quality improves, but computational cost increases
Solution Approach 1:
The patent merges the lip animation generation model and facial expression generation model into a single unified neural network architecture. By combining both animation tasks into one model with shared lower layers and specialized upper layers, the system achieves high animation quality through coordinated multi-task learning while reducing computational cost by eliminating redundant feature extraction operations that would occur in separate models
Solution Approach 2:
The patent performs preliminary feature extraction from the speech signal in the lower layers of the neural network, creating a shared representation that captures both lip animation and facial expression information. This preliminary processing allows the subsequent task-specific layers to focus on generating their respective outputs without re-processing the raw speech signal, thereby maintaining high quality while reducing overall computational cost
Data Source
AI summary
A method for outputting a blend shape value includes: performing feature extraction on obtained target audio data to obtain a target audio feature vector; inputting the target audio feature vector and a target identifier into an audio-driven animation model; inputting the target audio feature vector into an audio encoding layer, determining an input feature vector of a next layer at a (2t−n)/2 time point based on an input feature vector of a previous layer between a t time point and a t−n time point, determining a feature vector having a causal relationship with the input feature vector of the previous layer as a valid feature vector, outputting sequentially target-audio encoding features, and inputting the target identifier into a one-hot encoding layer for binary vector encoding to obtain a target-identifier encoding feature; and outputting a blend shape value corresponding to the target audio data.