Speech-Driven Facial Animation via Mel Spectrogram and Sliding Window
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current speech-driven facial animation technologies are limited by the accuracy of phoneme recognition, require large datasets, struggle with real-time generation, and have poor generalization when input speech differs from training speech, and often result in defects during segment splicing.
Innovation Solution
A method using Mel spectrogram features to extract frequency and time information, processed through a three-stage deep neural network to generate deformation gradients for facial motion, allowing for real-time generation of facial animation that can be controlled by a one-hot vector and applied to any facial model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If phoneme recognition and coarticulation rules are used for speech animation, then the animation can be generated systematically, but the accuracy is limited by phoneme recognition results and cannot produce real speech animation
Solution Approach 1:
The patent replaces the mechanical phoneme recognition and rule-based coarticulation system with a deep neural network that directly maps speech audio signals to facial animation parameters. This substitution eliminates the intermediate phoneme recognition step and its associated accuracy limitations, allowing the system to learn natural coarticulation patterns directly from data without being constrained by manual rule definitions
Solution Approach 2:
The patent introduces an intermediate representation layer that transforms raw speech audio into a compressed semantic feature space before generating animation parameters. This intermediate representation captures essential speech characteristics while filtering out noise, serving as a bridge between audio input and animation output that improves both accuracy and computational efficiency
2Reliability
If multiple animation segments are extracted from data samples for splicing, then the authenticity of animation is improved, but the effect is limited by the number of samples and there are defects at splicing positions
Solution Approach 1:
The patent implements a continuous animation generation approach where the deep neural network produces smooth, continuous animation sequences without discrete segment splicing. The network learns temporal dependencies and transitions between phonemes, ensuring continuous and natural facial movements throughout the animation sequence, eliminating the artifacts and discontinuities inherent in segment-based methods
Solution Approach 2:
The patent performs preliminary training of the deep neural network on large datasets of speech-audio and animation pairs before deployment. This pre-training phase allows the model to learn natural coarticulation patterns, transitions, and temporal dynamics in advance, so that during actual animation generation, the system can produce high-quality results rapidly without needing to process or splice multiple segments
3Measurement precision
If BiLSTM is used to learn mapping from speech to animation, then natural coarticulation mode is learned, but the system requires whole speech input and cannot be generated in real time
Solution Approach 1:
The patent segments the speech processing into fixed-size sliding windows with overlapping frames, allowing the system to process speech incrementally rather than requiring the entire speech sequence in advance. Each window is processed independently by the neural network to generate corresponding animation frames, enabling real-time or near-real-time animation generation while still capturing local coarticulation patterns effectively
Solution Approach 2:
The patent employs a dynamic architecture that adapts the processing window size and overlap based on the speech characteristics and desired animation quality. The system can adjust the degree of temporal context considered for each animation frame, balancing between capturing sufficient coarticulation information and maintaining real-time generation capability. This dynamic approach allows flexible optimization for different application scenarios
4Productivity
If sliding window technology with DNN is used to map phonemes to facial coefficients, then real-time generation is enabled, but the generalization is poor when input speech is greatly different from training speech
Solution Approach 1:
The patent transforms the input representation from discrete phoneme sequences to continuous Mel-frequency spectrogram features, which capture the spectral characteristics of speech in a manner that is more invariant to variations in speaker, accent, and speaking style. This parameter transformation allows the DNN to generalize better to unseen speech while maintaining real-time processing capability through efficient spectral feature extraction and neural network inference
Data Source
AI summary
The present disclosure discloses a speech-driven facial animation generation method. The method is mainly divided into six steps: extracting speech features, collecting frequency information, summarizing time information, decoding action features, driving a facial model, and sliding a signal window. The present disclosure can drive, according to an input speech audio signal, any facial model in real time under a particular delay to generate animation. The quality of the animation reaches the currently most advanced speech animation technology level, and has the characteristics of light weight and good robustness. The present disclosure can be used to generate speech animation under different scenes, such as VR virtual social networking, and virtual speech assistants and games.