Speech-Driven Animation Using PPG for Speaker-Independent Lip Sync

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing speech-driven animation systems struggle to accurately generate consistent mouth shapes for different speakers due to speaker-specific acoustic features, leading to degraded interactive experiences.

Innovation Solution

A speech-driven animation method using linguistics information, such as phonetic posterior grams (PPG), to determine expression parameters independent of the speaker, enabling accurate animation character expressions through neural network mapping models.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If acoustic features (such as MFCC) are extracted from speech to determine expression parameters, then the mapping model can accurately determine expression parameters for a specific speaker, but when the speaker changes, the determined expression parameter deviates greatly and the mouth shape becomes inconsistent with the speech

Engineering Contradiction:

Improveaccuracy of expression parameter determinationVSAvoidcompatibility with different speakers

Core Design Contradiction:

Measurement precisionVSAdaptability or versatility

Solution Approach 1:

The patent extracts and removes speaker-specific acoustic features from the speech processing pipeline. By eliminating features like MFCC that contain speaker identity information, the system prevents speaker-specific bias from affecting expression parameter determination, thereby achieving speaker-independent accurate lip-sync animation.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent changes the feature representation from acoustic features (speaker-dependent) to linguistic features (speaker-independent). This parameter transformation allows the mapping model to determine expression parameters based on phonetic content rather than speaker characteristics, resolving the contradiction between accuracy for specific speakers and adaptability to different speakers.

Inventive Principle:

Principle #35Parameter changes

2Reliability

If speaker-specific acoustic features are used for mapping, then accurate animation can be achieved for trained speakers, but the system cannot effectively support speech from any speaker without retraining

Engineering Contradiction:

Improveaccuracy of animation generationVSAvoidsupport for any speaker

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The patent makes the mapping model universal by removing speaker-specific dependencies. The model processes linguistic features that are common to all speakers rather than speaker-specific acoustic features, enabling the same model to accurately generate animation for any speaker without requiring speaker-specific training or adaptation.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS20260038178A1Speech-driven animation method and apparatus based on artificial intelligence

Publication Date: 2026.02.05 TENCENT TECHNOLOGY (SHENZHEN) CO LTD

AI summary

Embodiments of this application disclose a speech-driven animation method and apparatus based on artificial intelligence (AI). The method includes obtaining a first speech, the first speech comprising a plurality of speech frames; determining linguistics information corresponding to a speech frame in the first speech, the linguistics information being used for identifying a distribution possibility that the speech frame in the first speech pertains to phonemes; determining an expression parameter corresponding to the speech frame in the first speech according to the linguistics information; and enabling, according to the expression parameter, an animation character to make an expression corresponding to the first speech.