Duration-Informed Attention for Stable Audio-Visual Speech Synthesis

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

End-to-end attention based models in speech synthesis systems suffer from issues such as omitting and repeating words in the input text, leading to instability and poor controllability.

Innovation Solution

Introducing a Duration Informed Attention Network (DurIAN) that models phone duration to align input text with output spectrograms, ensuring sequential synthesis without skipping or repeating, and incorporating duration modeling into the end-to-end attention framework.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If end-to-end attention based models are used for speech synthesis, then naturalness and prosody of synthesized speech are improved, but words are omitted or repeated and controllability deteriorates

Engineering Contradiction:

Improvenaturalness of synthesized speechVSAvoidcontrollability of speech synthesis

Core Design Contradiction:

ReliabilityVSEase of operation

Solution Approach 1:

The patent introduces a duration model as an intermediary component between the text input and speech output in the end-to-end attention framework. This duration model predicts the duration of each phoneme or text unit, serving as a mediator that guides the attention mechanism to allocate appropriate time durations to different parts of the input text, thereby preventing word omission and repetition while maintaining natural speech prosody.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If end-to-end attention framework is used, then speech generation speed is improved, but synchronization between input text and output speech deteriorates

Engineering Contradiction:

Improvespeech generation speedVSAvoidsynchronization accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The patent applies preliminary action by having the duration model predict the duration of each text unit before the actual speech synthesis process. This pre-computed duration information is then used to guide the attention mechanism and control the timing of speech generation, ensuring that the output speech is properly synchronized with the input text while maintaining efficient end-to-end processing speed.

Inventive Principle:

Principle #10Preliminary action

3Ease of operation

If traditional speech synthesis systems are used, then controllability is improved, but naturalness and prosody deteriorate

Engineering Contradiction:

Improvecontrollability of synthesis processVSAvoidnaturalness of synthesized speech

Core Design Contradiction:

Ease of operationVSReliability

Solution Approach 1:

The patent merges the advantages of traditional controllable speech synthesis systems with modern end-to-end natural speech generation. By integrating the duration model with the attention-based end-to-end framework, the system combines the controllability and structure of traditional systems with the naturalness and prosody of end-to-end models, achieving both goals simultaneously.

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentEP3942548B1Duration-informed attention network (durian) for audio-visual synthesis

Publication Date: 2026.04.01 TENCENT AMERICA LLC

Figure 1
Figure 2
Figure 3

AI summary

A method and apparatus include receiving a text input that includes a sequence of text components. Respective temporal durations of the text components are determined using a duration model. A spectrogram frame is generated based on the duration model. An audio waveform is generated based on the spectrogram frame. Video information is generated based on the audio waveform. The audio waveform is provided as an output along with a corresponding video.