Duration-Informed Attention Network for Speech Synthesis Stability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Tacotron-based end-to-end speech synthesis systems suffer from instability issues such as skipping and repeating input text due to their uncontrollable attention mechanism, which affects the alignment between input text and speech signals.
Innovation Solution
Replacing the end-to-end attention mechanism with a duration-informed attention network that predicts the temporal duration of each input character or phoneme, ensuring sequential synthesis without skipping or repeating, and using HMM-based forced alignment to determine ground truth durations for accurate alignment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If an end-to-end attention mechanism is used in Tacotron systems, then speech synthesis naturalness is improved, but text skipping and repeating occur due to uncontrollable attention
Solution Approach 1:
The patent segments the attention mechanism by introducing character-level attention weights that independently control the synthesis of each character. This segmentation allows precise control over text alignment at the character level, preventing skipping and repeating while maintaining synthesis naturalness through the segmented attention structure.
Solution Approach 2:
The patent implements feedback through the teacher forcing mechanism where the ground truth character sequences are fed back into the network during training. This feedback ensures that the attention mechanism learns to align characters correctly with their corresponding phonemes, resolving the uncontrollable attention issue while maintaining synthesis quality.
2Measurement precision
If HMM-based forced alignment is used to determine ground truth durations, then alignment accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent applies preliminary action by pre-computing the ground truth character durations and attention weights using HMM-based forced alignment during the training phase. These pre-computed values are then stored and reused during inference, eliminating the need for complex real-time alignment calculations and reducing computational complexity at deployment.
Solution Approach 2:
The patent uses copying by creating a simplified version of the complex HMM alignment process that captures the essential duration information without requiring the full HMM computational overhead during inference. The ground truth durations are copied from the alignment results and used directly in the synthesis process, maintaining accuracy while reducing complexity.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method and apparatus include receiving a text input that includes a sequence of text components. Respective temporal durations of the text components are determined using a duration model. A first set of spectra is generated based on the sequence of text components. A second set of spectra is generated based on the first set of spectra and the respective temporal durations of the sequence of text components. A spectrogram frame is generated based on the second set of spectra. An audio waveform is generated based on the spectrogram frame. The audio waveform is provided as an output.