Parallel Tacotron TTS With Prosody Disentanglement and Duration Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Autoregressive text-to-speech (TTS) models are inefficient during inference due to their reliance on hundreds of calculations and prone to discrepancies between training and inference, leading to reduced quality in synthesized speech with issues like babbling, early cut-off, and word repetition.
Innovation Solution
A non-autoregressive TTS model augmented with a variational autoencoder (VAE)-based residual encoder that disentangles style/prosody information from reference audio signals, allowing for efficient prediction of mel-frequency spectrogram sequences with controlled prosody/style.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If an autoregressive TTS model is used to synthesize speech, then speech quality and naturalness are improved, but inference efficiency deteriorates due to hundreds of sequential calculations
Solution Approach 1:
The model segments the speech synthesis process into independent parallel components: a parallel decoder that processes multiple output tokens simultaneously, a duration predictor that estimates phoneme lengths in advance, and a prosody encoder that separately captures style information. This segmentation eliminates sequential dependencies while maintaining speech quality.
Solution Approach 2:
The duration predictor performs preliminary action by estimating phoneme durations before the main synthesis process. This advance prediction allows the parallel decoder to generate multiple speech tokens simultaneously without waiting for sequential processing, dramatically improving inference efficiency while preserving natural speech timing.
2Manufacturing precision
If an autoregressive TTS model is used to generate speech, then natural speech output is achieved, but training-inference discrepancy increases leading to quality degradation
Solution Approach 1:
The model implements feedback through a teacher-forcing mechanism during training where the parallel decoder receives ground truth durations and prosody embeddings, and through a duration loss function that provides explicit feedback on timing accuracy. This aligns training objectives with inference behavior, eliminating the training-inference discrepancy that plagues autoregressive models.
Solution Approach 2:
The model changes parameters by using explicit duration predictions and prosody embeddings as fixed inputs during both training and inference, rather than relying on sequential generation. This parameterization approach ensures consistent behavior across training and inference phases while maintaining natural speech quality.
3Manufacturing precision
If style/prosody information is mixed with content in TTS synthesis, then speech naturalness is improved, but controllability deteriorates
Solution Approach 1:
The model extracts style/prosody information from reference audio through a separate prosody encoder that outputs dedicated prosody embeddings. This extracted prosody representation is then combined with text content embeddings in a controlled manner, allowing independent manipulation of style and content while maintaining natural speech characteristics.
Solution Approach 2:
The prosody embeddings serve as an intermediary between the prosody encoder and the parallel decoder. This intermediary representation disentangles style information from content, enabling fine-grained control over prosody parameters such as pitch, duration, and intensity while preserving speech naturalness through the mediation of learned embeddings.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method (600) for training a non-autoregressive TTS model (300) includes receiving a reference audio signal (201) and a corresponding input text sequence (206). The method also includes encoding the reference audio signal into a variational embedding (220) that disentangles the style/prosody information from the reference audio signal and encoding the input text sequence into an encoded text sequence (219). The method also includes predicting a phoneme duration (240) for each phoneme in the input text sequence and determining a phoneme duration loss (244). The method also includes generating one or more predicted mel-frequency spectrogram sequences (302) for the input text sequence and determining a final spectrogram loss (280) based on the predicted mel-frequency spectrogram sequences and a reference mel-frequency spectrogram sequence (202). The method also includes training the TTS model based on the final spectrogram loss and the corresponding phoneme duration loss.