Non-Autoregressive TTS With VAE Prosody Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Autoregressive TTS models are inefficient during inference due to their sequential nature and prone to discrepancies between training and inference, leading to reduced quality in synthesized speech, especially for longer text inputs.
Innovation Solution
A non-autoregressive TTS model augmented with a variational autoencoder (VAE)-based residual encoder that disentangles style/prosody information from reference audio signals, allowing for efficient prediction of mel-frequency spectrogram sequences with controlled prosody/style.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If an autoregressive TTS model is used, then highly natural speech outputs can be synthesized, but inference efficiency deteriorates due to sequential processing requiring hundreds of calculations
Solution Approach 1:
The model segments the speech synthesis process into independent parallel components: a text encoder that processes input text, a duration predictor that estimates phoneme durations, and a spectrogram generator that creates mel-spectrograms. These components operate simultaneously rather than sequentially, enabling parallel computation while maintaining natural speech quality.
Solution Approach 2:
The model performs preliminary encoding of the entire input text sequence into contextual representations before generating the spectrogram. The text encoder processes all input tokens in parallel to create contextual embeddings, which are then used by the duration predictor and spectrogram generator, eliminating the need for sequential autoregressive generation.
2Adaptability or versatility
If an autoregressive TTS model is used, then text synthesis can be performed, but discrepancies between training and inference occur leading to reduced quality for longer text inputs
Solution Approach 1:
The model uses a parallel architecture that copies the training-time computational structure to inference time. Both during training and inference, the entire input sequence is encoded simultaneously and the spectrogram is generated in parallel, eliminating the training-inference discrepancy that plagues autoregressive models and ensuring consistent quality for texts of any length.
3Productivity
If a non-autoregressive model is used, then inference efficiency improves, but speech expressiveness and prosody control deteriorate
Solution Approach 1:
The model introduces a duration predictor as an intermediary component that estimates phoneme durations based on encoded text representations. This duration information serves as a mediator between the text encoder and spectrogram generator, enabling the model to control speech prosody and expressiveness while maintaining parallel processing efficiency.
Solution Approach 2:
The model controls prosody by dynamically adjusting the duration parameter for each phoneme based on the input text and learned patterns. By modifying these duration parameters in parallel, the model achieves expressive speech synthesis with proper prosody while maintaining the efficiency benefits of non-autoregressive processing.
Data Source
AI summary
A method for training a non-autoregressive TTS model includes receiving training data that includes a reference audio signal and a corresponding input text sequence. The method also includes encoding the reference audio signal into a variational embedding that disentangles the style/prosody information from the reference audio signal and encoding the input text sequence into an encoded text sequence. The method also includes predicting a phoneme duration for each phoneme in the input text sequence and determining a phoneme duration loss based on the predicted phoneme durations and a reference phoneme duration. The method also includes generating one or more predicted mel-frequency spectrogram sequences for the input text sequence and determining a final spectrogram loss based on the predicted mel-frequency spectrogram sequences and a reference mel-frequency spectrogram sequence. The method also includes training the TTS model based on the final spectrogram loss and the corresponding phoneme duration loss.