Learned Condition Text-to-Speech Synthesis via Conditional VAE
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional text-to-speech (TTS) systems fail to produce natural-sounding multiple renditions of the same sentence and suffer from over-regularization and poor latent representations due to the use of a standard Gaussian prior in probabilistic generative models, lacking controllability over the latent space.
Innovation Solution
The implementation of a learned conditional prior in a hierarchical Variational Autoencoder (VAE) structure, specifically using a Conditioning Secondary VAE (CSVAE) to learn a latent distribution associated with prosodic expressivity, which acts as the prior for the primary encoder, allowing for controllable sampling and maintaining variability in speech generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If a standard Gaussian prior is used in probabilistic generative models for TTS, then the model structure is simple and easy to implement, but the system suffers from over-regularization and poor latent representations, failing to produce natural-sounding multiple renditions of the same sentence
Solution Approach 1:
The prior distribution is segmented into multiple conditional priors, each corresponding to different prosodic conditions (e.g., different speakers, emotions, or speaking styles). This allows the model to maintain simplicity for each condition while achieving high-quality, varied speech synthesis across multiple conditions by selecting the appropriate conditional prior based on the desired output characteristics.
Solution Approach 2:
The system transitions from a static Gaussian prior to dynamic conditional priors that adapt based on input conditions. The conditional priors are selected or adjusted according to desired prosodic characteristics, enabling the model to produce natural-sounding variations while maintaining computational efficiency through conditional selection rather than retraining entire model structures.
2Device complexity
If a standard Gaussian prior is used in TTS systems, then the model structure is simple, but the system lacks controllability over the latent space and produces poor latent representations
Solution Approach 1:
Conditional variables (such as speaker identities, emotional states, or style indicators) are introduced as intermediaries between the input text and the latent space sampling process. These intermediaries select or modulate the appropriate conditional prior, providing controllability over the latent space without requiring complex modifications to the underlying VAE architecture. This allows users to control speech characteristics by selecting different conditional priors while maintaining a relatively simple model structure.
3Ease of manufacture
If traditional TTS systems are used, then the implementation is straightforward, but the system fails to produce natural-sounding multiple renditions of the same sentence
Solution Approach 1:
The system changes the parameters of the prior distribution based on desired prosodic characteristics. Instead of using a fixed Gaussian prior, the model employs conditional priors with different parameters (mean, variance) conditioned on specific prosodic attributes. This allows straightforward implementation by selecting pre-defined conditional priors while achieving high prosodic variability and natural-sounding multiple renditions through parameter selection rather than complex model architecture changes.
Data Source
AI summary
Devices and techniques are generally described for learned condition text-to-speech synthesis. In some examples, first data representing a selection of a type of prosodic expressivity may be received. In some further examples, a selection of content comprising text data may be received. First audio data may be determined that includes an audio representation of the text data. The first audio data may be generated based at least in part on sampling from a first latent distribution generated using a conditional primary variational autoencoder (VAE). The sampling from the first latent distribution may be conditioned on a first learned distribution associated with the type of prosodic expressivity. In various examples, the first audio data may be sent to a first computing device.