Unsupervised Text-to-Speech Synthesis via C-DSVAE
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current supervised text-to-speech systems require extensive and costly high-quality data for training, limiting their deployment in diverse applications such as AI assistants and entertainment, as they struggle with generating natural and flexible speech without sufficient resources.
Innovation Solution
An unsupervised text-to-speech method using a Conditional Disentangled Sequential Variational Auto-encoder (C-DSVAE) that maps input text to phoneme sequences, predicts speech duration, and generates mel spectrograms without requiring parallel speech and textual data, enabling flexible and natural speech synthesis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If supervised TTS systems are used to generate high-fidelity synthetic speech, then speech quality is improved, but data collection and labeling costs increase significantly
Solution Approach 1:
The system performs self-supervised learning by automatically learning speech representations from unlabeled audio data without requiring manual annotation. The model learns to predict future audio frames and reconstruct speech, enabling it to train on large quantities of unlabelled data while maintaining high speech quality, thus eliminating the need for expensive data collection and labeling processes
Solution Approach 2:
The system uses a pre-trained speech representation model that has learned from large amounts of unlabeled data and applies this learned representation to the TTS task. This allows the system to leverage knowledge from general speech data without requiring task-specific labeled parallel speech-text data, reducing data requirements while maintaining speech quality
2Manufacturing precision
If supervised TTS systems are deployed for customized speech synthesizers, then speech quality is improved, but deployment flexibility decreases due to data requirements
Solution Approach 1:
The system enables customized speech synthesizers to be deployed flexibly by using self-supervised learning that does not require task-specific labeled data. The model can be adapted to different speakers and languages by training on unlabeled data from the target domain, maintaining speech quality while enabling ubiquitous deployment across diverse applications without data collection bottlenecks
Solution Approach 2:
The speech representation model learned through self-supervised learning is universal and can be applied to multiple TTS tasks across different speakers, languages, and domains. This single pre-trained representation serves multiple purposes, enabling the system to generate customized speech synthesizers for various applications without requiring separate labeled datasets for each use case
Data Source
AI summary
An unsupervised text to speech system utilizing a lexicon to map input text to the phoneme sequence, which is expanded to the frame-level forced alignment with a speaker-dependent duration model. An alignment mapping module that converts the forced alignment to the unsupervised alignment (UA). Afterword, a Conditional Disentangled Sequential Variational Auto-encoder (C-DSVAE), serving as the self-supervised TTS AM, takes the predicted UA and a target speaker embedding to generate the mel spectrogram, which is ultimately converted to waveform with a neural vocoder.