Text-to-Speech Model Phoneme Embedding Diversity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current text-to-speech (TTS) models prioritize naturalness over diversity, resulting in synthesized speech that lacks variation, which can lead to less robust and accurate speech recognition systems when trained with conventional synthesized speech samples.
Innovation Solution
A TTS model is trained using a fine-grained approach that encodes speech-related attributes at the sub-portion level, such as phonemes, and incorporates an autoregressive prior model to generate diverse and natural speech samples by aligning speech embeddings with reference spectrograms and extracting latent features, while maintaining naturalness through optimized quantization and reconstruction loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If TTS models focus on naturalness, then the naturalness of synthesized speech is improved, but the diversity of synthesized speech deteriorates
Solution Approach 1:
The patent segments the speech representation into discrete phoneme units, each with its own embedding vector. This segmentation allows the model to manipulate individual phoneme representations independently, enabling diverse combinations while maintaining natural speech patterns through learned embeddings for each unit.
Solution Approach 2:
The patent changes the parameter representation by using learned embedding vectors for phonemes instead of traditional discrete symbols. By representing phonemes as continuous vectors in a learned space, the model can generate diverse variations while maintaining naturalness through the continuous nature of the embeddings.
2Ease of manufacture
If conventional synthesized speech samples are used for training, then the training process is simplified, but the robustness and accuracy of speech recognition systems deteriorates
Solution Approach 1:
The patent performs preliminary action by generating diverse phoneme embeddings and synthesizing speech samples with controlled variations before training the speech recognition system. This pre-prepared diverse training data ensures robustness and accuracy while keeping the training process itself relatively simple.
Solution Approach 2:
The patent creates synthetic copies of speech data with controlled diversities by combining phoneme embeddings in different sequences and contexts. These synthetic copies serve as training data, providing the diversity needed for robust speech recognition without requiring extensive real human speech recordings.
Data Source
AI summary
A method of generating diverse and natural text-to-speech (TTS) samples includes receiving a text and generating a speech sample based on the text using a TTS model. A training process trains the TTS model to generate the speech sample by receiving training samples. Each training sample includes a spectrogram and a training text corresponding to the spectrogram. For each training sample, the training process identifies speech units associated with the training text. For each speech unit, the training process generates a speech embedding, aligns the speech embedding with a portion of the spectrogram, extracts a latent feature from the aligned portion of the spectrogram, and assigns a quantized embedding to the latent feature. The training process generates the speech sample by decoding a concatenation of the speech embeddings and a quantized embeddings for the speech units associated with the training text corresponding to the spectrogram.