Multilingual TTS Model Training with Shared Encoder
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current multilingual text-to-speech (TTS) models face challenges in scaling to hundreds or thousands of languages due to the scarcity of high-quality paired training data, especially for low-resource languages.
Innovation Solution
A computer-implemented method that trains a TTS model using a data processing system, where training data includes multiple sets of utterances in different languages. The method generates encoded textual and speech representations, determines TTS losses, and trains the model to synthesize speech in each language without supervision.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multilingual TTS models are trained to support more languages, then language coverage is improved, but training data requirements increase
Solution Approach 1:
The patent implements a shared encoder architecture that processes multiple languages through a single unified model structure. The text encoder and speech encoder are designed to handle diverse linguistic inputs without requiring separate specialized models for each language, enabling one TTS system to serve multiple languages efficiently
Solution Approach 2:
The patent leverages pre-trained ASR models and their encoders as starting points for TTS model training. By copying and adapting existing speech encoders from ASR systems, the approach reuses already-trained representations that capture speech patterns across languages, reducing the need to train encoders from scratch for each target language
2Manufacturing precision
If paired training data is used for TTS training, then synthesis quality is improved, but data availability decreases
Solution Approach 1:
The patent introduces an ASR decoder as an intermediary component that bridges the gap between unpaired speech data and TTS synthesis requirements. The decoder generates speech recognition hypotheses from acoustic features, providing a pathway to utilize unpaired data while maintaining synthesis quality through the mediating hypothesis generation process
Solution Approach 2:
The patent transforms the previously problematic unpaired speech data (which cannot be directly used for supervised TTS training) into a beneficial training resource. By using unpaired data to pre-train speech encoders and ASR decoders, the approach converts data scarcity into an opportunity for unsupervised pre-training that improves overall model performance
3Productivity
If ASR decoders are integrated into TTS training, then data utilization is improved, but model complexity increases
Solution Approach 1:
The patent merges the TTS model architecture with an ASR decoder into a unified joint training system. The text encoder, speech encoder, and ASR decoder are integrated and trained together as a single system, allowing shared representations and joint optimization that improves data utilization while managing complexity through unified architecture design
Data Source
AI summary
A method includes receiving training data that includes a plurality of sets of training utterances each associated with a respective language. Each training utterance includes a corresponding reference speech representation paired with a corresponding input text sequence. For each training utterance, the method includes generating a corresponding encoded textual representation for the corresponding input text sequence, generating a corresponding speech encoding for the corresponding reference speech representation, generating a shared encoder output, and determining a text-to-speech (TTS) loss based on the corresponding encoded textual representation, the corresponding speech encoding, and the shared encoder output. The method also includes training a TTS model based on the TTS losses determined for the training utterances in each set of the training utterances to teach the TTS model to learn how to synthesize speech in each of the respective languages.