Multilingual TTS With Disentangled Speaker-Language Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing end-to-end neural text-to-speech models face challenges in transferring voices across different languages, especially when training data is imbalanced and languages have little overlap, leading to difficulties in voice cloning and accent ambiguity.
Innovation Solution
A multispeaker, multilingual text-to-speech model that uses a speaker embedding and language embedding to generate output audio features, incorporating an adversarial loss module to disentangle speaker identity from language, enabling voice transfer across distantly related languages without bilingual training data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If language-dependent input representations are used in TTS models, then speech synthesis quality for each language is improved, but voice transfer across different languages becomes difficult
Solution Approach 1:
The model segments speaker identity and language into separate embedding components. Speaker embeddings capture voice characteristics independent of language, while language embeddings handle language-specific features. This segmentation allows the model to maintain high speech synthesis quality for each language while enabling voice transfer across languages by manipulating only the language embedding component.
Solution Approach 2:
The patent introduces an intermediary adversarial loss module that acts as a mediator between the speaker embedding and language embedding. This intermediary component learns to distinguish between speaker identity and language features, forcing the model to separate these factors. As a result, the model can preserve speech quality for each language while enabling cross-language voice transfer by controlling the speaker embedding independently from the language embedding.
2Ease of manufacture
If training data is imbalanced across languages, then models can be trained with available data, but voice transfer to languages with limited data becomes difficult
Solution Approach 1:
The speaker embedding is designed to be universal across all languages, capturing speaker identity characteristics that are language-independent. This universal embedding can be trained on imbalanced data across different languages and then applied to any target language by simply changing the language embedding. The adversarial loss module reinforces this universality by preventing the speaker embedding from encoding language-specific information, enabling effective voice transfer even to languages with limited training data.
3Device complexity
If speaker identity is perfectly correlated with language in training data, then monolingual training is simple, but cross-language voice transfer becomes difficult
Solution Approach 1:
The adversarial loss module serves as an intermediary that actively prevents the spurious correlation between speaker identity and language from forming in the training process. By introducing this intermediary component during training, the model learns to separate speaker characteristics from language characteristics even when training data is monolingual. This allows the simple monolingual training process to produce models capable of cross-language voice transfer.
4Adaptability or versatility
If bilingual training data is collected to enable voice transfer, then cross-language synthesis is improved, but data collection cost increases significantly
Solution Approach 1:
The patent extracts speaker identity information from training data into a separate speaker embedding component that is independent of language. By taking out the speaker characteristics from the language-specific training data, the model can apply the same speaker embedding to multiple languages without requiring bilingual recordings. This extraction approach enables cross-language synthesis capability while avoiding the high cost of collecting bilingual training data.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method (300) includes receiving an input text sequence (114) to be synthesized into speech (150) in a first language and obtaining a speaker embedding (116a), the speaker embedding specifying specific voice characteristics of a target speaker (10) for synthesizing the input text sequence into speech that clones a voice of the target speaker. The target speaker includes a native speaker of a second language different than the first language. The method also includes generating, using a text-to-speech (TTS) model (100), an output audio feature representation (119) of the input text sequence by processing the input text sequence and the speaker embedding. The output audio feature representation includes the voice characteristics of the target speaker specified by the speaker embedding.