Multilingual Speech Synthesis with Separated Speaker-Language Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional speech synthesis models struggle to naturally generate speech for unseen speaker-language combinations and often produce low-quality synthesized speech lacking natural speech style or emotional expression, limiting their applications to specific uses.
Innovation Solution
A speech synthesis model is trained using a language embedding module and a speaker encoder to separate acoustic features, incorporating a metric learning loss to minimize the overlap between speaker and language embeddings, enabling zero-shot synthesis of speech by unseen speakers in different languages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional speech synthesis models are trained on speaker-language specific data, then they can generate speech for seen speaker-language combinations, but they fail to generate speech for unseen speaker-language combinations
Solution Approach 1:
The patent segments the speech representation into separate speaker embedding and language embedding components. The speaker encoder extracts speaker-specific features while the language identifier captures language-specific characteristics. This segmentation allows the model to independently control speaker identity and language, enabling flexible combination for unseen speaker-language pairs while maintaining speech quality.
Solution Approach 2:
The patent creates a universal speech synthesis framework that can handle multiple speaker-language combinations through a single model. By using separate embed a d i n g s for speakers and languages, the model achieves multi-functionality where one trained model can generate speech for any combination of trained speakers and languages, rather than requiring separate models for each combination.
2Adaptability or versatility
If speech synthesis models use separate speaker and language embeddings, then they achieve better adaptability for unseen combinations, but the model complexity increases
Solution Approach 1:
The model architecture is segmented into distinct modules: speaker encoder, language identifier, and embedding combination mechanism. This modular segmentation makes the complex functionality manageable and interpretable, where each component has a specific role in extracting and combining speaker and language features.
Solution Approach 2:
The patent introduces embedding vectors as intermediaries between the speaker encoder and the speech generation process. These embeddings serve as mediators that capture speaker and language characteristics in a compressed form, facilitating the zero-shot synthesis capability without requiring complex direct mappings.
3Adaptability or versatility
If conventional models generate speech for multiple speakers, then they cover more applications, but they cannot preserve natural speech style and emotional expression
Solution Approach 1:
By segmenting the speaker representation into dedicated speaker embeddings extracted by the speaker encoder, the model can preserve fine-grained speaker characteristics including speech style and emotional expression. This segmentation allows the system to maintain high fidelity to the original speaker's voice qualities while still enabling multi-speaker coverage.
Solution Approach 2:
The patent changes the parameter representation from speaker-id indices to continuous speaker embedding vectors. This parameter transformation enables the model to capture nuanced speaker characteristics and variations in speech style and emotion, rather than treating all speakers as discrete categories with uniform characteristics.
Data Source
AI summary
A method for multilingual and multispeaker speech synthesis includes receiving training text and a training audio signal obtained from a speaker uttering the training text. The method further includes identifying a language identifier corresponding to the training text and a training reference audio signal obtained from the speaker uttering reference text different from the training text. The method further includes training a speech synthesis model using training samples that include the training text, the training audio signal, the language identifier, and the training reference audio signal.