Multilingual Speech Synthesis Using Disentangled Speaker-Language Features
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional speech synthesis models struggle to generate natural-sounding speech for unseen speakers and languages, requiring complex fine-tuning and leading to unstable speech duration predictions when dealing with multiple languages.
Innovation Solution
A speech synthesis model is trained to exclude speaker and language information during the training stage and add these features during the inference stage, using a neural network architecture that normalizes latent variables to remove speaker and language-related features, allowing for more accurate multi-speaker and multi-lingual speech synthesis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If speaker and language embeddings are input to the text encoder during training, then the model can learn multiple languages, but complex fine-tuning is required to generate high-quality speech
Solution Approach 1:
The patent segments the speaker and language information processing from the main text encoding process. Instead of inputting speaker and language embeddings to the text encoder, the model separately processes text, speaker audio, and language information through distinct encoders, then combines their features later in the architecture. This segmentation eliminates the need for complex fine-tuning while maintaining multi-language capability.
Solution Approach 2:
The patent extracts speaker and language information from the training process by using a speaker encoder that processes audio samples to generate speaker embeddings, and a language encoder that processes language information to generate language embeddings. These extracted features are then combined with text features without requiring the text encoder to directly process speaker and language embeddings, thereby simplifying the training process.
2Adaptability or versatility
If speaker and language embeddings are input to the duration predictor, then the model learns duration features of multiple languages, but predicted speech duration becomes unstable when sentences are expressed in multiple languages
Solution Approach 1:
The patent segments the duration prediction process by using a separate duration predictor that takes text features, speaker embeddings, and language embeddings as inputs, rather than relying solely on the text encoder. This segmentation allows the model to learn duration features for multiple languages while maintaining stable predictions in code-mixed sentences by processing each component independently and combining them in the duration predictor.
3Manufacturing precision
If conventional speech synthesis models are trained with speaker-specific data, then they can generate natural speech for trained speakers, but they cannot synthesize speech for unseen speakers or languages
Solution Approach 1:
The patent implements universality by designing a speech synthesis model that can handle multiple speakers and languages through a unified architecture. The speaker encoder processes audio samples from any speaker to generate speaker embeddings, and the language encoder processes language information to generate language embeddings. These universal encoders allow the model to synthesize speech for unseen speakers and languages without requiring speaker-specific training, while maintaining high speech quality for trained speakers through the combination of text, speaker, and language features.
Data Source
AI summary
A speech synthesis apparatus includes a memory configured to store language information configured by a user and audio samples of a speaker corresponding to speaker information selected by the user. The speech synthesis apparatus also includes a processor configured to generate an audio signal corresponding to input text by applying a speech synthesis model to the input text, the language information, and the audio samples in response to a speech synthesis request of the user. The speech synthesis model is trained to generate an audio signal including features of training text and features of a training audio signal. Language information of the training text and speaker information of the training audio signal are removed from the generated audio signal.