Multilingual Speech Synthesis With Cross-Language Speaker Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional speech synthesis models struggle to naturally generate speech for unseen speaker-language combinations and often produce low-quality synthesized speech with incorrect tone or intonation, limiting their applications to specific uses.
Innovation Solution
A speech synthesis apparatus and method that includes a memory to store user-set language information and audio samples of a selected speaker, and a processor to generate audio signals using a speech synthesis model, allowing for natural speech synthesis even when audio data of the target speaker in the target language is sparse or absent.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional speech synthesis models are trained on speaker-language specific data, then they can generate speech for that specific combination, but they fail to synthesize speech for unseen speaker-language combinations
Solution Approach 1:
The speech synthesis model is segmented into separate language-specific processing components and a speaker-specific voice characteristic component. This allows the system to handle different languages independently while reusing speaker voice characteristics across language boundaries, enabling unseen speaker-language combinations to be synthesized with natural quality.
Solution Approach 2:
The patent introduces a new dimension by separating language-specific features from speaker-specific features. Instead of treating speaker-language combinations as a single coupled entity, the system processes language information and speaker information in separate dimensional spaces, then combines them to generate speech for any speaker-language pairing.
2Ease of manufacture
If speech synthesis models use limited training data for a specific speaker-language combination, then training is simpler, but the synthesized speech lacks natural tone and intonation
Solution Approach 1:
The speaker encoding component is designed to be universal across multiple languages. The same speaker encoder can extract voice characteristics that are then applied to generate speech in any language, making the speaker-specific portion of the model multi-functional and language-agnostic while maintaining natural tone and intonation.
3Stability of the object's composition
If single-speaker speech synthesis models are used, then they generate consistent speaker characteristics, but their applications are limited to specific uses
Solution Approach 1:
The system dynamically switches between different speaker encodings and language embeddings based on input requirements. The model can adapt its parameters to generate speech for any speaker-language combination while maintaining the stability and consistency of each individual speaker's voice characteristics through the speaker encoding mechanism.
Data Source
AI summary
A speech synthesis apparatus includes a memory configured to store language information set by a user and audio samples of a speaker selected by the user. The speech synthesis apparatus also includes a processor configured to generate audio signals corresponding to input text by applying a speech synthesis model to the input text, the language information, and the audio samples in response to a speech synthesis request of the user. The language information is different from a language related to the audio samples.