Cross-Accent Voice Cloning with Synthetic TTS Training Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text-to-speech (TTS) systems struggle to generate synthesized speech that clones the voice of a target speaker in a different accent or dialect without sufficient training data, particularly when the target speaker does not natively speak that accent or dialect.
Innovation Solution
A trained voice cloning system generates training synthesized speech representations that clone the voice of a target speaker in a different accent or dialect, which are then used to train a TTS system to produce expressive speech that matches the target speaker's voice in the intended accent or dialect.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If traditional TTS training methods are used with limited training data, then the system can be trained quickly, but the synthesized speech cannot accurately replicate the target speaker's voice in a different accent or dialect
Solution Approach 1:
The patent creates synthetic training data by copying and transforming existing speech data. A first TTS system generates synthetic speech representations that mimic the target speaker's voice characteristics in the desired accent/dialect. These synthetic copies are then used to train the second TTS system, eliminating the need for actual recording sessions with the target speaker in the target accent.
Solution Approach 2:
The patent performs preliminary data preparation by pre-generating synthetic training data before the actual TTS training process. The first TTS system is trained in advance on diverse speech data, and then used to create synthetic training examples that are prepared beforehand for training the second TTS system, streamlining the overall process.
2Adaptability or versatility
If synthetic training data is generated using a trained TTS system, then training data for rare accents/dialects can be created, but the complexity of the system increases
Solution Approach 1:
The first TTS system serves multiple functions: it acts as a voice cloning model, a data generation engine, and a transfer learning bridge. This multi-functional system enables the generation of synthetic training data for multiple different accents and dialects without requiring separate systems for each, reducing overall system complexity while maintaining versatility.
Solution Approach 2:
The first TTS system acts as an intermediary between the available training data and the target TTS system. It transforms standard speech data into synthetic representations that capture target speaker characteristics, mediating the transfer of voice qualities across different accents and dialects without direct human intervention in each case.
3Manufacturing precision
If professional voice actors are used for each accent/dialect combination, then high quality synthesized speech can be achieved, but the development cost and time increase significantly
Solution Approach 1:
Instead of recruiting professional voice actors for each accent/dialect combination, the system creates synthetic copies of speech data using the first TTS system. These synthetic representations capture the desired voice qualities and accent characteristics without requiring actual human performers, dramatically reducing data collection time and costs.
Solution Approach 2:
The system changes the parameters of speech synthesis by adjusting accent/dialect identifiers and speaker embedding vectors in the neural network. This allows the same underlying model to generate speech in multiple accents and dialects by modifying numerical parameters rather than requiring separate training datasets for each variation.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
A method (600) includes obtaining training data (10) including a plurality of training audio signals (102) and corresponding transcripts (106). Each training audio signal is spoken by a target speaker in a first accent/dialect. For each training audio signal, the method includes generating a training synthesized speech representation (202) spoken by the target speaker in a second accent/dialect and training a text-to-speech (TTS) system (300) based on the corresponding transcript and the training synthesized speech representation. The method also includes receiving an input text utterance (320) to be synthesized into speech in the second accent/dialect. The method also includes obtaining a speaker embedding (108) and an accent/dialect identifier (109) that identifies the second accent/dialect. The method also includes generating an output audio waveform (152) corresponding to a synthesized speech representation of the input text sequence that clones the voice of the target speaker in the second accent/dialect.