Multi-Speaker Speech Synthesis with Single-Speaker Pre-Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-speaker speech synthesis models require large datasets and extensive training times, making them inefficient for adapting to new speakers.
Innovation Solution
A method involving pre-training with a single-speaker dataset and fine-tuning with a multi-speaker dataset, using speaker similarity metrics to select an initial embedding value, and applying a target speech dataset to convert into a mel spectrogram.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a multi-speaker training dataset is used to train a speech synthesis model, then the model can synthesize speeches with multiple speakers, but a large amount of training data and extensive training time are required
Solution Approach 1:
The training process is segmented into two distinct phases: pre-training with a single-speaker dataset to establish foundational speech synthesis capabilities, and fine-tuning with a multi-speaker dataset to enable multi-speaker synthesis. This segmentation allows the model to achieve multi-speaker capability without requiring the entire training process to use large multi-speaker datasets, thereby reducing the overall data quantity requirement while maintaining adaptability across multiple speakers
2Adaptability or versatility
If a multi-speaker training dataset is used to train a speech synthesis model, then the model can synthesize speeches with multiple speakers, but extensive training time is required
Solution Approach 1:
The model performs preliminary pre-training with a single-speaker dataset before the actual multi-speaker fine-tuning. This preliminary action establishes the foundational speech synthesis capabilities and acoustic modeling in advance, so that when multi-speaker data is introduced for fine-tuning, the model only needs to adapt to different speakers rather than learning speech synthesis from scratch. This significantly reduces the training time required to achieve multi-speaker capability
3Reliability
If speaker embedding is trained for multi-speaker synthesis, then synthesis performance improves, but the complexity of the training process increases
Solution Approach 1:
The training process for speaker embedding is segmented into pre-training with single-speaker data to learn fundamental acoustic patterns, and fine-tuning with multi-speaker data to learn speaker-specific characteristics. This segmentation simplifies the training process complexity by breaking down the complex task of learning both speech synthesis and speaker differentiation into two manageable phases, while still achieving high synthesis performance through the cumulative learning from both phases
Data Source
AI summary
An exemplary embodiment of the present disclosure is a speech synthesis method based on multi-speaker training dataset of a speech synthesis apparatus including pre-training a speech synthesis model using a previously stored neural network with a single speaker training dataset with most spoken sentences, among training datasets of a plurality of speakers, fine-tuning the pre-trained speech synthesis model with a training dataset of a plurality of speakers, and applying a target speech dataset to the fine-tuned speech synthesis model to be converted into a mel spectrogram.