Expressive Neural TTS Training Using Expressivity-Scored Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text-to-speech (TTS) synthesis systems lack the ability to generate expressive speech that conveys emotional information and sounds natural and human-like, necessitating improved training methods to enhance vocal expressiveness.
Innovation Solution
A neural network-based TTS system is trained using a two-phase training process, utilizing a first sub-dataset with lower expressivity scores followed by a second sub-dataset with higher expressivity scores to generate expressive speech, and an expressivity score calculation method to evaluate audio samples accurately.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional TTS training methods are used, then the system can generate basic speech, but the speech lacks vocal expressiveness and emotional information
Solution Approach 1:
The training dataset is segmented into multiple sub-datasets with different expressivity score ranges. The training process is divided into phases, where each phase uses a specific sub-dataset. This segmentation allows the system to progressively learn expressive speech patterns without overwhelming complexity in a single training run.
Solution Approach 2:
The expressivity score serves as a key parameter for selecting and weighting training samples. By changing the parameter range of expressivity scores used in different training phases (from lower to higher ranges), the system adapts its learning focus to progressively improve vocal expressiveness while managing training complexity.
2Reliability
If a single training dataset is used, then the training process is simple, but the generated speech does not achieve high expressiveness
Solution Approach 1:
The expressivity score is calculated for all training samples beforehand, and samples are pre-organized into sub-datasets based on their expressivity score ranges. This preliminary action enables efficient sampling during training without requiring complex real-time evaluations, thus maintaining training efficiency while achieving high expressiveness.
Solution Approach 2:
Instead of using the entire training dataset uniformly, the method selectively uses specific portions (sub-datasets with higher expressivity scores) in later training phases. This partial action focuses computational resources on the most expressive samples, improving speech expressiveness without proportionally increasing training cost.
3Reliability
If uniform sampling from training data is used, then training is computationally efficient, but the system cannot capture highly expressive speech patterns
Solution Approach 1:
The method changes the sampling parameter from uniform distribution to expressivity-score-based distribution. By adjusting the expressivity score range parameter across training phases (from lower to higher ranges), the system captures increasingly expressive speech patterns while using a relatively simple scoring mechanism rather than complex data selection processes.
Data Source
AI summary
A method includes receiving text and inputting the received text in a prediction network. The method further includes generating, using the prediction network, speech data. The prediction network comprises a neural network that is trained to generate expressive speech data from text. The neural network is trained by: receiving a first training dataset comprising audio data and corresponding text data; acquiring a respective expressivity score for each audio sample of the audio data; selecting, from the first training dataset, a first subset of training data based on the respective expressivity scores of the audio data in the first training dataset; generating, for the first subset of training data, prediction audio data for the corresponding text data; and comparing the prediction audio data to the audio data of the first subset of training data.