Neural Network Multi-Speaker Speech Synthesis via Adversarial Vector Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional multi-speaker speech synthesis models require extensive data for training, making it time-consuming and costly to adapt to new speakers, especially in individual or small business environments.
Innovation Solution
A method using an artificial neural network that generates a speech learning model for new speakers by predicting a new speaker vector based on the most similar pre-trained speaker vector, utilizing adversarial training and cosine similarity calculations to minimize data requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a conventional multi-speaker speech synthesis model is used to generate speech for a new speaker, then the speech quality and naturalness can be maintained, but a large amount of speech data (tens of minutes per speaker) is required for training
Solution Approach 1:
The system performs preliminary action by pre-training the speech synthesis model with a diverse set of speaker data beforehand. When a new speaker needs to be synthesized, the model leverages this pre-learned knowledge from multiple speakers to generate speech for the new speaker with minimal additional data requirements, thus resolving the contradiction between maintaining speech quality and reducing data quantity.
Solution Approach 2:
The speech synthesis model is designed with multi-functionality to handle multiple speakers within a single unified model. By learning common speech patterns and characteristics from diverse speakers during pre-training, the model can universally generate natural speech for any new speaker without requiring separate models or extensive speaker-specific data, thereby maintaining speech quality while reducing the data quantity needed per speaker.
2Adaptability or versatility
If speech data is collected from multiple speakers to train a multi-speaker model, then the model can generate diverse voices, but the time and cost for data collection increase significantly
Solution Approach 1:
The system performs preliminary action by conducting comprehensive multi-speaker data collection and model training in advance. This pre-training phase captures diverse voice characteristics and speech patterns, enabling the model to quickly adapt to new speakers later without requiring time-consuming data collection for each individual speaker, thus resolving the contradiction between voice diversity and data collection time.
3Measurement precision
If extensive fine-tuning is performed for each new speaker, then the speech synthesis accuracy improves, but the processing time and computational resources increase
Solution Approach 1:
The system performs preliminary action by pre-training the model with multi-speaker data to learn general speech patterns and characteristics. This pre-learning phase enables the model to achieve good synthesis accuracy for new speakers with minimal fine-tuning, thus resolving the contradiction between synthesis accuracy and processing speed by doing the heavy learning work beforehand.
Solution Approach 2:
The system utilizes parameter changes by leveraging speaker embedding vectors that capture essential speaker characteristics. When adapting to a new speaker, the model adjusts these embedding parameters based on minimal speaker-specific data, allowing quick adaptation while maintaining high synthesis accuracy, thereby resolving the contradiction between accuracy and processing speed.
Data Source
AI summary
According to an aspect, a method of synthesizing a multi-speaker speech using an artificial neural network, the method comprises generating a speech learning model for a plurality of users based on speech data of the plurality of users, generating a first speaker vector for speech data of a new speaker and a plurality of second speaker vectors for speech data of the plurality of users using a speaker recognition model, determining a third speaker vector having a highest correlation with the first speaker vector among the plurality of second speaker vectors based on a present criterion and predicting a new speaker vector of the new user based on the third speaker vector and the first speaker vector using an adversarial training method.