Speech Synthesis and Recognition Using Synthesized Paired Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech synthesis and recognition technologies require large amounts of training data, which is scarce for over 6,000 languages, leading to high data collection costs and low accuracy in low-resource languages.
Innovation Solution
A method to train Text to Speech (TTS) and Automatic Speech Recognition (ASR) models using minimal paired data, pre-training on rich-resource languages, and iteratively optimizing models through dual transformation and knowledge distillation to enhance accuracy for low-resource languages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a large amount of speech training data is collected to ensure model accuracy, then the accuracy of TTS and ASR models is improved, but the data collection cost and time increase significantly
Solution Approach 1:
The patent applies pre-training on rich-resource languages before fine-tuning on target language data. The pre-trained models acquire general speech processing capabilities beforehand, so that when fine-tuning on limited target language data, the models can achieve high accuracy with much less data collection time compared to training from scratch.
Solution Approach 2:
The patent uses knowledge distillation where a pre-trained teacher model generates synthesized training data and soft labels for student models. This copying of knowledge from the teacher model allows student models to learn effective representations without requiring extensive manual annotated data, reducing data collection requirements while maintaining accuracy.
2Measurement precision
If a large amount of speech training data is collected to ensure model accuracy, then the accuracy of TTS and ASR models is improved, but the data collection cost increases
Solution Approach 1:
The patent uses knowledge distillation where a pre-trained teacher model generates synthesized training data and soft labels for student models. This copying of knowledge from the teacher model allows student models to learn effective representations without requiring extensive manual annotated data, reducing data collection costs while maintaining accuracy.
Solution Approach 2:
The patent employs self-training where the model generates its own training data through synthesis and recognition processes. The model uses its own outputs to create training examples, eliminating the need for expensive manual annotation of large datasets, thereby reducing data collection costs while achieving high accuracy.
3Quantity of substance
If TTS and ASR models are trained from scratch on limited data, then data collection cost is reduced, but model accuracy deteriorates
Solution Approach 1:
The patent applies pre-training on rich-resource languages before fine-tuning on target language data. The pre-trained models acquire general speech processing capabilities beforehand, so that when fine-tuning on limited target language data, the models can achieve high accuracy with much less data collection time compared to training from scratch.
Solution Approach 2:
The patent introduces a teacher model as an intermediary that transfers knowledge to student models through knowledge distillation. The teacher model, pre-trained on abundant data, acts as a mediator that imparts learned representations to student models trained on limited data, enabling high accuracy without requiring extensive data collection.
4Measurement precision
If pre-training on rich-resource languages is performed followed by fine-tuning on target language data, then model accuracy for low-resource languages is improved, but training time increases
Solution Approach 1:
The patent applies pre-training on rich-resource languages before fine-tuning on target language data. The pre-trained models acquire general speech processing capabilities beforehand, so that when fine-tuning on limited target language data, the models can achieve high accuracy with much less data collection time compared to training from scratch.
Data Source
AI summary
Implementations of the subject matter described herein provide a solution for speech synthesis and speech recognition. In this solution, a Text to Speech (TTS) model and an Automatic Speech Recognition (ASR) model supporting at least one language are obtained. The TTS model and the ASR model are adjusted, based on a first set of paired data in a target language, to support the target language. The TTS model is optimized based on the first set of paired data and a first set of synthesized paired data in the target language generated by the ASR model while the ASR model is optimized based on the first set of paired data and a second set of synthesized paired data in the target language generated by the TTS model. As such, the solution can provide TTS and ASR models with high accuracy for languages lacking training data by using less training data.