Voice Conversion Training for Consistent ASR Generalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning-based automatic speech recognition (ASR) models tend to overfit training data, leading to difficulties in generalizing unseen data, especially when the training data is limited and diverse in speaker identities and characteristics, and synthesized speech impacts ASR training differently than human speech.
Innovation Solution
Implement voice conversion to manipulate speaker traits and use a consistent loss term to train ASR models on both real and synthesized speech representations of the same utterance, promoting consistent predictions across both types of speech.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If ASR models are trained on limited training data, then training time and data collection costs are reduced, but the models tend to overfit and have difficulties generalizing to unseen data from diverse speaker identities and characteristics
Solution Approach 1:
The patent uses voice conversion technology to generate synthetic speech copies that mimic the acoustic characteristics of target speakers while preserving the linguistic content. By copying speaker traits through voice conversion models trained on diverse speaker data, the system creates artificial training examples that expand the effective training dataset without requiring actual recordings from every possible speaker, thereby improving generalization capability while avoiding the need to collect extensive training data from all speaker demographics.
2Measurement precision
If ASR models are trained on larger training datasets with diverse speaker identities, then generalization accuracy is improved, but the complexity of data collection and processing increases
Solution Approach 1:
The patent introduces voice conversion models as intermediary systems that bridge the gap between limited available training data and the need for diverse speaker representation. These intermediary models synthesize speech samples with target speaker characteristics, eliminating the need for direct data collection from every speaker群体. The voice conversion models act as mediators that transform existing speech data into diverse training examples, significantly reducing data collection and processing complexity while maintaining high recognition accuracy.
3Quantity of substance
If synthesized speech is used to augment training data, then the volume of training data is increased, but the gap between synthesized and human speech may cause inconsistent predictions
Solution Approach 1:
The patent applies parameter changes by adjusting the voice conversion models to progressively reduce the acoustic gap between synthesized and human speech. By modifying parameters such as spectral characteristics, prosody, and timbre in the voice conversion process, the system generates synthetic speech that more closely resembles natural human speech. This parameter optimization ensures that synthesized training data produces consistent predictions with human speech data, maintaining reliability while expanding training data volume.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
A method includes receiving a set of training utterances each including a non-synthetic speech representation (304), and for each training utterance, generating a corresponding synthetic speech representation (306) by a voice conversion model (400). The non-synthetic speech representation and the synthetic speech representation form a corresponding training utterance pair (302). At each of a plurality of output steps for each training utterance pair, the method also includes generating, for output by a speech recognition model (200), a first probability distribution (311) over possible non-synthetic speech recognition hypotheses for the non-synthetic speech representation and a second probability distribution (312) over possible synthetic speech recognition hypotheses for the synthetic speech representation. The method also includes determining a consistent loss term for the corresponding training utterance pair based on the first and second probability distributions and updating parameters of the speech recognition model based on the consistent loss term.