Streaming ASR Consistency Training With Real and Synthetic Speech
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning-based ASR models tend to overfit training data, leading to difficulties in generalizing unseen data, especially when training data is limited, and synthesized speech impacts training differently than human speech due to mismatches in data characteristics.
Innovation Solution
Train ASR models on both real/human and synthesized speech representations of the same utterance, incorporating a consistent loss term to promote consistent predictions across both types of speech, and apply data augmentation techniques to enhance model robustness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If ASR models are trained on larger training datasets to improve accuracy, then model accuracy is improved, but training data availability and cost increase
Solution Approach 1:
The patent creates synthetic speech copies by converting text to speech using TTS models. These synthetic speech samples serve as artificial training data that mimics real human speech characteristics. By copying and synthesizing speech from text sources, the system expands the effective training dataset without requiring proportional increases in actual human-recorded speech data, thus improving model accuracy while controlling the quantity of required human training data.
Solution Approach 2:
The patent applies data augmentation techniques that modify speech samples by changing parameters such as adding noise, adjusting speed, modifying pitch, and applying various acoustic transformations. These parameter changes create diverse variations of the same speech content, effectively increasing the training data volume and improving model generalization without requiring additional human speech recordings.
2Quantity of substance
If synthesized speech is used to augment training data, then training data volume increases, but model generalization performance deteriorates due to mismatches in data characteristics
Solution Approach 1:
The patent applies different processing and weighting strategies to different portions of the training data. Synthetic speech samples undergo specific augmentation techniques and are combined with real speech data in controlled proportions. The system adjusts the characteristics of synthetic speech locally to better match real speech distributions, ensuring that the augmented portions contribute positively to generalization rather than degrading it.
Solution Approach 2:
The patent creates a composite training dataset that combines real human speech data with synthetic speech data in specific configurations. This composite approach allows the system to leverage the advantages of both data types while mitigating their individual weaknesses. The blended dataset maintains the natural characteristics of human speech while incorporating the structured diversity of synthetic speech, improving overall model generalization.
3Reliability
If data augmentation techniques are applied to enhance model robustness, then model robustness is improved, but training complexity increases
Solution Approach 1:
The patent divides the data augmentation process into separate, modular stages. Text-to-speech conversion, acoustic transformations, noise addition, and other augmentation techniques are applied as distinct processing steps. This segmentation allows each augmentation operation to be independently controlled and optimized, reducing the overall training complexity while maintaining robustness improvements.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
A method (600) for training a speech recognition model (200) includes receiving a set of training utterance pairs (302) each including a non-synthetic speech representation (304) and a synthetic speech representation (306) of a same corresponding utterance (106). The method also includes determining a consistent loss term (352) for the corresponding training utterance pair based on a first probability distribution (311) over possible non-synthetic speech recognition hypotheses generated for the corresponding non- synthetic speech representation and a second probability distribution (312) over possible synthetic speech recognition hypotheses generated for the corresponding synthetic speech representation. The first and second probability distributions are generated for output by the speech recognition model. The method also includes updating parameters of the speech recognition model based on the consistent loss term.