Multi-Speaker ASR Augmentation for Speaker-Change Robustness
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
End-to-end automatic speech recognition models, particularly RNN-T models, struggle with performance degradation when processing longer utterances, multiple speakers, or mismatched acoustic conditions, leading to failures in test recordings.
Innovation Solution
An on-the-fly data augmentation method that combines single speaker utterances to create multi-speaker training data, guided by a teacher model to map multi-speaker encoder embeddings to better performing single speaker representations, using a combined loss of RNN-T and mean squared error to enhance model robustness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If E2E ASR models are trained on clean, short duration, single speaker utterances, then they achieve state-of-the-art performance on matched test sets, but performance degrades when test utterances contain multiple speakers, longer durations, or mismatched acoustic conditions
Solution Approach 1:
The patent applies preliminary action by pre-processing training utterances to create augmented multi-speaker and long-form utterances before model training. Synthetic multi-speaker utterances are generated by concatenating single-speaker utterances with speaker change indicators, and the model is pre-exposed to these augmented data patterns during training, enabling it to handle speaker changes and longer durations without degradation
Solution Approach 2:
The patent changes key parameters of the training data including utterance duration (extending from short to long-form), speaker composition (from single to multi-speaker), and acoustic conditions (adding noise and channel variations). These parameter changes in the training set enable the model to learn robust representations that generalize to diverse test conditions
2Ease of manufacture
If training data consists of single speaker utterances, then model training is simpler and achieves high accuracy on single speaker tests, but the model fails on test recordings with multiple speakers and significant noise
Solution Approach 1:
The patent introduces speaker change indicators as intermediary elements that mark transition points between different speakers in concatenated utterances. These indicators serve as mediators that guide the model to recognize and adapt to speaker changes, bridging the gap between simple single-speaker training data and complex multi-speaker test scenarios
Solution Approach 2:
The patent performs preliminary data augmentation by synthesizing multi-speaker utterances from single-speaker recordings before training. This pre-processing step creates training examples with speaker changes and varied acoustic conditions, enabling the model to learn robust speaker-invariant features while maintaining training simplicity
Data Source
AI summary
Features of two or more single speaker utterances are concatenated together and corresponding labels of the two or more single speaker utterances are concatenated together. Single speaker acoustic embeddings for each of the single speaker utterances of the concatenated single speaker utterances are generated using a single speaker teacher encoder network. An enhanced model is trained on the concatenated single speaker utterances using a classification loss LCLASS and a representation similarity loss LREP, the representation similarity loss LREP defined to influence an embedding derived from the concatenated single speaker utterances, the influence being based on the single speaker acoustic embeddings derived from the single speaker teacher encoder network.