Augmented Speech Embeddings for Unseen-Data ASR Robustness
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data augmentation techniques for Automatic Speech Recognition (ASR) systems fail to effectively enhance performance on unseen data, particularly in recognizing unknown speech signals, as they do not adequately translate augmentations in the feature space to the embedding space.
Innovation Solution
A method and system for augmented speech embeddings that replace a subset of randomly selected embeddings with Gaussian noise during training, simulating partial loss of speech segments, which forces the network to learn with incomplete data without requiring additional data or significant computational overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional data augmentation techniques (Speed Perturbation, Vocal Tract Length Perturbation, Spec Augment) are used to improve ASR performance on unseen data, then the diversity of training data is improved, but the accuracy in recognizing unknown speech signals deteriorates
Solution Approach 1:
The patent transitions from conventional feature-space augmentation (spectral domain) to embedding-space augmentation. By operating on the embedding representations produced by the encoder rather than the raw spectral features, the system introduces a new dimension of augmentation that directly affects the semantic content of speech representations, enabling better generalization to unknown speakers while maintaining recognition accuracy.
Solution Approach 2:
The patent changes the parameter space where augmentation occurs from the spectral domain to the embedding domain. By modifying the embedding parameters (replacing random embeddings with noise-corrupted embeddings) rather than spectral parameters, the system achieves augmentation that preserves phonetic information while introducing variability in speaker representations, thereby improving both adaptability and precision.
2Reliability
If data augmentation schemes create additional data or transform existing data to improve training data diversity, then ASR robustness is improved, but computational overhead and data processing complexity increase
Solution Approach 1:
The patent extracts the augmentation operation from the data generation pipeline and places it in the embedding processing stage. Instead of creating multiple versions of training data through complex transformations, the system extracts the encoder's embedding representations and applies simple noise corruption to these embeddings, significantly reducing computational complexity while maintaining robustness improvements.
Solution Approach 2:
The patent uses a simplified copying approach where rather than generating entirely new augmented data samples through complex transformations, it copies the existing training embeddings and introduces noise to create augmented embedding versions. This copying strategy reduces the computational burden of data generation while still achieving the desired augmentation effect.
3Object-affected harmful factors
If speech signals are corrupted by artificial noise during training to improve noise robust ASR, then performance in noisy conditions is improved, but the model's ability to handle complete speech signals deteriorates
Solution Approach 1:
The patent applies local quality by selectively corrupting only a subset of embeddings with noise rather than corrupting the entire speech signal. This localized augmentation approach allows the model to learn robustness to missing/embedding corruption without being overwhelmed by excessive noise, thereby maintaining both noise robustness and recognition accuracy for complete signals.
Solution Approach 2:
The patent applies partial action by replacing only a random subset of embeddings with noise-corrupted versions rather than corrupting all embeddings. This partial corruption strategy provides sufficient noise robustness training while preserving enough clean signal information to maintain high recognition accuracy, avoiding the excessive action that would degrade performance.
Data Source
AI summary
Though several data augmentation techniques have been explored in the signal or feature space, very few studies have explored augmentation in the embedding space for Automatic Speech Recognition (ASR). The outputs of the hidden layers of a neural network can be seen as different representations or projections of the features. The augmentations performed on the features may not necessarily translate into augmentation of the different projections of the features as obtained from the output of the hidden layers. To overcome the challenges of the conventional approaches, embodiments herein provide a method and system for augmented speech embeddings based automatic speech recognition. The present disclosure provides an augmentation scheme which works on the speech embeddings. The augmentation works by replacing a set of randomly selected embeddings by noise during training. It does not require additional data, works online during training, and adds very little to the overall computational cost.