Speech Embedding Augmentation for Robust ASR Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional automatic speech recognition systems face challenges in accurately recognizing unknown speech signals due to insufficient data augmentation in the embedding space, leading to reduced performance on unseen data.
Innovation Solution
A method and system for augmented speech embeddings based on replacing a subset of randomly selected embeddings with Gaussian noise during training, which simulates partial loss of speech segments, forcing the network to learn with incomplete data without requiring additional data or significant computational overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional data augmentation schemes are used, then training data diversity is improved, but computational overhead and complexity increase
Solution Approach 1:
The patent applies SpecAugment which operates in the spectral domain (Mel-spectrogram) rather than the raw time-domain signal. By transforming the data into a different representation space and applying masks in frequency and time dimensions, the system achieves data augmentation with minimal computational overhead compared to traditional time-domain augmentation methods
Solution Approach 2:
The patent changes the parameters of existing data by applying random masks to Mel-spectrograms during training. This involves modifying the frequency and time parameters of the spectral representation, creating augmented versions of the training data without requiring additional recording or complex processing
2Reliability
If more data augmentation techniques are applied, then ASR performance on unseen data is improved, but model training complexity increases
Solution Approach 1:
The patent applies data augmentation techniques during the training phase (preliminary action) rather than during deployment. By pre-training the model with augmented data including SpecAugment and embedding noise augmentation, the system prepares the model to handle unseen and noisy data without adding complexity to the inference stage
Solution Approach 2:
The patent applies multiple layers of augmentation (SpecAugment with time/frequency masks, plus embedding noise augmentation) that may seem excessive but are designed to work together synergistically. The combination of these augmentation techniques provides robust performance improvement without requiring equally complex architectural changes
3Adaptability or versatility
If data augmentation creates additional transformed data, then training diversity is improved, but processing time and computational resources increase
Solution Approach 1:
The patent creates augmented training examples by copying and transforming existing training data through SpecAugment and embedding noise augmentation. Rather than collecting new data, the system generates multiple augmented versions of existing recordings by applying random masks and noise in the spectral domain, significantly reducing processing time compared to traditional augmentation methods
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach significantly reduces Word Error Rate (WER) on unseen data, particularly in mismatched speech scenarios, by enhancing network performance through embedding-level augmentation, even when combined with existing data augmentation techniques like SpecAugment.
Implementation Method 1
generating a transformed speech signal corresponding to each of the plurality of speech frames by applying Fourier Transform (FT) on each of the plurality of speech frames
Implementation Method 2
generating a log Mel spectrogram associated with each of the plurality of speech frames by passing the transformed speech signal corresponding to each of the plurality of speech frames through a Mel filter bank
Implementation Method 3
generating a plurality of augmented speech embeddings by replacing the set of random speech embeddings with a randomly generated Gaussian noise
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
Though several data augmentation techniques have been explored in the signal or feature space, very few studies have explored augmentation in the embedding space for Automatic Speech Recognition (ASR). The outputs of the hidden layers of a neural network can be seen as different representations or projections of the features. The augmentations performed on the features may not necessarily translate into augmentation of the different projections of the features as obtained from the output of the hidden layers. To overcome the challenges of the conventional approaches, embodiments herein provide a method and system for augmented speech embeddings based automatic speech recognition. The present disclosure provides an augmentation scheme which works on the speech embeddings. The augmentation works by replacing a set of randomly selected embeddings by noise during training. It does not require additional data, works online during training, and adds very little to the overall computational cost.