Chunk-Wise Attention for Longform ASR Generalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning-based ASR models tend to overfit training data, leading to difficulties in generalizing unseen data, especially when the training data is not extensive, and using unpaired text data to train ASR models presents challenges in combining speech and text modalities due to interference and capacity limitations.
Innovation Solution
A training process that includes using multilingual unspoken textual utterances, un-transcribed non-synthetic speech utterances, and transcribed non-synthetic speech utterances to pre-train an audio encoder with contrastive and alignment loss terms, employing a stack of self-attention layers and auxiliary decoders to improve generalization across domains.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep learning-based ASR models are trained with extensive parameters, then accuracy improves, but overfitting increases and generalization to unseen data deteriorates
Solution Approach 1:
The training data is segmented into multiple modalities (speech-only, text-only, and paired speech-text) with different weighting schemes. The loss function is segmented into multiple components that are optimized at different stages, allowing the model to learn from diverse data types without overfitting to any single source
Solution Approach 2:
The model employs dynamic parameter adjustment during training, including learning rate scheduling, weight decay, and adaptive mixing ratios between different loss terms. The training configuration allows flexible adjustment of hyperparameters to balance accuracy and generalization based on available data
2Quantity of substance
If unpaired text data is used to train ASR models, then training data quantity increases, but interference between speech and text modalities and capacity limitations worsen
Solution Approach 1:
An alignment model serves as an intermediary between unpaired text data and the ASR model. The alignment model generates alignment outputs that map text to speech temporal structure, enabling the ASR model to learn from unpaired text without direct speech-text pairing. This intermediary layer resolves the interference problem by providing a bridge between modalities
Solution Approach 2:
The alignment model performs preliminary processing of unpaired text data before it reaches the ASR model. By pre-aligning text with speech temporal patterns and generating alignment outputs, the system prepares the text modality in advance, reducing the complexity of real-time modality integration during ASR training
Data Source
AI summary
A method includes receiving training data including a corpus of multilingual unspoken textual utterances, a corpus of multilingual un-transcribed non-synthetic speech utterances, and a corpus of multilingual transcribed non-synthetic speech utterances. For each un-transcribed non-synthetic speech utterance, the method includes generating a target quantized vector token and a target token index, generating contrastive context vectors from corresponding masked audio features, and deriving a contrastive loss term. The method also includes generating an alignment output, generating a first probability distribution over possible speech recognition hypotheses for the alignment output, and determining an alignment output loss term. The method also includes generating a second probability distribution over possible speech recognition hypotheses and determining a non-synthetic speech loss term. The method also includes pre-training an audio encoder based on the contrastive loss term, the alignment output loss term, and the non-synthetic speech loss term.