Self-Training ASR Model Using Oracle Hypothesis Loss
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automatic speech recognition (ASR) training methods face exposure bias, mismatch between training and inference objectives, and label bias, leading to performance degradation and inefficiencies in using both supervised and unsupervised data.
Innovation Solution
A self-training objective that boosts an oracle hypothesis from an n-best list of predicted hypotheses using word error rate (WER) versus ground-truth transcription, integrated with hard distillation for supervised training, allowing efficient use of both supervised and unsupervised data and reducing the gap between top-ranked and oracle hypothesis WER.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional ASR training methods are used, then training data can be processed, but exposure bias and mismatch between training and inference objectives occur leading to performance degradation
Solution Approach 1:
The system performs self-training by using its own n-best hypothesis lists to generate training data. The model processes acoustic frames, generates hypotheses, evaluates them against ground truth, and uses the results to retrain itself, creating a self-improving cycle that eliminates the need for external manual annotation and reduces dependence on perfectly labeled training data
Solution Approach 2:
The system implements feedback by evaluating the model's own predictions (n-best lists) against ground truth transcriptions, calculating word error rates, and using these error metrics to guide subsequent training iterations. This closed-loop feedback mechanism allows the model to learn from its mistakes and progressively improve accuracy
2Reliability
If n-best lists are generated and all hypotheses are processed, then comprehensive training data is obtained, but computational costs and processing time increase significantly
Solution Approach 1:
The system extracts only the necessary information from the n-best lists for training purposes. Instead of processing all hypotheses exhaustively, it selects and processes only the top-ranked hypotheses and their associated acoustic frames, extracting minimal sufficient data to achieve effective training while significantly reducing computational burden
Solution Approach 2:
The system applies partial action by processing only a subset of hypotheses from the n-best lists rather than all of them. It focuses computation on the most promising hypotheses (top-ranked and those with lowest word error rates) while leaving less likely candidates unprocessed, achieving adequate training quality with reduced computational resources
3Productivity
If top-ranked hypothesis is used for training, then training is simple and fast, but the gap between top-ranked and oracle hypothesis WER remains large reducing training effectiveness
Solution Approach 1:
The system merges the training approaches by combining the simplicity of top-ranked hypothesis training with the effectiveness of oracle hypothesis training. It processes both the top-ranked hypothesis and the oracle hypothesis (lowest WER from n-best list) together, integrating their strengths to achieve both efficiency and effectiveness in the training process
Data Source
AI summary
A method includes, for each training sample of a plurality of training samples, processing, using an RNN-T model, a corresponding sequence of acoustic frames to obtain an n-best list of speech recognition hypotheses, and, for each speech recognition hypothesis of the n-best list, determining a corresponding number of word errors relative to a corresponding ground-truth transcription. For a top-ranked hypothesis from the n-best list, the method includes determining a first loss based on the corresponding ground-truth transcription. The method includes identifying, as an oracle hypothesis, the speech recognition hypothesis from the n-best list having the smallest corresponding number of word errors relative to the corresponding ground-truth transcription, and determining a second loss for the oracle hypothesis based on the corresponding ground-truth transcription. The method includes determining a corresponding self-training combined loss based on the first and second losses, and training the model based on the corresponding self-training combined loss.