Token-wise Training for Attention-Based Speech Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Attention-based end-to-end (A-E2E) automatic speech recognition (ASR) systems face challenges in efficiently correcting early decoding errors due to the dependency of output tokens, leading to unstable convergence and inefficiencies in error propagation.
Innovation Solution
A token-wise training (TWT) scheme that focuses on correcting the first wrong token in a hypothesis, combined with a beam search approach (TWTiB) that selects hypotheses with the longest correct prefix, allowing for more stable and flexible training with various loss functions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If standard early update method is applied to A-E2E ASR models, then training efficiency is improved by focusing on first wrong token, but gradient assignment becomes suboptimal as nonzero gradients are assigned to beginning correct tokens
Solution Approach 1:
The patent extracts and isolates the gradient computation to only the first wrong token position, removing the unnecessary gradient assignments to correct tokens. This is achieved by masking the loss tensor to retain only the loss at the first error position, thereby improving training efficiency while maintaining precise gradient assignment.
Solution Approach 2:
The patent applies local quality by making the gradient assignment position-dependent: only the first wrong token position receives non-zero gradients, while all other positions (including correct tokens) receive zero gradients. This localized gradient assignment optimizes both efficiency and precision by focusing computational resources exactly where needed.
2Measurement precision
If n-best hypotheses-based loss estimation is used, then loss estimation accuracy is improved, but computational complexity increases compared to sampling-based methods
Solution Approach 1:
The patent applies partial action by generating only a limited number of n-best hypotheses (top-k) rather than exhaustively searching all possible hypotheses. This partial exploration provides sufficient loss estimation accuracy for practical applications while avoiding the prohibitive computational complexity of complete hypothesis enumeration.
3Stability of the object's composition
If softmax smoothing is applied during n-best hypotheses generation, then loss estimation stability is improved, but model convergence speed may be reduced
Solution Approach 1:
The patent applies parameter changes by dynamically adjusting the temperature parameter in softmax smoothing during training. By modifying the temperature parameter, the system can control the degree of smoothing: higher temperatures provide more stability in loss estimation, while lower temperatures accelerate convergence. This parameter tuning allows balancing stability and speed based on training progress.
Data Source
AI summary
A method of attention-based end-to-end (A-E2E) automatic speech recognition (ASR) training, includes performing cross-entropy training of a model, based on one or more input features of a speech signal, determining a posterior probability vector at a time of a first wrong token among one or more output tokens of the model of which the cross-entropy training is performed, and determining a loss of the first wrong token at the time, based on the determined posterior probability vector. The method further includes determining a total loss of a training set of the model of which the cross-entropy training is performed, based on the determined loss of the first wrong token, and updating the model of which the cross-entropy training is performed, based on the determined total loss of the training set.