Sequence Prediction Shift for ASR Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
End-to-end automatic speech recognition (ASR) systems using CTC loss function suffer from time delay between acoustic features and output symbols due to the lack of frame-level alignment supervision, leading to increased latency in streaming ASR applications.
Innovation Solution
A novel training technique that shifts the sequence of predictions with respect to the input sequence of observations, allowing the model to output predictions at appropriate timing by adjusting the length of the prediction sequence to match the input sequence, thereby reducing latency and improving timing alignment without requiring frame-level forced alignments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If CTC models are trained with training samples having different lengths of acoustic features and output symbols without frame-level alignment supervision, then the model can operate faster and reduce power consumption, but time delay between acoustic features and output symbols increases
Solution Approach 1:
The patent applies preliminary action by shifting the prediction sequence forward in time during training. Specifically, the prediction sequence is shifted by a predetermined amount (e.g., 1 frame) so that predictions are made earlier relative to the acoustic features, effectively pre-compensating for the inherent time delay in CTC models without requiring frame-level alignment supervision
2Loss of time
If frame-level forced alignments are applied between acoustic features and output symbols, then time delay is reduced, but iterative steps are required prior to model training
Solution Approach 1:
The patent extracts and removes the complex iterative frame-level alignment preparation steps from the training process. Instead of requiring forced alignments, the invention directly applies a simple sequence shifting operation to the predictions, eliminating the need for preliminary alignment iterations while still achieving time delay reduction
Solution Approach 2:
The patent changes the temporal parameter of the prediction sequence by applying a fixed shift amount rather than requiring iterative alignment. This parameter change approach transforms the complex alignment problem into a simple temporal offset adjustment, significantly reducing training complexity
3Loss of time
If neural network topologies are modified to reduce time delay, then latency may be reduced, but the system complexity increases
Solution Approach 1:
The patent uses copying by creating a shifted version of the existing prediction sequence rather than modifying the neural network topology itself. The same model architecture is used, but the output sequence is temporally shifted to achieve latency reduction, avoiding the need for complex architectural changes
Data Source
AI summary
A technique for training a model is disclosed. A training sample including an input sequence of observations and a target sequence of symbols having length different from the input sequence of observations is obtained. The input sequence of observations is fed into the model to obtain a sequence of predictions. The sequence of predictions is shifted by an amount with respect to the input sequence of observations. The model is updated based on a loss using a shifted sequence of predictions and the target sequence of the symbols.