Streaming RNN-T Deliberation With Non-Autoregressive Alignment Refinement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing end-to-end automatic speech recognition (ASR) models face challenges in balancing streaming performance with improved accuracy, as deliberation models excel in accuracy but incur high latency and model size, making them unsuitable for streaming and on-device applications, while non-autoregressive models have lower latency but lower accuracy.
Innovation Solution
Implementing a non-autoregressive decoder that receives an initial alignment from a transducer decoder model and uses additional audio context to refine output labels through a cascading encoder structure, allowing simultaneous updates across multiple output positions without left-to-right constraints, enhancing accuracy with minimal latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deliberation models are used to improve speech recognition accuracy, then accuracy is improved, but latency and model size increase significantly
Solution Approach 1:
The model is segmented into two distinct components: a streaming RNN-T model for real-time transcription with low latency, and a non-autoregressive transformer model for accuracy improvement. This segmentation allows each component to specialize - the RNN-T handles streaming requirements while the transformer handles accuracy refinement, resolving the contradiction between speed and precision
Solution Approach 2:
The initial alignment from the RNN-T model serves as an intermediary that bridges the streaming and accuracy requirements. The non-autoregressive transformer uses this intermediate alignment to generate refined alignments without requiring autoregressive processing, thus improving accuracy without incurring the full latency penalty of traditional deliberation models
2Measurement precision
If deliberation models are used to improve speech recognition accuracy, then accuracy is improved, but model size increases
Solution Approach 1:
By segmenting the system into a compact RNN-T model and a non-autoregressive transformer, the overall model size is better managed than using a single large deliberation model. The RNN-T model remains lightweight for streaming, while the transformer only processes alignment data rather than full audio, reducing its required size
Solution Approach 2:
The non-autoregressive transformer serves multiple functions: it refines alignments, corrects errors, and improves accuracy all in a single pass. This multi-functionality reduces the need for multiple separate models or components that would increase overall system complexity
3Loss of time
If non-autoregressive decoding is used to reduce latency, then latency is reduced, but accuracy decreases
Solution Approach 1:
The system changes the processing parameters by using non-autoregressive decoding that processes all time steps simultaneously rather than sequentially. This parameter change in the decoding approach maintains low latency while the subsequent alignment refinement process improves accuracy without introducing autoregressive delays
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method (400) includes receiving an initial alignment (234) for a candidate hypothesis (120a) generated by a transducer decoder model (230) during a first pass. Here, the candidate hypothesis corresponds to a candidate transcription for an utterance (106) and the initial alignment for the candidate hypothesis includes a sequence of output labels. Each output label corresponds to a blank symbol or a hypothesized sub-word unit. The method also include receiving a subsequent sequence of audio encodings characterizing the utterance (222). During an initial refinement step, the method also includes generating a new alignment (324) for a rescored sequence of output labels using a non-autoregressive decoder (300). The non-autoregressive decoder is configured to receive the initial alignment for the candidate hypothesis and the subsequent sequence of audio encodings.