Two-Pass ASR Model with Shared Encoder for On-Device Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current speech recognition systems face challenges in achieving low word error rate and latency, especially in on-device applications where resources are limited and user interaction requires real-time decoding and accurate recognition of diverse pronunciations.
Innovation Solution
A two-pass automatic speech recognition (ASR) model is implemented, combining a streaming RNN-T decoder with a non-streaming LAS decoder sharing a common encoder, which reduces model size and computational resources, and incorporates an additional encoder to adapt outputs for improved LAS processing, along with techniques to mitigate endpointer latency and enhance robustness to different accents and domains.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a two-pass ASR model with shared encoder is used, then model size and computational resources are reduced, but recognition accuracy may deteriorate
Solution Approach 1:
The ASR model is divided into two distinct passes: first-pass streaming recognition for real-time transcription and second-pass non-streaming recognition for final accurate transcription. This segmentation allows each pass to be optimized for its specific function while sharing the encoder, thus maintaining accuracy without requiring a complete separate model.
Solution Approach 2:
The encoder is merged and shared between both first-pass and second-pass decoders. This combining approach reduces the overall model size and computational resources by eliminating redundant encoder computations, while the separate decoders maintain their specialized functionality for different recognition scenarios.
2Loss of time
If streaming first-pass recognition is used, then latency is reduced, but word error rate increases
Solution Approach 1:
The recognition process is segmented into two temporal phases: immediate streaming recognition that provides low-latency preliminary results, and subsequent non-streaming recognition that refines accuracy. This allows the system to deliver fast initial responses while maintaining high final accuracy through the second pass.
Solution Approach 2:
The first-pass streaming recognition performs preliminary transcription during speech playback, providing early results with acceptable accuracy. The second-pass then performs preliminary refinement on the same audio data after playback completes, correcting errors and improving word error rate without adding significant latency.
3Measurement precision
If dedicated encoders are used for RNN-T and LAS decoders, then recognition accuracy improves, but computational resources increase
Solution Approach 1:
A single encoder is merged and shared between both RNN-T and LAS decoders. The encoder processes audio data once and generates features that are utilized by both decoder types, eliminating redundant computation. This merging strategy significantly reduces computational resources and energy consumption while maintaining the specialized capabilities of each decoder through their respective architectures.
Data Source
AI summary
Two-pass automatic speech recognition (ASR) models can be used to perform streaming on-device ASR to generate a text representation of an utterance captured in audio data. Various implementations include a first-pass portion of the ASR model used to generate streaming candidate recognition(s) of an utterance captured in audio data. For example, the first-pass portion can include a recurrent neural network transformer (RNN-T) decoder. Various implementations include a second-pass portion of the ASR model used to revise the streaming candidate recognition(s) of the utterance and generate a text representation of the utterance. For example, the second-pass portion can include a listen attend spell (LAS) decoder. Various implementations include a shared encoder shared between the RNN-T decoder and the LAS decoder.