Two-Pass Speech Recognition With Shared Encoder for Streaming Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automated speech recognition systems face challenges in achieving low word error rate and latency while operating on-device, particularly when interacting with limited computational resources and requiring real-time streaming capabilities.
Innovation Solution
A two-pass automatic speech recognition model is employed, combining a recurrent neural network-transformer (RNN-T) decoder with a listen attend spell (LAS) decoder, sharing a common encoder to reduce model size and computational demands, and utilizing a combined loss function and minimum word error rate training to refine the model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a single-pass streaming ASR model is used, then real-time processing capability is improved, but speech recognition accuracy deteriorates
Solution Approach 1:
The ASR system is segmented into two distinct passes: a first-pass streaming ASR model that provides real-time processing capability, and a second-pass non-streaming ASR model that enhances accuracy. The first-pass model processes audio continuously with lower computational demands, while the second-pass model performs detailed analysis on selected hypotheses, thereby resolving the contradiction between speed and accuracy through functional division.
Solution Approach 2:
An intermediary component is introduced to bridge the first-pass and second-pass models. This intermediary selects candidate hypotheses from the first-pass output and prepares them for refinement by the second-pass model, enabling the system to leverage both streaming speed and high accuracy without requiring the entire system to operate at full computational capacity simultaneously.
2Measurement precision
If a two-pass ASR model with separate encoders is used, then speech recognition accuracy is improved, but computational resource consumption increases
Solution Approach 1:
The first-pass and second-pass ASR models share a common encoder component, merging their computational resources. This shared encoder extracts acoustic features and contextual information once, and both decoders utilize this processed information. This merging eliminates redundant computations that would occur with separate encoders, significantly reducing computational resource consumption while maintaining the accuracy benefits of the two-pass architecture.
Solution Approach 2:
The shared encoder is designed to serve multiple functions: it processes audio input for the streaming first-pass decoder and simultaneously provides feature representations for the non-streaming second-pass decoder. This multi-functionality allows a single computational component to support both real-time processing and high-accuracy recognition pathways, reducing overall resource requirements.
3Measurement precision
If a two-pass ASR model with separate encoders is used, then speech recognition accuracy is improved, but model size increases
Solution Approach 1:
The encoder components of the first-pass and second-pass ASR models are merged into a single shared encoder. This consolidation eliminates the need to store separate encoder parameters and weights, directly reducing model size. The shared encoder maintains the capability to support both streaming and non-streaming decoding pathways while occupying less memory space than two independent encoders would require.
Data Source
AI summary
Two-pass automatic speech recognition (ASR) models can be used to perform streaming on-device ASR to generate a text representation of an utterance captured in audio data. Various implementations include a first-pass portion of the ASR model used to generate streaming candidate recognition(s) of an utterance captured in audio data. For example, the first-pass portion can include a recurrent neural network transformer (RNN-T) decoder. Various implementations include a second-pass portion of the ASR model used to revise the streaming candidate recognition(s) of the utterance and generate a text representation of the utterance. For example, the second-pass portion can include a listen attend spell (LAS) decoder. Various implementations include a shared encoder shared between the RNN-T decoder and the LAS decoder.