Two-Pass Speech Recognition With Deliberation for Low Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing end-to-end speech recognition systems face challenges in achieving high accuracy and low latency, particularly in mobile environments, due to the limitations of recurrent neural network transducer (RNN-T) models and non-streaming listen-attend-spell (LAS) models, which hinder real-time voice transcription and on-device deployment.
Innovation Solution
A two-pass speech recognition system is implemented, comprising a first-pass RNN-T network followed by a second-pass LAS network, enhanced with a deliberation network that attends to both acoustics and first-pass text hypotheses, utilizing shared encoders and joint training to improve accuracy and reduce latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If a single-pass RNN-T model is used for speech recognition, then the system achieves low latency and streaming capability, but the Word Error Rate is high
Solution Approach 1:
The speech recognition system is divided into two distinct passes: a first-pass RNN-T decoder that performs streaming recognition with low latency, and a second-pass LAS decoder that performs accurate but non-streaming recognition. The system segments the recognition task temporally, allowing each pass to optimize for its specific function while collectively achieving both low latency and high accuracy through the merging of their output hypotheses.
2Measurement precision
If a large conventional model is used to achieve high accuracy, then the Word Error Rate is reduced, but the model size and computational complexity increase, preventing on-device deployment
Solution Approach 1:
The system merges the outputs of two separate, relatively small models (RNN-T and LAS decoders) to achieve the accuracy of a large conventional model. By combining the hypothesis outputs from both decoders and selecting the best result, the system attains high recognition accuracy without requiring a single large model, thus enabling deployment on mobile devices with limited computational resources.
3Measurement precision
If a two-pass system with separate encoders is used to improve accuracy, then the Word Error Rate is reduced, but the computational overhead and latency increase
Solution Approach 1:
The encoder is designed as a universal, shared component that serves both the RNN-T decoder and the LAS decoder in the two-pass system. This multi-functional encoder processes acoustic features and generates representations that are utilized by both decoders, eliminating the need for separate encoders and thereby reducing computational overhead while maintaining the accuracy benefits of the two-pass architecture.
Data Source
Figure 1A
Figure 1B
Figure 2A
AI summary
Disclosed herein is a computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations comprising: receiving a sequence of audio features characterizing an utterance; based on the sequence of audio features, generating, using a first-pass decoder model, a plurality of first-pass speech recognition hypotheses, each first-pass speech recognition hypothesis corresponding to a candidate transcription of the utterance; generating, using a long short-term memory (LSTM) encoder, a first-pass encoding of the plurality of first-pass speech recognition hypotheses; and based on the sequence of audio features and the first-pass encoding, generating, using a second-pass decoder model, a second-pass hypothesis that rescores the plurality of first-pass speech recognition hypotheses.