Two-Pass On-Device Speech Recognition with a Shared Encoder
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automated assistants face challenges in providing real-time, low-latency, and accurate speech recognition on client devices with limited resources, as conventional models require network connectivity and consume significant computational resources.
Innovation Solution
A two-pass automatic speech recognition (ASR) model is employed, combining a recurrent neural network-transformer (RNN-T) decoder with a listen attend spell (LAS) decoder, sharing a common encoder to reduce model size and computational demands, allowing on-device processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a conventional speech recognition model is used, then network connectivity is required and computational resources are consumed, but real-time processing and low resource consumption cannot be achieved
Solution Approach 1:
The speech recognition model is divided into two distinct passes: a first-pass streaming portion that provides real-time candidate transcriptions with low latency, and a second-pass non-streaming portion that refines accuracy. This segmentation allows the system to balance real-time processing requirements with accuracy improvements without requiring excessive computational resources at any single moment.
Solution Approach 2:
The system dynamically switches between streaming and non-streaming processing modes. The first-pass RNN-T decoder operates in streaming mode during speech input, providing immediate responses. The second-pass LAS decoder then processes the same audio data in non-streaming mode to improve accuracy. This dynamic approach optimizes resource consumption by using lightweight streaming processing during active speech and more intensive processing only when needed for final accuracy.
2Measurement precision
If a larger model is used to improve speech recognition accuracy, then model size and computational requirements increase, but client devices with limited resources cannot support it
Solution Approach 1:
The patent merges two different decoding approaches (RNN-T streaming decoder and LAS non-streaming decoder) into a unified two-pass system that shares a common encoder. This combination allows the system to achieve the accuracy benefits of a larger model while keeping individual model components smaller and more suitable for client device deployment. The shared encoder further reduces overall model size by eliminating redundancy.
Solution Approach 2:
Instead of deploying a single large high-accuracy model that would exceed device resources, the system uses partial action by implementing two smaller pass models that each perform specific functions. The first-pass model provides sufficient accuracy for real-time feedback, while the second-pass model adds refinement. This partial approach to accuracy at each stage achieves overall high accuracy without requiring a single oversized model.
3Loss of time
If streaming processing is used for real-time response, then latency is reduced, but speech recognition accuracy deteriorates
Solution Approach 1:
The recognition process is segmented into two temporal phases: first-pass streaming recognition that operates during speech input to provide low-latency candidate transcriptions, and second-pass non-streaming recognition that operates after speech completion to improve accuracy. This segmentation resolves the contradiction by assigning different accuracy-latency requirements to different time periods in the processing pipeline.
Solution Approach 2:
The first-pass streaming decoder performs preliminary speech recognition during the speech input phase, providing immediate candidate transcriptions with acceptable accuracy for real-time feedback. This preliminary action allows the system to meet latency requirements while the second-pass decoder subsequently performs refinement to achieve final high accuracy, thus resolving the accuracy-latency tradeoff.
Data Source
AI summary
Two-pass automatic speech recognition (ASR) models can be used to perform streaming on-device ASR to generate a text representation of an utterance captured in audio data. Various implementations include a first-pass portion of the ASR model used to generate streaming candidate recognition(s) of an utterance captured in audio data. For example, the first-pass portion can include a recurrent neural network transformer (RNN-T) decoder. Various implementations include a second-pass portion of the ASR model used to revise the streaming candidate recognition(s) of the utterance and generate a text representation of the utterance. For example, the second-pass portion can include a listen attend spell (LAS) decoder. Various implementations include a shared encoder shared between the RNN-T decoder and the LAS decoder.