Text-Only Trained Deliberation ASR for Low-Latency On-Device Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automated speech recognition (ASR) systems face challenges in achieving high-quality, low-latency, real-time transcription, particularly in mobile environments, due to limitations in streaming capabilities and reliance on remote computing, which introduces latency and unreliability issues.
Innovation Solution
A two-pass deliberation model is employed, utilizing a text encoder and a deliberation decoder trained on unspoken textual utterances, combined with a recurrent neural network-transducer (RNN-T) decoder, to enhance speech recognition accuracy and reduce latency by incorporating text-only data and semi-supervised training, leveraging causal and non-causal encoders for improved hypothesis generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If conventional ASR systems use streaming decoding to achieve low latency, then response time is improved, but recognition accuracy deteriorates
Solution Approach 1:
The decoding process is divided into two distinct passes: a first-pass streaming decoder that operates in real-time to provide low-latency hypotheses, and a second-pass deliberation model that refines these hypotheses using both acoustic and text contextual information to improve accuracy. This segmentation allows each pass to optimize for its specific function without compromising the other.
Solution Approach 2:
The system performs preliminary decoding in the first pass to generate initial hypotheses quickly, then uses these preliminary results as input for the second-pass deliberation model. This preliminary action enables the system to maintain low latency while still having the opportunity to correct errors in a subsequent refinement stage.
2Measurement precision
If ASR systems rely on remote computing for high accuracy, then recognition quality is improved, but system reliability deteriorates due to network dependency
Solution Approach 1:
The deliberation model is designed to operate autonomously on-device by integrating acoustic embeddings with text hypotheses and contextual information from the device's own text encoder. This self-service capability eliminates the need for continuous remote computing while maintaining high recognition quality through locally processed deliberation.
Solution Approach 2:
The system combines multiple functions (acoustic decoding, text encoding, contextual analysis, and hypothesis refinement) into a unified on-device deliberation model. This multi-functionality allows the system to perform high-quality recognition independently without relying on external services, thereby improving reliability.
3Productivity
If the deliberation model uses text-only data for training, then training efficiency is improved, but model accuracy may deteriorate without acoustic data
Solution Approach 1:
The model performs preliminary training using text-only data to establish strong linguistic patterns and contextual understanding efficiently. After this preliminary training phase, the model is fine-tuned with acoustic embeddings to capture speech-specific patterns, achieving both training efficiency and high accuracy.
Solution Approach 2:
The approach merges text-only training data with acoustic embedding data in a unified training framework. The text encoder processes linguistic information while the acoustic model processes speech patterns, and both are combined through the deliberation mechanism to achieve superior accuracy without sacrificing training efficiency.
4Measurement precision
If the system processes both acoustic and text information in the deliberation model, then recognition accuracy is improved, but computational complexity increases
Solution Approach 1:
The computational process is segmented into distinct stages: acoustic embedding extraction, text encoding, and deliberation refinement. Each stage processes specific types of information independently before integrating results, which manages computational complexity while maintaining comprehensive accuracy through multi-source information processing.
Data Source
AI summary
A method of text-only and semi-supervised training for deliberation includes receiving training data including unspoken textual utterances that are each not paired with any corresponding spoken utterance of non-synthetic speech, and training a deliberation model that includes a text encoder and a deliberation decoder on the unspoken textual utterances. The method also includes receiving, at the trained deliberation model, first-pass hypotheses and non-causal acoustic embeddings. The first-pass hypotheses is generated by a recurrent neural network-transducer (RNN-T) decoder for the non-causal acoustic embeddings encoded by a non-causal encoder. The method also includes encoding, using the text encoder, the first-pass hypotheses generated by the RNN-T decoder, and generating, using the deliberation decoder attending to both the first-pass hypotheses and the non-causal acoustic embeddings, second-pass hypotheses.