Joint Acoustic-Text Decoder for Low-Latency Rare-Word Transcription
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing end-to-end speech recognition systems face challenges in achieving real-time transcription due to high latency and reliance on remote computing, which introduces unreliability, and struggle with accuracy on rare words and long tail entities when trained only on supervised audio-text pairs.
Innovation Solution
A two-pass speech recognition model incorporating a recurrent neural network transducer (RNN-T) followed by a listen-attend-spell (LAS) network, enhanced with a joint acoustic and text decoder (JATD) to handle both paired and unpaired data, reducing latency and improving accuracy on rare words.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a two-pass end-to-end model with joint acoustic and text decoder is used, then accuracy on rare words and long tail entities is improved, but model complexity increases
Solution Approach 1:
The model is divided into two distinct passes: a streaming pass for real-time transcription and a non-streaming pass for accuracy improvement. This segmentation allows each pass to be optimized for its specific function while working together to solve the overall problem of rare word recognition
Solution Approach 2:
The joint acoustic and text decoder merges acoustic information from the streaming pass with text information from the non-streaming pass. By combining these different information sources in a unified decoder architecture, the system achieves improved accuracy on rare words while maintaining a cohesive model structure
2Loss of time
If streaming speech recognition is implemented for real-time transcription, then latency is reduced, but accuracy deteriorates
Solution Approach 1:
The speech recognition system is segmented into two operational modes: a streaming mode that provides low-latency real-time transcription, and a non-streaming mode that processes the same input with higher accuracy. This segmentation allows the system to meet both latency and accuracy requirements by using the appropriate mode for different scenarios
Solution Approach 2:
The non-streaming pass provides feedback to the streaming pass by offering corrected or improved transcriptions. The system uses the higher accuracy results from non-streaming processing to refine and improve the real-time streaming transcription quality, creating a feedback loop that continuously enhances accuracy
3Reliability
If the model is deployed on-device for direct user interactivity, then reliability is improved, but computational requirements increase
Solution Approach 1:
The system applies partial processing through the streaming pass for immediate results, and optionally applies the more computationally intensive non-streaming pass selectively for accuracy improvement. This partial action approach allows on-device deployment by enabling the system to use full computational resources only when necessary rather than continuously
Data Source
Figure 1A
Figure 1B
Figure 2A
AI summary
A method (300) includes receiving a training example (203) for a listen-attend- spell (LAS) decoder (230) of a two-pass streaming neural network model (200) and determining whether the training example corresponds to a supervised audio-text pair or an unpaired text sequence. When the training example corresponds to an unpaired text sequence, the method also includes determining a cross entropy loss based on a log probability associated with a linguistic context vector (246) of the training example. The method also includes updating the LAS decoder and the linguistic context vector based on the determined cross entropy loss.