RNN-T Shared Embeddings for Low-Latency Streaming Speech
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern automated speech recognition systems face challenges in achieving low latency and high accuracy while performing real-time streaming speech recognition, particularly in mobile devices, where user tolerance for latency is low.
Innovation Solution
A tied and reduced recurrent neural network-transducer (RNN-T) model is employed, utilizing a multi-headed attention mechanism with shared embedding matrices and position vectors to generate weighted averages of embeddings, and a joint network to predict probability distributions over speech recognition hypotheses, reducing computational footprint and memory requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a traditional speech recognition system is used to ensure high accuracy, then the recognition quality is improved, but the latency increases and real-time performance deteriorates
Solution Approach 1:
The speech recognition system is segmented into two distinct parts: a lightweight initial recognition model that operates in real-time with minimal latency, and a more complex final recognition model that processes the complete utterance for high accuracy. This segmentation allows the system to provide immediate preliminary results while maintaining the option for refined accuracy, thus resolving the contradiction between speed and precision.
Solution Approach 2:
The initial speech recognition model performs preliminary recognition on incoming speech data before the complete utterance is available. This preliminary action provides early results with low latency, and the system can later refine these results using the final recognition model, thereby achieving both real-time responsiveness and high accuracy without sacrificing one for the other.
2Measurement precision
If a complex speech recognition model is deployed to improve accuracy, then the recognition quality is improved, but the computational footprint and memory requirements increase
Solution Approach 1:
The model architecture is segmented into two distinct components with different complexity levels: a compact initial recognition model suitable for mobile devices with limited resources, and a more comprehensive final recognition model. This segmentation enables the system to operate effectively on resource-constrained devices while maintaining the capability for high-accuracy recognition when computational resources are available.
Solution Approach 2:
The system is designed to be universal by supporting both lightweight initial recognition and more complex final recognition within the same architecture. The initial model handles real-time constraints on mobile devices, while the final model provides enhanced accuracy when resources permit, making the system adaptable to various deployment scenarios without requiring separate specialized systems.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A RNN-T model (200) includes a prediction network (300) configured to, at each time step subsequent to an initial time step, receive a sequence of non-blank symbols (301). For each non-blank symbol the prediction network is also configured to generate, using a shared embedding matrix (304), an embedding (306) of the corresponding nonblank symbol, assign a respective position vector (308) to the non-blank symbol, and weight the embedding proportional to a similarity between the embedding and the respective position vector. The prediction network is also configured to generate a single embedding vector (350) at the corresponding time step. The RNN-T model also includes a joint network (230) configured to, at each of the plurality of time steps subsequent to the initial time step, receive the single embedding vector generated as output from the prediction network at the corresponding time step and generate a probability distribution over possible speech recognition hypotheses.