Transformer-Transducer Y-Architecture for Low-Latency Speech Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech recognition models struggle to balance low latency and high accuracy, with streaming models being inaccurate and non-streaming models having high latency, and separate models for different applications requiring significant computational resources and memory.
Innovation Solution
A single transformer-transducer model that integrates streaming and non-streaming speech recognition capabilities, using a y-architecture with parallel low and high latency branches to provide real-time transcription and improved accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If streaming models are used for real-time speech recognition, then latency is reduced, but accuracy deteriorates
Solution Approach 1:
The model is segmented into two independent branches: a streaming branch for real-time recognition and a non-streaming branch for accurate recognition. Each branch processes audio independently through separate transformer-transducer networks, allowing simultaneous optimization for latency and accuracy without compromising either performance.
2Adaptability or versatility
If separate streaming and non-streaming models are used to satisfy different application needs, then adaptability is improved, but device complexity and computational resources increase
Solution Approach 1:
A single unified transformer-transducer model is designed to perform both streaming and non-streaming speech recognition functions. The model incorporates shared acoustic and language models that can adapt to different operational modes, eliminating the need for separate models while maintaining both low latency and high accuracy capabilities.
Solution Approach 2:
The unified model is segmented into shared common components (acoustic model, language model) and mode-specific components (streaming branch, non-streaming branch). This segmentation allows efficient resource utilization by reusing common components across both functions while maintaining separate processing paths for different performance requirements.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
A transformer-transducer model (200) includes an audio encoder (300), a label encoder (220), and a joint network (230). The audio encoder receives a sequence of acoustic frames (110), and generates, at each of a plurality of time steps, a higher order feature representation for each acoustic frame. The label encoder receives a sequence of non-blank symbols output by a softmax layer (240), and generates, at each of the plurality of time steps, a dense representation. The joint network receives the higher order feature representation and the dense representation at each of the plurality of time steps, and generates a probability distribution over possible speech recognition hypotheses. The audio encoder of the model further includes a neural network having an initial stack (310) of transformer layers (400) trained with zero look ahead audio context, and a final stack (320) of transformer layers (400) trained with a variable look ahead audio context.