Efficient Memory Transformer for Low Latency Speech Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer-based models for speech recognition face challenges in online applications due to the need for access to the entire utterance before producing output and quadratic growth in computational cost and memory usage with respect to input sequence length, making them impractical for low latency streaming scenarios.
Innovation Solution
The implementation of an augmented memory transformer (AM-TRF) with a memory bank to store segment-level information, allowing attention to access information beyond the current segment, and the efficient memory transformer (Emformer) which reduces duplicated computations and applies parallelized block processing for low latency training, caching key and value projections to save computation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If transformer-based models use attention to access information beyond current segment, then speech recognition accuracy is improved, but computational cost and memory usage grow quadratically with input sequence length
Solution Approach 1:
The input sequence is divided into multiple segments that are processed independently through parallel block processing. Each segment is handled by a separate transformer block, preventing the quadratic growth of computational cost while maintaining the ability to access information across segments through the memory bank mechanism.
Solution Approach 2:
A memory bank is introduced as an intermediary structure to store segment-level information. This memory bank allows attention mechanisms to access information beyond the current segment without requiring quadratic computation, as the memory bank provides a compressed representation that can be efficiently queried.
2Measurement precision
If transformer-based models process entire utterance, then speech recognition accuracy is improved, but latency increases making them impractical for low latency streaming scenarios
Solution Approach 1:
Segment-level information is pre-computed and stored in the memory bank during the encoding phase. This preliminary action allows the decoding phase to quickly retrieve necessary information without waiting for the entire utterance to be processed, thereby reducing latency while maintaining accuracy.
Solution Approach 2:
The utterance is segmented into multiple blocks that can be processed and output incrementally. This segmentation enables low-latency streaming recognition by allowing the system to produce intermediate results for each segment rather than waiting for the complete utterance.
3Productivity
If parallelized block processing is applied, then training efficiency is improved, but memory management complexity increases
Solution Approach 1:
The training process is segmented into independent block processing units that can be executed in parallel. Each block processes a specific segment of the input sequence, enabling efficient utilization of parallel computing resources while maintaining manageable memory requirements through localized processing.
Solution Approach 2:
Intermediate results from parallel block processing are temporarily stored in the memory bank and then recovered when needed for final decoding. This approach allows efficient parallel training by discarding intermediate details during processing and recovering only the necessary segment-level information afterward, reducing peak memory requirements.
Data Source
AI summary
In one embodiment, a method includes accessing a machine-learning model configured to generate an encoding for an utterance by using a module to process data associated with each segment of the utterance in a series of iterations, performing operations associated with an i-th segment during an n-th iteration by the module, which include receiving an input comprising input contextual embeddings generated for the i-th segment in a preceding iteration and a memory bank storing memory vectors generated in the preceding iteration for segments preceding the i-th segment, generating attention outputs and a memory vector based on keys, values, and queries generated using the input, and generating output contextual embeddings for the i-th segment based on the attention outputs, providing the memory vector to the module for performing operations associated with the i-th segment in a next iteration, and performing speech recognition by decoding the encoding of the utterance.