Causal Encoder Linear Attention Reduces ASR Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Automated speech recognition (ASR) systems with integrated models face increased latency due to the large number of internal states maintained by self-attention layers, which exceed the limited memory bandwidth of devices, hindering efficient inference performance.
Innovation Solution
The implementation of a causal encoder with a stack of causal encoder layers, each incorporating a Recurrent Neural Network (RNN) Attention-Performer module that applies linear attention, replacing multi-head attention modules to reduce latency and model size, while retaining benefits from self-attention layers during pre-training and fine-tuning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multi-head attention layers are used in integrated ASR models, then speech recognition accuracy is improved, but latency increases due to large number of internal states
Solution Approach 1:
The patent changes the computational parameters of the attention mechanism from quadratic complexity (standard self-attention) to linear complexity (linear attention). This is achieved by modifying the attention computation to use a factorized form where the attention matrix is decomposed into lower-rank components, reducing the number of parameters and operations required while maintaining the essential attention functionality for accurate speech recognition.
Solution Approach 2:
The patent extracts and removes the problematic multi-head attention layers from the encoder architecture, replacing them with linear attention mechanisms. This extraction eliminates the source of excessive internal states and latency while preserving the necessary attention-based feature processing capabilities through the lighter linear attention alternative.
2Reliability
If multiple self-attention layers are used to maintain internal states, then feature representation quality is improved, but memory bandwidth is exceeded on devices
Solution Approach 1:
The patent changes the memory usage parameters by reducing the dimensionalities and quantities of internal states in the attention layers. Linear attention uses fewer parameters and maintains smaller state vectors compared to multi-head self-attention, thereby reducing memory bandwidth requirements while preserving sufficient feature representation quality for accurate speech recognition on device hardware.
Solution Approach 2:
The patent replaces expensive, memory-intensive multi-head attention layers with cheaper, more memory-efficient linear attention layers. This substitution uses less memory bandwidth and computational resources, making the model suitable for deployment on devices with limited memory capacity and bandwidth constraints.
3Productivity
If integrated models with multiple attention layers are deployed, then speech recognition performance is improved, but device complexity increases
Solution Approach 1:
The patent extracts and removes the complex multi-head attention sub-architecture from the model, replacing it with simpler linear attention mechanisms. This extraction reduces the overall architectural complexity, number of parameters, and computational overhead while maintaining the integrated end-to-end speech recognition performance through the simplified attention-based encoder.
Data Source
AI summary
A computer-implemented method includes receiving a sequence of acoustic frames as input to an automatic speech recognition (ASR) model. Here, the ASR model includes a causal encoder and a decoder. The method also includes generating, by the causal encoder, a first higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. The method also includes generating, by the decoder, a first probability distribution over possible speech recognition hypotheses. Here, the causal encoder includes a stack of causal encoder layers each including a Recurrent Neural Network (RNN) Attention-Performer module that applies linear attention.