Streaming ASR Latency Tuning With Chunked Self-Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing streaming APIs for automatic speech recognition (ASR) face a tradeoff between latency and recognition performance, with sub-second latency leading to a 30% relative reduction in word error rate, failing to meet varying use case requirements for latency and transcription quality.
Innovation Solution
Implementing a chunk-based self-attention mechanism that segments utterances into chunks with a hop size of 50%, allowing the model to attend to current and past frames, with dynamic chunk sizes influenced by a tunable latency parameter, enabling flexible latency and accuracy adjustments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If streaming ASR processes audio in real-time with limited context, then latency is reduced, but transcription accuracy deteriorates with more than 30% relative reduction in word error rate
Solution Approach 1:
The system dynamically adjusts the context window size based on the latency parameter. When latency is reduced, the context window is limited to only past and current frames. When latency is increased, the context window can include future frames, allowing the model to access more contextual information for improved accuracy.
Solution Approach 2:
The invention changes the latency parameter to control the tradeoff between speed and accuracy. By adjusting this parameter, users can specify how much forward context should be available, enabling the system to adapt to different use case requirements without retraining the model.
2Measurement precision
If batch transcription processes complete audio files, then transcription accuracy is improved, but latency increases making it unsuitable for real-time applications
Solution Approach 1:
The ASR model is designed to serve multiple functions: it can operate in streaming mode with limited context for low-latency applications, or in batch mode with full context for high-accuracy applications. The same model handles both scenarios by adjusting the context window based on the latency parameter.
3Productivity
If streaming ASR uses limited context to improve latency, then processing speed increases, but performance degradation occurs with 30% relative reduction in word error rate
Solution Approach 1:
The system dynamically adjusts the context window size based on the latency parameter. When latency is reduced, the context window is limited to only past and current frames. When latency is increased, the context window can include future frames, allowing the model to access more contextual information for improved accuracy.
4Loss of time
If existing streaming APIs optimize for sub-second latency, then real-time processing is achieved, but transcription quality deteriorates failing to meet diverse use case requirements
Solution Approach 1:
The invention changes the latency parameter to control the tradeoff between speed and accuracy. By adjusting this parameter, users can specify how much forward context should be available, enabling the system to adapt to different use case requirements without retraining the model.
Solution Approach 2:
The ASR model is designed to serve multiple functions: it can operate in streaming mode with limited context for low-latency applications, or in batch mode with full context for high-accuracy applications. The same model handles both scenarios by adjusting the context window based on the latency parameter.
Data Source
AI summary
Techniques for customizable latency, from the customer's side, for automatic speech recognition (ASR) are described. In particular, the customer may specify a parameter that controls how fast or how slow the customer's media content will be streamed or processed. Slower processing means higher accuracy, with near real-time latency, while faster processing means lower accuracy, but offers much lower latency (e.g., less than 600 ms). Enabling tuning of the latency-versus-accuracy tradeoff of the ASR system offers customers the flexibility to meet varying needs for different ASR applications.