Adaptive Frame Batching for Low-Latency Speech Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network-based automatic speech recognition (ASR) systems experience undesirable user-perceived latency due to batch processing of audio frames, and conventional approaches increase training and deployment costs by using multiple models to address latency requirements.
Innovation Solution
Implement dynamic batch sizing and look-ahead frames in ASR processing, adjusting batch size and look-ahead frames based on generated word hypotheses and end-of-speech detection to reduce latency while maintaining recognition accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If audio frames are batched into a batch of two or more frames for joint processing, then recognition accuracy and performance are improved, but user-perceived latency increases due to waiting for all frames to be received
Solution Approach 1:
The batch size is made dynamic rather than fixed. The system starts with a small initial batch size to reduce latency, then adaptively increases the batch size after generating a word hypothesis to improve recognition accuracy. This dynamic adjustment resolves the contradiction by allowing the system to prioritize speed initially, then accuracy later in the processing pipeline.
Solution Approach 2:
The system performs preliminary processing with a small initial batch size before committing to larger batch processing. By generating an early word hypothesis with minimal batching, the system provides quick initial results to the user, then refines accuracy in subsequent processing stages without requiring the user to wait for complete large-batch processing.
2Loss of time
If several physically-different models are used to satisfy different latency requirements, then latency requirements are met, but processor-related training and deployment costs multiply
Solution Approach 1:
A single ASR model is designed to perform multiple functions by processing batches of varying sizes. The same model handles both low-latency scenarios (small batches) and high-accuracy scenarios (large batches), eliminating the need for multiple specialized models and reducing training and deployment costs while satisfying different latency requirements.
Solution Approach 2:
Instead of changing models to satisfy different latency requirements, the system changes the batch size parameter. By dynamically adjusting the batch size fed to a single model, the system can optimize for latency when needed (small batches) or for accuracy when needed (large batches), achieving the same effect as multiple models without the associated costs and complexity.
Data Source
Figure 1A~1B
Figure 1C~1D
Figure 2
AI summary
Embodiments may include collection of a first batch of acoustic feature frames of an audio signal, the number of acoustic feature frames of the first batch equal to a first batch size, input of the first batch to a speech recognition network, collection, in response to detection of a word hypothesis output by the speech recognition network, of a second batch of acoustic feature frames of the audio signal, the number of acoustic feature frames of the second batch equal to a second batch size, and input of the second batch to the speech recognition network.