Non-autoregressive ASR with LLM Rescoring for Accuracy and Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automatic speech recognition (ASR) systems face challenges in accurately recognizing infrequently spoken words due to limited training data, and integrating language models to improve recognition often increases latency and requires significant memory and computational resources.
Innovation Solution
A non-autoregressive and multilingual language-model-fused ASR system that generates multiple candidate speech recognition hypotheses for each audio segment using a speech recognition model, concatenates these hypotheses with previous transcriptions, and rescoring using a large language model to select the hypothesis with the highest probability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If language models are integrated into ASR systems to improve recognition of infrequently spoken words, then transcription accuracy is improved, but latency increases and computational resources are consumed
Solution Approach 1:
The system segments the language model processing into discrete candidate hypotheses that are generated independently for each audio segment. This allows parallel processing of multiple hypotheses simultaneously rather than sequential generation, reducing overall latency while maintaining the benefits of language model integration for improving transcription accuracy of infrequently spoken words.
Solution Approach 2:
The system performs preliminary action by generating multiple candidate speech recognition hypotheses in advance using the speech recognition model before final selection. These candidate hypotheses are prepared with associated probabilities, allowing the system to quickly evaluate and select the best hypothesis without requiring extensive real-time computation during the recognition process itself.
2Measurement precision
If language models are integrated into ASR systems to improve recognition of infrequently spoken words, then transcription accuracy is improved, but computational resources and memory are consumed
Solution Approach 1:
The system applies local quality by assigning different computational resources to different candidate hypotheses based on their individual probabilities and characteristics. Instead of uniformly processing all possible hypotheses with equal computational effort, the system focuses computational resources on the most promising candidates, thereby improving transcription accuracy while reducing overall computational resource consumption.
3Measurement precision
If multiple candidate hypotheses are generated and rescoring is performed using LLM, then transcription accuracy is improved, but device complexity increases
Solution Approach 1:
The system implements dynamics by making the hypothesis selection process adaptive based on confidence levels and probability distributions. The system dynamically adjusts which candidate hypotheses are pursued further and which are discarded, allowing flexible resource allocation across different audio segments and speech scenarios. This dynamic approach improves transcription accuracy while managing system complexity through intelligent, context-aware decision making.
Data Source
AI summary
A method includes, for each respective audio segment in a series of audio segments: generating, using a speech recognition model, multiple candidate speech recognition hypotheses for the respective audio segment and concatenating each respective candidate speech recognition hypothesis from the multiple candidate speech recognition hypotheses with a previously generated transcription corresponding to N prior audio segments. Each respective candidate speech recognition hypothesis includes a corresponding probability. For each respective audio segment, the method also includes rescoring, using a large language model (LLM), the corresponding probability of each respective candidate speech recognition hypothesis based on the concatenation and generating a transcription of the respective speech segment by selecting a respective one of the candidate speech recognition hypotheses that includes a highest rescored probability.