ASR Hypothesis Rescoring With LLMs for Domain Transcription Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional automatic speech recognition systems suffer from performance issues such as lack of accuracy, leading to incorrect words and unnatural phrases in transcriptions, particularly in specialized domains like medicine, law, or engineering.
Innovation Solution
Utilizing multiple language models, including a conventional ASR system and a large language model like ChatGPT-3 or ChatGPT-4, to refine speech recognition outputs by rescoring hypotheses with constraints, ensuring accuracy and tailoring to user preferences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional ASR systems are used for speech recognition, then the system is simple and easy to operate, but the accuracy and reliability of transcription are poor
Solution Approach 1:
The patent combines conventional ASR systems with large language models (LLMs) to create a hybrid system. The LLM rescoring component is integrated with the ASR pipeline to refine transcription hypotheses, merging the strengths of both approaches: the speed and basic accuracy of conventional ASR with the contextual understanding and accuracy enhancement of LLMs.
Solution Approach 2:
The patent introduces an LLM-based rescoring component as an intermediary between the ASR system and the final transcription output. This intermediary component receives ASR hypotheses, evaluates them using LLM contextual understanding, and produces refined transcriptions, thereby mediating the interaction between audio processing and language generation.
2Reliability
If multiple language models are used to enhance accuracy, then transcription quality improves, but computational resources and processing time increase
Solution Approach 1:
The patent applies partial action by using LLM rescoring selectively rather than applying full LLM processing to all transcriptions. The system uses the LLM to refine only the top hypotheses generated by the ASR system, rather than regenerating all possible transcriptions, thereby reducing computational energy consumption while maintaining high accuracy.
Solution Approach 2:
The patent performs preliminary action by having the ASR system generate multiple candidate hypotheses before LLM processing. This preliminary hypothesis generation narrows down the input to the LLM, allowing the computationally intensive LLM to process only the most promising candidates rather than all possible transcriptions, thus reducing overall energy consumption.
3Reliability
If domain-specific language models are trained to improve accuracy in specialized fields, then transcription accuracy in that domain improves, but the complexity of model training and deployment increases
Solution Approach 1:
The patent uses a universal LLM approach that can be applied across multiple domains without requiring domain-specific training. The LLM's general language understanding and contextual reasoning capabilities enable it to accurately transcribe speech in various specialized domains (medical, legal, technical) using a single unified model, eliminating the need for separate domain-specific model training and deployment.
Data Source
AI summary
One example method includes receiving an audio stream comprising speech; generating, by automatic speech recognition (“ASR”) software, a plurality of hypotheses, each hypothesis comprising a transcription of a first portion of the speech; rescoring, using a first trained language model, each hypothesis of the plurality of hypotheses; and responsive to a first hypothesis not satisfying a threshold, generating and outputting, using a trained large language model (“LLM”), a final transcription based on the plurality of hypotheses.