Transcription-Assisted Speech Translation to Reduce LM Hallucinations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing language model-based automatic speech translation technologies do not fully utilize prompt engineering to enhance translation accuracy and reduce hallucinations.
Innovation Solution
A two-stage processing approach is employed, where speech input is first transcribed into text using an ASR model and then processed with a language model, incorporating the transcription as context to facilitate more accurate translations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If direct speech-to-translation conversion is used, then processing speed is maintained, but translation accuracy deteriorates due to misidentified words and hallucinations
Solution Approach 1:
The translation process is segmented into two distinct stages: first, speech transcription to text using an ASR model; second, translation of the transcribed text using a language model. This segmentation allows each stage to specialize in its function, improving overall translation accuracy by reducing misidentified words and hallucinations, while the modular structure manages processing complexity through clear separation of concerns.
Solution Approach 2:
The patent introduces an intermediate transcription step as a mediator between speech input and translation output. The ASR model generates a text transcription that serves as an intermediate representation, which then feeds into the language model for translation. This intermediary process reduces direct speech-to-translation errors and hallucinations, improving translation accuracy while the structured intermediate step helps manage processing complexity.
2Measurement precision
If transcription-assisted translation is used, then translation accuracy improves, but processing time increases due to two-stage processing
Solution Approach 1:
The ASR model performs preliminary transcription of speech to text before the translation stage. By completing the transcription action in advance and storing it as an intermediate result, the system enables the language model to focus solely on translation without reprocessing the speech signal, thereby improving translation accuracy while managing processing time through efficient staged execution.
Solution Approach 2:
The system maintains continuous useful action by ensuring that the output of the first stage (transcription) directly feeds into the second stage (translation) without redundant processing. The transcription result is preserved and continuously utilized by the language model, eliminating wasted computational steps and reducing overall processing time while maintaining improved translation accuracy.
Data Source
AI summary
Disclosed are apparatuses, systems, and techniques that implement training and deployment of automatic transcription-assisted translation systems that use language models. The techniques include processing, using a first speech-to-text (S2T) model, a first input that includes a speech in a first language to generate a transcription of the speech. The techniques further include processing, using a second S2T model, a second input to generate a translation of the speech to a second language. The second input includes at least a representation of the speech, and the transcription of the speech.