Inverted Projection Training for ASR-Robust Speech Translation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech translation systems struggle with errors introduced by automatic speech recognition (ASR) and automatic punctuation, particularly in long-form scenarios, leading to poor translations due to differences in spoken and written language structures and segmentation issues.
Innovation Solution
An inverted projection approach is used to align system transcripts with human transcripts, introducing ASR and punctuation errors into training data to create robust machine translation models, specifically through Levenshtein alignment and segment robustness training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional translation models are trained on clean, well-structured text, then translation quality for written text is improved, but translation robustness to ASR errors and punctuation mistakes deteriorates
Solution Approach 1:
The patent converts harmful ASR errors and punctuation mistakes into beneficial training data by injecting these errors into the training corpus. The translation model learns to handle erroneous inputs during training, transforming the previously harmful effect of ASR errors into a useful learning opportunity that improves robustness without sacrificing translation quality.
Solution Approach 2:
The patent performs preliminary action by pre-training the translation model on clean, well-structured text to establish strong translation capabilities. Then, it fine-tunes the model on data containing injected ASR errors and punctuation mistakes. This staged approach ensures the model first learns accurate translation before learning to tolerate and correct errors, resolving the contradiction between quality and robustness.
2Productivity
If cascaded systems use automatic speech recognition and automatic punctuation, then translation productivity is improved, but translation accuracy deteriorates due to error amplification
Solution Approach 1:
The patent implements feedback by using the output of ASR and automatic punctuation systems as input for training the translation model. The model learns from the errors introduced by these automated systems during training, enabling it to compensate for these errors during actual translation tasks. This feedback loop allows the system to maintain high productivity while improving accuracy despite error amplification.
3Ease of manufacture
If training data does not include ASR errors and segmentation errors, then training simplicity is maintained, but model robustness to real-world speech input deteriorates
Solution Approach 1:
The patent applies segmentation by dividing the training process into distinct stages: first training on clean, well-structured text to establish baseline translation capabilities, then fine-tuning on data containing segmented ASR errors and punctuation mistakes. This segmentation of the training process maintains simplicity in the initial phase while progressively introducing complexity to improve adaptability to real-world speech input.
Data Source
AI summary
The technology provides an approach to train translation models that are robust to transcription errors and punctuation errors. The approach includes introducing errors from actual automatic speech recognition and automatic punctuation systems into the source side of the machine translation training data. A method for training a machine translation model includes performing automatic speech recognition on input source audio to generate a system transcript. The method aligns a human transcript of the source audio to the system transcript, including projecting system segmentation onto the human transcript. Then the method performs segment robustness training of a machine translation model according to the aligned human and system transcripts, and performs system robustness training of the machine translation model, e.g., by injecting token errors into training data.