Internal Language Model Estimation for E2E ASR Domain Adaptation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
End-to-end automatic speech recognition (ASR) systems face challenges in integrating acoustic and language models due to the lack of clear division between these components, leading to susceptibility to domain shift during testing, especially when trained on different data domains.
Innovation Solution
The implementation of an internal language model estimation method that computes an estimated internal language model score by removing the contribution of the intrinsic acoustic model, allowing for integration with external language models without additional training steps, thereby improving speech recognition accuracy across domains.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If acoustic model, language model, and pronunciation model are incorporated in a single deep neural network for E2E ASR, then end-to-end performance is improved, but susceptibility to domain shift from training to testing increases
Solution Approach 1:
The patent segments the E2E model into two independent parts: a source-domain trained E2E model for acoustic modeling and an external target-domain language model. This segmentation allows each component to be optimized for its specific domain, reducing domain shift susceptibility while maintaining end-to-end performance benefits.
Solution Approach 2:
The patent introduces an internal language model score estimation as an intermediary component that bridges the source-domain E2E model and target-domain external language model. This intermediary enables seamless integration by computing the language model score from the E2E model's internal representations without requiring direct training on target-domain data.
2Adaptability or versatility
If external language model is integrated with source-domain E2E model, then adaptability to target domain is improved, but computational complexity increases
Solution Approach 1:
The patent implements self-service by enabling the source-domain E2E model to automatically generate internal language model scores that can be directly combined with external language model scores. This self-service mechanism eliminates the need for complex retraining or additional computational infrastructure, achieving target-domain adaptability through a simple scoring combination process.
3Measurement precision
If internal language model score is computed by removing acoustic model contribution, then language model integration accuracy is improved, but processing time increases
Solution Approach 1:
The patent applies partial action by removing only the acoustic model contribution from the E2E model's internal representations to obtain the internal language model score, rather than completely recomputing the entire model. This partial removal approach achieves sufficient language model score accuracy while minimizing the additional processing time required.
Data Source
AI summary
A computer device is provided that includes one or more processors configured to receive an end-to-end (E2E) model that has been trained for automatic speech recognition with training data from a source-domain, and receive an external language model that has been trained with training data from a target-domain. The one or more processors are configured to perform an inference of the probability of an output token sequence given a sequence of input speech features. Performing the inference includes computing an E2E model score, computing an external language model score, and computing an estimated internal language model score for the E2E model. The estimated internal language model score is computed by removing a contribution of an intrinsic acoustic model. The processor is further configured to compute an integrated score based at least on E2E model score, the external language model score, and the estimated internal language model score.