Cross-Training Acoustic and Language Models to Reduce Disjointedness
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Automatic speech recognition (ASR) systems suffer from disjointedness between language and acoustic models, leading to incorrect word recognition, as these models become progressively disjoined over time with training, resulting in poor synchronization and increased error rates.
Innovation Solution
A method that selects and cross-trains a model-pair comprising an acoustic model and a language model by computing disjointedness, generating training speech patterns, and retraining models to correct errors, thereby reducing the degree of disjointedness and improving word recognition rates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If language model and acoustic model are trained separately over a domain, then each model's domain-specific performance improves, but the models become progressively more disjointed from each other
Solution Approach 1:
The patent merges the training processes of the language model and acoustic model by introducing a joint training mechanism where both models are trained together on the same domain-specific data. This involves computing gradients from both models and performing synchronized parameter updates, ensuring that both models adapt to the domain while maintaining consistency with each other, thereby preventing disjointedness while preserving domain-specific performance improvements.
2Stability of the object's composition
If models are retrained to reduce disjointedness, then synchronization between models improves, but training time and computational resources increase
Solution Approach 1:
The patent implements preliminary action by pre-computing domain-specific training data and preparing joint training configurations before actual model training begins. This includes pre-processing corpora to extract domain-relevant features, pre-configuring the joint training pipeline, and pre-warming up the synchronization mechanism. By performing these preparatory steps beforehand, the actual joint training process runs more efficiently, reducing the overall training time despite the added complexity of synchronized training.
Data Source
AI summary
A model-pair is selected to recognize spoken words in a speech signal generated from a speech, which includes an acoustic model and a language model. A degree of disjointedness between the acoustic model and the language model is computed relative to the speech by comparing a first recognition output produced from the acoustic model and a second recognition output produced from the language model. When the acoustic model incorrectly recognizes a portion of the speech signal as a first word and the language model correctly recognizes the portion of the speech signal as a second word, a textual representation of the second word is determined and associated with a set of sound descriptors to generate a training speech pattern. Using the training speech pattern, the acoustic model is trained to recognize the portion of the speech signal as the second word.