Neural Transducer Training With External Language Model Guidance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing end-to-end speech recognition systems, particularly RNN-T models, face challenges in achieving real-time transcription and efficient on-device operation due to large search graphs and reliance on remote connections, which can lead to latency and unreliability.
Innovation Solution
Integrate external language model information into neural transducer models like RNN-T by combining language model probabilities with posterior lattices to train neural transducers, using linear or log-linear interpolation, and minimize divergence to improve accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a neural transducer model is trained to directly map acoustic signals to text without external language models, then training speed is improved, but language understanding capability deteriorates
Solution Approach 1:
The patent introduces an external language model as an intermediary component that receives embeddings from the neural transducer model and generates text output. This mediator handles complex language understanding tasks separately, allowing the neural transducer to focus on acoustic-to-embedding transformation while maintaining high training speed and improved language capability through the specialized language model component.
2Measurement precision
If the neural transducer model uses a large vocabulary size to improve transcription accuracy, then transcription accuracy is improved, but model complexity increases
Solution Approach 1:
The patent segments the transcription system into two specialized components: a neural transducer model optimized for acoustic signal processing and embedding generation, and an external language model optimized for text generation. This segmentation allows each component to be trained independently with appropriate data and architectures, reducing overall model complexity while maintaining high transcription accuracy through the combined system.
Solution Approach 2:
The neural transducer model serves multiple functions: it processes acoustic signals, generates embeddings, and works with variable vocabulary sizes. The external language model handles text generation across different vocabularies. This multi-functionality allows the system to adapt to different vocabulary requirements without requiring a single monolithic model to handle all scenarios, thereby managing complexity effectively.
3Reliability
If the neural transducer model is trained end-to-end with external language models, then language understanding is improved, but training time increases
Solution Approach 1:
The patent applies preliminary action by pre-training the external language model on large text corpora before combining it with the neural transducer model. This pre-training establishes strong language understanding capabilities in advance, so that during joint training or inference, the system leverages these pre-learned linguistic patterns without requiring extensive end-to-end training time, thus improving language understanding while minimizing additional training time.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A computer-implemented method for training a neural transducer is provided including, by using audio data and transcription data of the audio data as input data, obtaining outputs from a trained language model and a seed neural transducer, respectively, combining the outputs to obtain a supervisory output, and updating parameters of another neural transducer in training so that its output is close to the supervisory output. The neural transducer can be a Recurrent Neural Network Transducer (RNN-T).