RNN Language Model Training Data Modification for Rescoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Recurrent Neural Network (RNN) language models used for speech recognition and rescoring collapse when encountering errors in hypotheses, leading to negative effects in calculating language model probabilities and increasing error rates in recognition tasks.
Innovation Solution
The RNN language model is trained using modified training data where the sequence is split and a predetermined symbol is added to allow estimation of probabilities starting or ending in the middle of the sequence, and the hidden state is reset based on confidence conditions to mitigate error impacts during rescoring.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the RNN language model is trained with complete sequences from the beginning, then the model can capture the full context and history, but the model performance degrades when encountering errors in the middle of sequences during rescoring
Solution Approach 1:
The training sequence is divided into multiple parts, with the model trained to predict only a specific part (e.g., the latter half) given the first part as input. This segmentation allows the model to learn conditional probabilities for sequence portions rather than entire sequences, reducing the impact of errors in any single position while maintaining contextual understanding from the input portion.
2Adaptability or versatility
If the model uses the full history from the beginning of the sequence, then it can maintain strong modeling capability, but errors in the hypothesis cause the hidden state to collapse
Solution Approach 1:
The first part of the sequence is provided as input to the model before prediction begins, establishing a reliable initial hidden state. By pre-providing this context, the model starts its prediction task with accurate information already encoded in the hidden state, preventing error propagation from earlier positions and maintaining stability throughout the prediction process.
3Loss of information
If the model is trained to predict from the start of the sequence, then it can utilize all available context, but the error rates increase when rescoring hypotheses with errors
Solution Approach 1:
Different portions of the sequence are treated differently: the first part is used as input context (with higher reliability), while the second part is the target for prediction (where errors may occur). This local differentiation allows the model to leverage reliable context from the input portion while making predictions for the target portion, improving overall rescoring accuracy by focusing computational effort where it is most needed.
Data Source
AI summary
A computer-implemented method for training a model is disclosed. The model is capable of retaining a history of one or more preceding elements and has a direction of prediction. The method includes obtaining a training sequence of elements. The method also includes splitting the training sequence into a plurality of parts. The method further includes selecting one part of the plurality of the parts depending on the direction of the model to generate a modified training data. The method includes further training the model using the modified training data.