Language Model Training With Negative Data for Signal Removal
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing language models are susceptible to generating low perplexity scores for ungrammatical text due to the inclusion of negative signals during training, leading to false positives and inaccurate predictions.
Innovation Solution
Train a target language model using both positive and negative training data, utilizing a negative language model to generate ungrammatical outputs, and employ a maximum likelihood function to maximize the likelihood of predicting positive data while minimizing the likelihood of predicting negative data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If language models are trained only on positive training data to maximize prediction accuracy for grammatical text, then the ability to predict positive text is improved, but the model becomes susceptible to generating low perplexity scores for ungrammatical text
Solution Approach 1:
The patent applies preliminary anti-action by introducing negative training data before the model is fully trained on positive data. This negative data is specifically designed to counteract the tendency of the model to incorrectly assign low perplexity to ungrammatical text. By pre-exposing the model to negative examples during training, the model learns to distinguish between grammatical and ungrammatical patterns, thereby preventing the development of false positive predictions while maintaining accuracy on positive data.
2Adaptability or versatility
If language models are trained on large corpora to improve generalization and context recognition, then the model's ability to recognize long-distance dependencies is improved, but the model learns and retains negative signals from the training data
Solution Approach 1:
The patent applies the taking out principle by extracting and isolating negative signals from the training process. Instead of allowing negative signals to be embedded in the model during standard training on large corpora, the method separately identifies and extracts these negative patterns. The extracted negative data is then used to retrain or fine-tune the model, effectively removing the harmful negative signals while preserving the beneficial generalization capabilities learned from the large corpora.
3Productivity
If traditional maximum likelihood training is used to maximize prediction of training data, then training efficiency is improved, but the model cannot distinguish between grammatical and ungrammatical text patterns
Solution Approach 1:
The patent applies merging by combining traditional maximum likelihood training with negative data training into a unified training framework. The loss function is modified to incorporate both positive and negative examples, allowing the model to simultaneously learn from grammatical patterns while being penalized for predicting ungrammatical patterns. This merged approach maintains the computational efficiency of maximum likelihood training while adding the discriminative capability to distinguish between grammatical and ungrammatical text.
Data Source
AI summary
A method for training a language model using negative data may include accessing a first training corpus comprising positive training data and accessing a second training corpus comprising negative training data. The method may further include training a first language model using at least the first training corpus, the second training corpus, and a maximum likelihood function. The maximum likelihood function may maximize the likelihood of the first language model predicting the positive training data while minimizing the likelihood of the first language model predicting the negative training data.