Contrastive Encoder Training for Accurate Text Rewriting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text generative models suffer from exposure bias during training, leading to error accumulation in predicting subsequent words based on previously incorrect predictions, resulting in inaccurate rewritten texts.
Innovation Solution
A model training method that utilizes contrastive learning to optimize encoder parameters by constraining the similarity between positive and negative semantic distributions, using loss functions to enhance the accuracy of semantic representations, and incorporating word-level and cross-level contrastive learning to refine keyword associations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional text generative models are trained using standard sequence-to-sequence methods, then the model can generate rewritten texts, but error accumulation occurs during training leading to reduced accuracy in predicting subsequent words
Solution Approach 1:
The patent introduces contrastive learning as an intermediary training mechanism that mediates between the encoder's semantic representation and the decoder's text generation. By adding a contrastive learning module that operates alongside the standard sequence-to-sequence training, the system can distinguish positive samples (correct semantic representations) from negative samples (incorrect representations), thereby preventing error accumulation without compromising the generative capability of the model.
Solution Approach 2:
The patent applies preliminary action by performing contrastive learning on the encoder before the decoder generates text. The encoder is pre-trained to accurately distinguish semantic representations of positive and negative samples, ensuring that the semantic foundation is solid before text generation begins. This preliminary semantic discrimination prevents error propagation to subsequent word predictions.
2Measurement precision
If the model focuses on generating accurate rewritten texts, then semantic accuracy improves, but the complexity of the training process increases due to contrastive learning requirements
Solution Approach 1:
The patent segments the training process into distinct components: contrastive learning for the encoder and sequence-to-sequence learning for the decoder. This segmentation allows each component to be optimized independently - the encoder focuses on semantic discrimination through contrastive learning while the decoder focuses on text generation. The loss function is similarly segmented into a contrastive loss component and a generation loss component, making the overall complex training process more manageable and interpretable.
3Measurement precision
If contrastive learning is implemented to distinguish positive and negative samples, then semantic distribution similarity is improved, but computational resources and training time increase
Solution Approach 1:
The patent applies partial action by implementing contrastive learning selectively - not all training steps require full contrastive learning computation. The system can adjust the balance between contrastive learning and standard generation training, applying contrastive constraints only when needed to improve semantic discrimination. This partial application of contrastive learning achieves the necessary semantic distribution similarity while reducing unnecessary computational overhead and training time.
Data Source
AI summary
Computer-implemented methods, apparatuses, and computer-readable media for model training and text processing are disclosed. In an example, an original text, a positive sample, and a negative sample are obtained. The positive sample and the negative sample are rewritten texts of the original text. A first, second, and third distribution parameter corresponding to the original text, the positive sample, and the negative sample, respectively, are determined by using an encoder. The first, second, and third distribution parameter are used to describe a first, second, and third semantic distribution, respectively. A model parameter of the encoder is optimized by using a loss function based on the first, second, and third distribution parameter. The loss function includes a first term that is used to constrain a degree of similarity between the first and second semantic distribution to be higher than a degree of similarity between the first and the third semantic distribution.