Pretrained Language Model Training With Static Token Meaning Constraints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current BERT-based pre-trained models do not fully consider the original meaning of tokens, leading to increased training time and reduced precision and generalization performance due to insufficient consideration of the influence of static token meanings.
Innovation Solution
A method that involves tokenizing text into one-hot codes, generating static and dynamic token vectors, calculating their similarity, and using this similarity as a constraint to adjust the loss function during training, thereby incorporating the original meaning of tokens while considering context.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the original meaning of tokens is fully considered during training, then the precision and generalization performance of the model is improved, but the training time increases
Solution Approach 1:
The patent pre-calculates and stores static token meaning vectors in a lookup table before training begins. During training, these pre-computed vectors are directly retrieved and used as constraints, eliminating the need to recalculate them iteratively. This preliminary preparation resolves the contradiction by enabling full consideration of original token meanings without the computational overhead during the training process.
Solution Approach 2:
The patent introduces static token meaning vectors as an intermediary constraint between the input tokens and the model's predictions. These vectors serve as a bridge that guides the dynamic token vectors to maintain fidelity to original meanings while still adapting to context. The constraint term in the loss function acts as a mediator that balances context adaptation with original meaning preservation, improving precision without proportionally increasing training time.
2Productivity
If the original meaning of tokens is not considered, then the training time is reduced, but the precision performance of the model is reduced
Solution Approach 1:
The static token meaning vectors serve as an intermediary constraint that guides the training process. By incorporating these vectors as a constraint term in the loss function, the model receives directional guidance toward preserving original token meanings without requiring complex additional processing. This intermediary constraint improves precision performance while maintaining reasonable training efficiency.
Solution Approach 2:
The patent modifies the loss function by adding a constraint term that incorporates static token meaning vectors. This parameter change transforms the original loss landscape to include guidance from original token meanings. The modified loss function balances context adaptation with meaning preservation, achieving better precision performance without substantially increasing training complexity or time.
3Adaptability or versatility
If only context meaning is considered without original token meaning, then the model adapts well to context, but the generalization performance is reduced
Solution Approach 1:
The static token meaning vectors act as an intermediary constraint that prevents the model from over-adapting to training context at the expense of generalization. By maintaining a constraint term that references original token meanings, the model learns to adapt to context while preserving core semantic information. This intermediary guidance improves generalization performance by anchoring the model's context-dependent representations to meaningful base meanings.
Solution Approach 2:
The constraint term involving static token meaning vectors provides continuous feedback during training about how much the dynamic representations are deviating from original meanings. This feedback mechanism guides the optimization process to maintain a balance between context adaptation and meaning preservation, thereby improving generalization performance without sacrificing context adaptability.
Data Source
AI summary
A method and apparatus for training a natural language pre-trained model, a device, and a storage medium are provided. The method includes: tokenizing a text by using a dictionary, and converting tokens into one-hot codes; inputting the one-hot codes into a token embedding layer, and performing mapping by using the token embedding layer to obtain a static token vector corresponding to each token; adding the static token vector, a segment embedding vector, and a position embedding vector to obtain an input vector of each token, and taking the input vector as an input to obtain a dynamic token vector corresponding to each token; calculating similarity between the static token vector and the dynamic token vector, and taking a similarity calculation result as a constraint item; and adjusting an original loss function by using the constraint item, and training the natural language pre-trained model with the adjusted original loss function.