Continual Pre-Training Using Cross-Domain Loss for Language Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional pre-trained language models require extensive retraining on new data, which is costly and inefficient, and struggle to maintain generalization while avoiding the forgetting of prior knowledge.
Innovation Solution
A method for continual pre-training that iteratively adjusts model parameters using cross-domain loss values to integrate knowledge from multiple domains, incorporating soft prompt features and latent features to enhance language model training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional learning methods retrain models from scratch on new data, then the model can learn new domain knowledge, but the training cost increases significantly and prior knowledge is lost
Solution Approach 1:
The patent applies preliminary action by pre-training a language model on general domain data before fine-tuning on specific domain data. The pre-training phase prepares the model with general language understanding and knowledge, which serves as a foundation for subsequent domain-specific adaptation. This avoids starting from scratch when encountering new domains, significantly reducing training time and computational cost while maintaining the ability to learn new domain knowledge through fine-tuning.
2Adaptability or versatility
If pre-trained language models are used, then general language understanding is achieved, but the models forget prior knowledge when faced with new data
Solution Approach 1:
The patent merges general domain pre-trained knowledge with specific domain fine-tuned knowledge through a unified model architecture. The model combines the general language understanding capabilities acquired during pre-training with domain-specific knowledge acquired during fine-tuning, allowing it to maintain both broad language comprehension and specialized domain expertise without forgetting prior knowledge.
Solution Approach 2:
The patent applies parameter changes by adjusting model parameters during fine-tuning while preserving the core pre-trained parameters. This selective parameter adjustment allows the model to adapt to new domains by modifying only the necessary parameters for domain-specific tasks, while retaining the general language understanding capabilities embedded in the pre-trained parameters, thus avoiding catastrophic forgetting.
3Adaptability or versatility
If extensive retraining is performed on new data, then the model adapts to new domains, but the computational resources and training costs increase
Solution Approach 1:
The patent performs preliminary pre-training on general domain data before fine-tuning on specific domain data. This preliminary action creates a robust foundation that reduces the extent of retraining needed for new domains, thereby lowering computational resources and training costs while maintaining strong domain adaptation capabilities.
Solution Approach 2:
The patent applies partial action by fine-tuning only the necessary portions of the model for domain-specific tasks rather than retraining the entire model from scratch. This selective fine-tuning approach significantly reduces computational resources and training costs while still achieving effective domain adaptation through targeted parameter adjustments.
Data Source
AI summary
Embodiments of this specification provide language model training methods and apparatuses based on continual pre-training. In one example method, a soft prompt feature corresponding to each current training sample in a current domain is obtained. A latent feature corresponding to each piece of text data in the current domain is obtained. A cross-domain loss value is determined based on a difference between the obtained latent feature and a latent feature obtained based on an initial current language model in a previous domain. When a training termination condition for the current domain is not satisfied, model parameters are adjusted based on the cross-domain loss value. When a training termination condition for the current domain is satisfied, the model training process is repeated by continuing using a training sample set in a next domain, until a training termination condition for continual pre-training is satisfied.