Large Language Model Parameter Fusion for Long-Text Adaptability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training large language models face challenges such as complex processes, high computing resource consumption, difficulty in obtaining high-quality long text data, poor balance between long and short text abilities, and insufficient universality, making it difficult to meet the requirements of different task scenarios efficiently.
Innovation Solution
A method involving phased training with short and long text data followed by parameter fusion to create a large language model, using high-quality short text data for initial training, mixed long and short text data for further training, and adjusting fusion coefficients to optimize the model's abilities, thereby enhancing its flexibility and applicability across various text lengths.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple rounds of fine-tuning using short text and long-short text are performed, then the model's long text ability is improved, but the training process becomes complex and computing resource consumption increases
Solution Approach 1:
The patent segments the training process into two independent phases: first training a short text model on short text data, then training a long text model on long text data. These separate models are later fused through parameter integration. This segmentation eliminates the need for multiple complex fine-tuning rounds while achieving both short and long text capabilities, directly resolving the contradiction between improved long text ability and reduced training complexity.
2Adaptability or versatility
If multiple rounds of fine-tuning using short text and long-short text are performed, then the model's long text ability is improved, but computing resource consumption increases
Solution Approach 1:
By dividing the training into separate short text model and long text model training stages, each phase can be optimized independently with appropriate computing resources. The short text model training uses efficient short text data, while the long text model training focuses specifically on long text capabilities. This segmentation reduces overall computing resource consumption compared to multiple rounds of comprehensive fine-tuning.
Solution Approach 2:
The patent creates a long text model by copying the base model structure and training it specifically on long text data. This copied model can then be fused with the short text model through parameter integration. This approach avoids the need to repeatedly fine-tune the same model multiple times, thereby reducing computing resource consumption while achieving the desired long text ability.
3Adaptability or versatility
If high-quality long text data is obtained for training, then the model's long text ability is improved, but data acquisition difficulty increases
Solution Approach 1:
The patent segments the data requirements into separate short text data and long text data training phases. By training independent models on separately acquired data sets, the system can leverage abundant short text data for the first model and acquire long text data specifically for the second model. This segmentation reduces the overall difficulty of data acquisition compared to requiring comprehensive high-quality long-short text data for multiple fine-tuning rounds.
Data Source
AI summary
A method for training a large language model, a method for generating a large language model, an electronic device and a storage medium are provided, relating to the fields of large language model, model training, text processing and other technologies. The method for training a large language model includes: training an initial model according to first training data to obtain a first model; wherein the first training data comprises a first type of text; training the initial model according to second training data to obtain a second model; wherein the second training data comprises the first type of text and a second type of text; and performing parameter fusion according to the first model and the second model to obtain the trained large language model.