Large Language Model Training via Source Weight Initialization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The pre-training process of large-scale pre-trained language models (PLMs) is computationally expensive and generates a significant carbon footprint, leading to high training costs and resource consumption.
Innovation Solution
A method for efficient model training that initializes a target model using learned knowledge from a smaller source model by expanding weight matrices and training sub-models in parallel, reducing computational complexity and costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large-scale PLMs are pre-trained from scratch, then model performance and language understanding capabilities are improved, but computational cost and training time increase significantly
Solution Approach 1:
The patent applies preliminary action by pre-training a small-scale PLM before using it to initialize the large-scale target model. This preliminary training phase allows the small model to learn foundational language patterns that can then be transferred and scaled up, avoiding the need to train the large model completely from scratch and significantly reducing computational costs.
Solution Approach 2:
The patent uses copying by transferring weights from the pre-trained small-scale source model to initialize the large-scale target model. Instead of creating new random weights for the large model, the method copies and adapts the learned representations from the smaller model, preserving valuable linguistic knowledge while scaling up the model capacity.
2Adaptability or versatility
If large-scale PLMs are trained independently from scratch, then model capacity and parameter count are increased, but training time and resource consumption increase
Solution Approach 1:
The method performs preliminary training of a small-scale model to establish foundational knowledge before scaling up. This preliminary phase captures essential language patterns that can be transferred to the larger model, reducing the time needed to train the high-capacity model while maintaining its adaptability and versatility.
Solution Approach 2:
The patent applies parameter changes by modifying the scale and configuration of the model during the transfer process. The method adjusts parameters such as model dimension, number of layers, and attention heads when transitioning from the source model to the target model, enabling the large model to maintain optimal performance while training more efficiently.
3Ease of manufacture
If small-scale PLM knowledge is not utilized, then training process is simpler, but computational resources and costs increase significantly
Solution Approach 1:
The patent maintains training process simplicity by using a straightforward weight initialization approach where the large model is initialized by copying and adapting weights from the small model. This preliminary action of pre-training the small model creates a ready-to-use initialization scheme that is easier to implement than alternative approaches, while simultaneously reducing computational resource consumption.
Data Source
AI summary
A computer-implemented method is provided for model training performed by a processing system. The method comprises determining a set of first weights based on a first matrix associated with a source model, determining a set of second weights based on the set of first weights, forming a second matrix associated with a target model based on the set of first weights and the set of second weights, initializing the target model based on the second matrix, and training the target model.