Transformer Model Scaling via Linear Parameter Growth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for training larger transformer models typically start from scratch, overlooking the implicit knowledge in smaller, extant models, which leads to wasteful computation and inefficient training.
Innovation Solution
The Learning to Grow (LIGO) framework uses a linear mapping of parameters from a smaller transformer to initialize a larger one, employing width- and depth-growth operators and Kronecker factorization to encode architectural knowledge, thereby reducing computational cost and accelerating training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If larger transformer models are trained from scratch, then model size and capacity are increased, but computational cost and training time increase significantly
Solution Approach 1:
The patent applies preliminary action by pre-training smaller transformer models before using them as bases for larger models. The smaller models are trained in advance and their parameters serve as initialization points for larger models, avoiding the need to train from random initialization. This preliminary training of smaller models captures implicit knowledge that can be transferred and scaled to larger architectures, reducing the computational burden of training large models from scratch.
Solution Approach 2:
The patent implements the nested doll principle by embedding smaller pretrained models within larger model architectures. The parameters of smaller models are linearly transformed and integrated into the parameter space of larger models through width-growth and depth-growth operators. This nested structure allows the larger model to contain and leverage the knowledge from smaller models, enabling efficient scaling while maintaining computational efficiency.
2Reliability
If training starts from scratch for each model size, then model performance can be optimized, but training efficiency decreases
Solution Approach 1:
The patent applies parameter changes by linearly transforming the parameters of smaller models to match the parameter space of larger models. Through width-growth operators that expand hidden dimensions and depth-growth operators that extend layer sequences, the parameters are systematically adjusted while maintaining their underlying patterns and relationships. This parameter transformation preserves the performance benefits of pretrained models while adapting them to larger architectures.
Solution Approach 2:
The patent uses copying by replicating and transforming the parameter structures of smaller models into larger models. Instead of learning all parameters from random initialization, the system copies the organized parameter patterns from smaller pretrained models and scales them appropriately. This copying approach, combined with linear transformation, maintains the performance characteristics while reducing training time significantly.
3Use of energy by moving object
If smaller pretrained models are reused to initialize larger models, then computational cost is reduced, but model architecture scaling becomes complex
Solution Approach 1:
The patent applies segmentation by decomposing the architecture scaling process into distinct width-growth and depth-growth operators. The width-growth operator handles expansion of hidden dimensions and attention heads, while the depth-growth operator manages the addition and transformation of layer sequences. This segmentation of scaling operations into modular, composable transformations simplifies the overall process of scaling from smaller to larger models while maintaining computational efficiency.
Data Source
AI summary
Parameters of a first transformer are accessed, and size dimensions of a second transformer that is to be trained and is larger than the first transformer are received. The parameters of the first transformer are linearly transformed using a combination of a width-growth operator and a depth-growth operator, wherein the linear transformation produces a set of new parameters, the set corresponding to the size dimensions of the second transformer. The second transformer is initialized with the set of new parameters.