Neural Network Pre-Training with Shared Layers for Stable Convergence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional pre-training language models using the Post-Layer Norm (PostLN) structure face instability and failure to converge when the number of model parameters is large, affecting their prediction accuracy.
Innovation Solution
A pre-training method for neural network models that employs a two-stage training approach: initially training with shared hidden layer parameters to stabilize convergence, followed by individual parameter updates for each layer when the loss value meets a threshold, enhancing both convergence and prediction accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the number of model parameters is increased to improve prediction accuracy, then the model's prediction accuracy is improved, but the training stability deteriorates and the model fails to converge
Solution Approach 1:
The training process is segmented into two distinct stages: first stage with shared hidden layer parameters across multiple layers, and second stage with independent parameters for each layer. This segmentation allows the model to first achieve stable convergence with fewer effective parameters, then progressively unlock higher accuracy by allowing individual parameter optimization in the second stage.
Solution Approach 2:
The first training stage serves as a preliminary action where the model establishes a stable foundation with shared parameters before proceeding to the second stage. This preliminary training with constrained parameters prevents the instability that would occur if full parameter optimization were attempted from the beginning, enabling subsequent achievement of high prediction accuracy.
2Measurement precision
If each hidden layer has independent parameters to improve prediction accuracy, then the model's prediction accuracy is improved, but the training complexity and difficulty of convergence increase
Solution Approach 1:
The system dynamically adjusts the parameter configuration during training: in the first stage, parameters are shared across layers to reduce complexity and ensure convergence; in the second stage, parameters become independent for each layer to maximize prediction accuracy. This dynamic transition resolves the contradiction between training simplicity and model performance.
Solution Approach 2:
The training process employs periodic action through two distinct training stages. The first periodic phase uses shared parameters for stability, and the second periodic phase uses independent parameters for accuracy optimization. This periodic structure manages training complexity by breaking down the challenging task of training large models into two more manageable phases.
3Quantity of substance
If traditional Post-Layer Norm structure is used with large parameter counts, then the model capacity is sufficient for high accuracy, but the training process becomes unstable and fails to converge
Solution Approach 1:
The invention changes the parameter configuration strategy rather than altering the Post-Layer Norm structure itself. By transitioning from independent parameters per layer to shared parameters across layers in the first training stage, the model maintains large parameter capacity while achieving stable convergence. This parameter change approach allows large models to be trained successfully without modifying the underlying architectural norm structure.
Data Source
AI summary
A pre-training method of a neural network model, an electronic device, and a medium. The pre-training data is inputted to the initial neural network model, and the initial neural network model is pre-trained in the first training mode, in the first training mode, the plurality of hidden layers share one hidden layer parameter, and the loss value of the initial neural network model is obtained, if the loss value of the initial neural network model is less than a preset threshold, the initial neural network model continues to be pre-trained in the second training mode, in the second training mode, each of the plurality of hidden layers has its own hidden layer parameter.