Decoupled Model Pretraining Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional machine learning model pretraining approaches often degrade the performance of main models, especially when using large auxiliary models, due to joint optimization of shared parameters, leading to suboptimal performance in natural language processing tasks.
Innovation Solution
The technique involves separate optimization of auxiliary and main models during pretraining, where the auxiliary model predicts masked tokens and the main model determines if these tokens were present or replaced, allowing for independent optimization and improved performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If joint optimization of auxiliary and main models is performed during pretraining, then both models can be trained together, but the performance of the main model degrades especially when using large auxiliary models
Solution Approach 1:
The patent divides the optimization process into separate stages: first optimizing the auxiliary model independently, then optimizing the main model independently. This segmentation prevents the performance degradation that occurs when both models are jointly optimized, as each model can be tuned to its optimal parameters without interference from the other model's gradient updates.
2Device complexity
If shared parameters are used between auxiliary and main models, then model architecture is simplified, but optimization becomes coupled and harms main model performance
Solution Approach 1:
The patent segments the parameter optimization by creating independent optimization streams for the auxiliary model and main model. Even though the models may share architectural components, the parameters are optimized separately in distinct training phases, preventing the coupling that would otherwise occur through joint gradient computation and update operations.
3Manufacturing precision
If separate optimization is performed for auxiliary and main models, then main model performance improves, but training process becomes more complex
Solution Approach 1:
The patent implements periodic action by alternating between two distinct optimization phases: a first phase dedicated to optimizing the auxiliary model, followed by a second phase dedicated to optimizing the main model. This periodic switching between optimization targets provides a structured approach that improves main model performance while keeping the overall process manageable through clear phase separation.
Data Source
AI summary
The disclosed concepts relate to pretraining of machine learning models. One example method involves performing separate optimization of a first machine learning model and a second machine learning model. The first machine learning model can be optimized based at least on first predictions and the second machine learning model can be optimized based at least on second predictions. The first predictions can represent predictions of masked values in first sequences of values values, and the second predictions can represent whether or not the first values were replaced with different values predicted by the first machine learning model.