Early-Exit Machine Learning Model Training Method
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing transfer learning methods for machine learning models, such as early exit applications, suffer from reduced inference accuracy due to untrained random parameters in classifiers, leading to adverse gradient effects during fine-tuning, which compromises model performance and inference speed.
Innovation Solution
The proposed solution involves a model generation device that fine-tunes an early-exit-directed model by training only the untrained heads (classifiers) initially and then the entire model, ensuring all classifiers are trained before back-propagation, thereby minimizing adverse gradient effects and improving inference accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If early exit is applied to achieve high-speed inference, then inference speed is improved, but inference accuracy deteriorates due to untrained random parameters in classifiers
Solution Approach 1:
The patent applies preliminary action by training the classifiers (heads) before performing the main inference task. Specifically, the method involves: (1) initializing multiple heads with random parameters, (2) training these heads using supervision data before inference, and (3) then using the trained heads for early exit inference. This preliminary training ensures that the random parameters are properly initialized and trained, eliminating the accuracy degradation while maintaining the speed benefits of early exit.
2Device complexity
If classifiers with random parameters are used in early exit, then device complexity is reduced, but harmful gradient effects occur during fine-tuning
Solution Approach 1:
The patent eliminates harmful gradient effects during fine-tuning by performing preliminary training of the heads before the main fine-tuning process. The method trains each head independently using supervision data in advance, which properly initializes the random parameters. This preliminary action prevents the adverse gradient effects that would otherwise occur during subsequent fine-tuning, while maintaining the simplified model structure of early exit.
Data Source
AI summary
A non-transitory computer-readable recording medium stores a machine learning program for causing a computer to execute a process including: for a machine learning model that includes a plurality of preliminarily trained layers, a first output layer formed according to a downstream task and coupled to a final layer of the plurality of layers, and a plurality of second output layers that is coupled to respective outputs of layers other than the final layer of the plurality of layers and has a same configuration as the first output layer, training only the first output layer and the second output layer of the machine learning model using the downstream task; and training the entire machine learning model that includes the first output layer and the second output layer using the downstream task.