Student Model Training with Adaptive Distillation Loss
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models with large parameter quantities have high precision but slow prediction speed and high memory usage, while models with small parameter quantities have fast speed but low precision, and the distillation process struggles to maintain precision while ensuring speed in student models.
Innovation Solution
A model training method that involves inputting training samples with ground-truth labels into both a student and teacher model, determining losses between their predictions and the ground-truth, and adjusting the student model's parameters using a total loss function that incorporates an adaptive parameter based on the teacher model's gradient to dynamically balance the impact of the second loss, ensuring stability and consistency during training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a large parameter model is used, then precision is improved, but prediction speed deteriorates and memory occupation increases
Solution Approach 1:
The patent creates a student model that copies the knowledge and prediction patterns from a teacher model. The student model is trained using knowledge distillation, where it learns to replicate the teacher model's output distributions and decision boundaries. This allows the smaller student model to achieve high precision by copying the behavior of the larger teacher model, thereby resolving the contradiction between model size and precision while maintaining fast prediction speed.
2Measurement precision
If knowledge distillation is used to improve student model precision, then precision is improved, but training stability deteriorates due to gradient conflict
Solution Approach 1:
The patent dynamically adjusts the amplitude of the distillation loss through an adaptive parameter that is related to the gradient of the teacher model. This dynamic adjustment mechanism allows the training process to adaptively balance the competing gradients from the ground truth and teacher model predictions. When gradient conflict is detected, the adaptive parameter reduces the distillation loss amplitude to maintain training stability, thereby resolving the contradiction between precision improvement and training stability.
3Ease of manufacture
If fixed amplitude distillation loss is used, then implementation simplicity is maintained, but distillation effect deteriorates due to gradient conflict
Solution Approach 1:
The patent implements a self-adjusting mechanism where the adaptive parameter automatically regulates the distillation loss amplitude based on the teacher model's gradient characteristics. The system serves itself by detecting gradient conflict and autonomously adjusting the loss weighting without requiring manual intervention or complex external control mechanisms. This self-service approach maintains implementation simplicity while significantly improving the distillation effect by dynamically optimizing the training process.
Data Source
Figure 1~2
Figure 3~4
Figure 5
AI summary
The present disclosure relates to a model training method, a medium and an electronic device, and relates to the field of computer technology. The method includes: inputting a training sample carrying a ground-truth label into a student model and a teacher model, and obtaining the first prediction result output by the student model and the second prediction result output by the teacher model; determining the first loss between the first prediction result and the ground-truth label, and determining the second loss between the first prediction result and the second prediction result; determining a total loss function according to the first loss, the second loss and an adaptive parameter; and adjusting a parameter of the student model based on the total loss function to obtain a trained student model. According to the gradient of the teacher model, the impact of the second loss on the total loss function can be dynamically adjusted, so as to dynamically adjust the gradient impact of the second loss on the parameter weight of the student model, which makes the training of the student model maintain stability and consistency, thereby maximizing the accuracy of the student model.