Dynamic Loss Weighting for Accurate Knowledge Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional knowledge transferring methods in machine learning often transfer incorrect knowledge from a 'teacher' model to a 'student' model, leading to inefficiencies and inaccuracies, particularly when the 'teacher' model's confidence is low.
Innovation Solution
A method utilizing a comprehensive loss function that weights the accuracy of the 'teacher' model's output to differentiate between processing results, ensuring the 'student' model learns accurate knowledge by adjusting the role of the first and second loss functions based on the 'teacher's' confidence, thereby preventing the transfer of incorrect knowledge.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional knowledge transferring methods are used to train a student model using a teacher model, then the student model can acquire the expression capacity of the teacher model, but incorrect knowledge may be transferred when the teacher model's confidence is low
Solution Approach 1:
The patent applies dynamics by making the weight of the first loss function variable rather than fixed. The weight is dynamically adjusted based on the teacher model's confidence level for each training sample, allowing the system to adaptively emphasize or de-emphasize knowledge transfer from the teacher model depending on its reliability for that specific sample.
Solution Approach 2:
The patent changes the parameter of the loss function weight based on the teacher model's confidence level. By modifying the weight parameter dynamically according to confidence metrics, the system optimizes the balance between transferring teacher knowledge and maintaining training accuracy, preventing incorrect knowledge transfer while preserving beneficial knowledge.
2Reliability
If the student model is trained to output the same result as the teacher model, then knowledge transfer occurs, but processing speed may be compromised due to model size
Solution Approach 1:
The patent applies local quality by differentiating the treatment of different training samples based on the teacher model's confidence level. High-confidence samples receive stronger knowledge transfer weighting, while low-confidence samples receive reduced weighting. This localized approach ensures accurate knowledge transfer where reliable while maintaining student model efficiency.
Data Source
AI summary
There are provided a method of knowledge transferring, an information processing apparatus and a storage medium. The method of knowledge transferring includes: obtaining a first model which has been trained in advance with respect to a predetermined task; and training a second model with respect to the predetermined task by utilizing a comprehensive loss function, such that the second model has knowledge of the first model. The comprehensive loss function is based on a first loss function weighted by accuracy of an output result of the first model for a training sample in regard to the predetermined task, and a second loss function. The first loss function represents a difference between processing results of the second model and the first model for the training sample. The second loss function represents accuracy of an output result of the second model for the training sample in regard to the predetermined task.