Self-Knowledge Distillation for Deep Learning Model Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models face challenges in fields with insufficient data, as existing self-knowledge distillation methods often ignore knowledge with low confidence, limiting their performance.
Innovation Solution
A method that involves inputting training data twice to a deep learning model to generate soft target vectors, sorting output vectors and soft target vectors based on confidence scores, and training the model to minimize a loss function determined by partial distributions, enhancing learning for high-confidence classes without ignoring low-confidence knowledge.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If self-knowledge distillation uses prediction results from previous learning iterations, then knowledge transfer efficiency is improved, but low confidence knowledge is ignored leading to incomplete learning
Solution Approach 1:
The patent segments the loss function into two distinct components: a first loss function that processes high-confidence predictions and a second loss function that processes low-confidence predictions. This segmentation allows each component to be optimized independently, ensuring that low-confidence knowledge is not lost while maintaining the efficiency of high-confidence knowledge transfer.
Solution Approach 2:
The patent applies different processing strategies to different parts of the prediction output based on confidence levels. High-confidence predictions undergo one type of processing while low-confidence predictions undergo another type of processing, allowing each to be handled with appropriate quality and attention, thereby preventing loss of low-confidence knowledge.
2Reliability
If deep learning models are trained with insufficient data, then model performance deteriorates, but acquiring more data increases cost and time
Solution Approach 1:
The patent implements self-knowledge distillation where the model generates its own training targets from previous learning iterations. The model serves itself by creating soft labels from its own predictions, eliminating the need for additional external data sources and reducing data acquisition time while maintaining performance.
Solution Approach 2:
The patent performs preliminary learning iterations to generate prediction results that are then used as soft targets for subsequent training. By preparing training targets in advance through self-generated predictions, the model can proceed with training without requiring additional external data collection time.
3Productivity
If knowledge distillation transfers knowledge from teacher network to student network, then learning efficiency is improved, but additional model complexity is introduced
Solution Approach 1:
The patent merges the teacher and student networks into a single model instance that performs both roles. The same model generates predictions and uses those predictions as training targets, eliminating the need for separate teacher and student network architectures while maintaining the knowledge distillation learning efficiency.
Solution Approach 2:
The model performs self-knowledge distillation by generating its own soft targets from previous iterations. This self-service approach eliminates the need for a separate teacher network, reducing model architecture complexity while preserving the benefits of knowledge distillation through iterative self-learning.
Data Source
AI summary
There is provided a deep learning model training method using a self-knowledge distillation algorithm. The method comprises inputting training data to a deep learning model at a first time to obtain first output vectors and inputting the training data to the deep learning model at a second time before the first time to obtain second output vectors; generating soft target vectors at the first time point with respect to the training data using the second output vectors and label data; sorting the first output vectors and the soft target vectors and generating a first partial distribution for the sorted first output vectors and a second partial distribution for the sorted soft target vectors; and training the deep learning model to minimize a first loss function determined on the basis of the first partial distribution and the second partial distribution.