Knowledge Distillation Using Backward-Pass Divergence Samples
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing knowledge distillation methods, such as vanilla KD, fail to ensure convergence of teacher and student deep neural networks in areas beyond the training dataset, leading to prediction divergence and inefficiency in deploying compressed models to resource-constrained devices.
Innovation Solution
Generate new auxiliary training data samples by maximizing divergence between teacher and student models using backward pass knowledge, augmenting the training dataset with perturbed input values to enhance convergence and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If knowledge distillation is used to compress the DNN model, then the number of parameters is reduced for deployment on resource-constrained devices, but the accuracy and convergence performance deteriorate in areas beyond the training dataset
Solution Approach 1:
The method performs preliminary action by generating auxiliary training data samples before final model deployment. Specifically, it maximizes the divergence between teacher and student model outputs during training to create challenging training examples that force the student model to learn more robust features, thereby improving generalization performance on unseen data while maintaining model compression
Solution Approach 2:
The method implements feedback by using the output divergence between teacher and student models as a training signal. The loss function is designed to maximize the difference between teacher and student outputs on auxiliary data, creating a feedback loop that continuously improves student model accuracy by learning from its deviations from the teacher model
2Reliability
If the training dataset is augmented with perturbed input values, then the convergence between teacher and student models is improved, but the training time and computational resources increase
Solution Approach 1:
The method applies partial action by selectively augmenting the training dataset with only the most beneficial auxiliary samples. Instead of exhaustively generating all possible perturbed inputs, it uses gradient-based optimization to identify and generate only those auxiliary training samples that maximally improve the divergence objective, thereby achieving good convergence with moderate training time
Solution Approach 2:
The method changes parameters by dynamically adjusting the perturbation magnitude and type during training. It uses learnable transformation parameters that are optimized during the training process, allowing the model to adapt the amount and nature of data augmentation based on training progress, thus balancing convergence improvement with training efficiency
Data Source
AI summary
Methods and systems are provided for compressing a deep neural network (NN) model using knowledge distillation. The method includes training a student NN model to minimize a first loss between student model output values generated by the student NN model for a set of original input values and teacher model output values generated by a teacher NN model for the set of original input values, generating, for at least some of the original input values, a respective perturbed value that maximizes a second loss between an output value generated by the student NN model and an output value generated by the teacher NN model, adding the perturbed values to the set of original input values to provide a set of augmented input values and retraining the student NN model using the set of augmented input values.