Neural Network Model Compression via Knowledge Distillation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial neural network (ANN) models face challenges in maximizing recognition rates while minimizing their size, which is essential for efficient pattern recognition in computer systems.
Innovation Solution
The method involves using a teacher model and a student model, where the student model is trained to mimic the output of the larger teacher model, utilizing techniques like knowledge distillation and adversarial training to achieve model compression and improved recognition rates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the size of the neural network is reduced, then the efficiency and speed of pattern recognition improve, but the recognition rate deteriorates
Solution Approach 1:
The patent creates a simplified copy (student model) of the original complex model (teacher model) that replicates its recognition capabilities. The student model is trained to mimic the teacher model's output probabilities, effectively copying its knowledge while using fewer parameters and computational resources, thus resolving the contradiction between model size and recognition rate.
Solution Approach 2:
The patent changes the parameters of the neural network by training a student model with fewer layers and neurons compared to the teacher model. Through knowledge distillation, the student model learns to achieve similar recognition performance with reduced parameters, improving efficiency while maintaining recognition rate.
2Device complexity
If the size of the neural network is reduced, then the model compression is achieved, but the learning capability deteriorates
Solution Approach 1:
The patent introduces temperature as an intermediary parameter in the knowledge distillation process. By adjusting the temperature parameter during training, the student model can effectively learn from the teacher model's softened probability distributions, preserving learning capability despite the reduced model size and complexity.
3Reliability
If adversarial training is applied, then the recognition rate is improved, but the training complexity increases
Solution Approach 1:
The patent implements self-service adversarial training where the student model generates its own adversarial examples and uses them for training. This eliminates the need for a separate adversarial example generator, reducing training process complexity while still improving recognition rate through adversarial robustness.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A model training method and apparatus is disclosed, where the model training method acquires a recognition result of a teacher model and a recognition result of a student model for an input sequence and trains the student model to minimize a distinction between the recognition result of the teacher model and the recognition result of the student model, based on an objective that the recognition result of the teacher model and the recognition result of the student model are not distinguishable from each other.