Neural Network Knowledge Distillation for Model Accuracy and Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network model learning methods face issues with overlearning when data is small, leading to lower accuracy models, and require a huge teacher model for high accuracy, resulting in long calculation times and high learning costs.
Innovation Solution
A model learning device that uses a learned first model to set parameters for a second model with the same network structure, calculating output probability distributions and updating parameters based on cross entropy between the two models to reduce a weighted loss function, allowing for accurate model learning without a huge teacher model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a huge teacher model is used to achieve high accuracy, then model accuracy is improved, but calculation time increases and learning cost increases
Solution Approach 1:
The patent uses a student model that copies the network structure and initialization parameters from a teacher model, then fine-tunes it using knowledge distillation. This allows the student model to achieve high accuracy without requiring the full computational resources of a huge teacher model during inference, thus reducing calculation time while maintaining accuracy.
Solution Approach 2:
The patent changes the training parameters by using knowledge distillation with temperature parameter T and weighting parameter α. The student model is trained with modified loss function that incorporates teacher model outputs scaled by temperature T and weighted by α, enabling efficient learning of high accuracy models without proportionally increasing computational time.
2Measurement precision
If a huge teacher model is used to achieve high accuracy, then model accuracy is improved, but learning cost increases
Solution Approach 1:
The student model copies the architectural parameters and initialization from the teacher model, allowing it to inherit the knowledge structure. This copying approach enables the student model to achieve comparable accuracy with significantly reduced training and inference costs, as it only needs to fine-tune rather than train from scratch or match the full teacher model capacity.
Solution Approach 2:
The patent introduces temperature parameter T and weighting parameter α to modify the knowledge distillation process. By adjusting these parameters, the system optimizes the balance between learning from teacher model knowledge and maintaining computational efficiency, reducing energy consumption and learning costs while achieving high accuracy.
3Loss of energy
If learning data is small, then learning cost is reduced, but overlearning occurs and accuracy decreases
Solution Approach 1:
The teacher model acts as an intermediary that provides distilled knowledge through its output probability distributions. This intermediary knowledge transfer mechanism allows the student model to learn from high-quality representations without requiring large amounts of training data, preventing overlearning while maintaining accuracy by leveraging the teacher's pre-learned patterns.
Solution Approach 2:
The patent uses temperature parameter T to control the softness of probability distributions during knowledge distillation. By adjusting T and the weighting parameter α, the system optimizes information transfer from teacher to student model, enabling effective learning with small datasets by enhancing the quality and quantity of learned representations through parameter optimization.
Data Source
AI summary
A model learning device comprises: an initial value setting part that uses a parameter of a learned first model including a neural network to set a parameter of a second model including a neural network having a same network structure as the first model; a first output probability distribution calculating part that calculates a first output probability distribution including a distribution of an output probability of each unit on an output layer, using learning features and the first model; a second output probability distribution calculating part that calculates a second output probability distribution including a distribution of an output probability of each unit on the output layer, using learning features and the second model; and a modified model update part that obtains a weighted sum of a second loss function calculated from correct information and from the second output probability distribution, and a cross entropy between the first output probability distribution and the second output probability distribution, and updates the parameter of the second model so as to reduce the weighted sum.