Teacher Model Selection for Neural Network Distillation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In neural networks, size reduction of models for efficient computation and storage in limited resources leads to accuracy issues during distillation, especially when the size reduction ratio between the student and teacher models is large, and requires pre-learning for both models, increasing costs.
Innovation Solution
An information processing device that selects a teacher model based on the size ratio between the student and target models, and selects initial parameters from the target model using evaluation values to learn the student model by distillation, eliminating the need for pre-learning of the student model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of moving object
If the size reduction ratio between student and teacher models is large, then the model size is reduced efficiently, but the accuracy of learning by distillation deteriorates
Solution Approach 1:
The patent segments the teacher model selection process into multiple candidates with different size reduction ratios. Instead of using a single teacher model, the system divides the problem into selecting from multiple segmented options (different teacher models with different size ratios), allowing the student model to learn from the most appropriate teacher for its specific size reduction scenario.
Solution Approach 2:
The patent implements dynamic teacher model selection based on the student model's size reduction ratio. The system adaptively chooses the teacher model that best matches the current student model's characteristics, making the distillation process dynamic rather than static. This allows the system to optimize accuracy for each specific size reduction scenario.
2Manufacturing precision
If pre-learning is performed for both target and student models, then the accuracy of the student model is improved, but the cost of learning increases
Solution Approach 1:
The patent extracts and removes the pre-learning step from the student model training process. By selecting appropriate initial parameters directly from the target model and using an appropriately selected teacher model for distillation, the system takes out the unnecessary pre-learning requirement, reducing computational cost while maintaining accuracy.
Solution Approach 2:
The patent performs preliminary action by pre-selecting optimal initial parameters from the target model before the distillation process begins. This preliminary parameter selection, combined with appropriate teacher model selection, prepares the student model for effective distillation without requiring separate pre-learning, thus reducing overall learning cost.
3Ease of manufacture
If initial parameters are randomly initialized, then the learning process is simple, but the accuracy of learning by distillation deteriorates
Solution Approach 1:
The patent performs preliminary action by pre-selecting optimal initial parameters from the target model before distillation begins. This preliminary parameter selection ensures high accuracy from the start of distillation while maintaining process simplicity, as the parameter selection is automated and integrated into the workflow.
Solution Approach 2:
The patent implements feedback by using evaluation values to assess the quality of initial parameter selections. The system evaluates different initial parameter configurations and selects those that yield the best distillation performance, creating a feedback loop that ensures high accuracy while maintaining automated simplicity.
Data Source
AI summary
According to one embodiment, an information processing device includes a target model learning unit, a change unit, a selection unit, and a student model learning unit. The target model learning unit learns a target model to be subjected to size reduction. The change unit changes the target model into a student model with a size smaller than a size of the target model. The selection unit selects, as a teacher model, one of a plurality of models including the target model and one or more intermediate models with a size smaller than the size of the target model in accordance with a comparison result between the size of the target model and the size of the student model. The student model learning unit learns the student model by distillation using the selected teacher model.