Neural Network Generation via Teacher Subnetwork Decomposition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
It is challenging to increase the complexity of student neural networks to match that of teacher neural networks due to restrictions, making it difficult to enhance the mimic level of student neural networks.
Innovation Solution
A neural network generation method that involves decomposing a trained teacher neural network into subnetworks and using these subnetworks to train a student neural network, with the goal of determining weight data for each layer of the student neural network to match the teacher's outputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the complexity of the student neural network is increased to match the teacher neural network, then the mimic level is improved, but the device complexity and computational load increase beyond acceptable restrictions
Solution Approach 1:
The trained teacher neural network is decomposed into multiple subnetworks, where each subnetwork corresponds to a specific layer of the student neural network. This segmentation allows the student network to learn from targeted portions of the teacher network rather than requiring full complexity matching, resolving the contradiction between mimic level and device complexity.
Solution Approach 2:
The teacher neural network is pre-trained and decomposed into subnetworks before the student network training begins. The weight data from the teacher's subnetworks is prepared in advance and used as target outputs for training the corresponding student layers, enabling efficient knowledge transfer without requiring the student network to achieve full teacher complexity.
2Device complexity
If the student neural network complexity is restricted, then the device complexity is reduced, but the mimic level deteriorates
Solution Approach 1:
Instead of requiring uniform complexity across the entire student neural network, the invention applies local quality by training each student layer using weight data from the corresponding teacher subnetwork. This allows the student network to achieve high mimic level in specific local regions (layers) while maintaining overall structural simplicity.
Solution Approach 2:
The invention copies the weight data from the teacher neural network's subnetworks to the student neural network's layers during training. Rather than requiring the student network to independently discover complex patterns, it learns by copying and adapting the teacher's proven weight configurations, achieving high mimic level with restricted complexity.
3Productivity
If knowledge distillation is applied to train the student neural network, then the training efficiency is improved, but the computational load during the search process increases
Solution Approach 1:
The knowledge distillation process is segmented by decomposing the teacher network into subnetworks that correspond to student layers. This segmentation allows parallel processing of different layer trainings and avoids the computational burden of processing the entire teacher network at once, maintaining training efficiency while reducing peak computational load.
Data Source
AI summary
A neural network generation method includes: decomposing a trained teacher neural network including M layers into N subnetworks to generate a trained teacher neural network including N subnetworks; and generating a trained student neural network by (i) inputting a data set into each of the trained teacher neural network and a student neural network including N layers and (ii) training the student neural network. The generating of the trained student neural network includes: associating N teacher outputs and N student outputs in order of processing from an input layer toward an output layer; and determining weight data for each of the N layers in order of association, the N teacher outputs corresponding one to one to the N subnetworks, the N student outputs corresponding one to one to the N layers of the student neural network.