Neural Network Knowledge Distillation Downscaling Layer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Knowledge Distillation (KD) techniques face challenges in training simple neural networks when the scale of layers between the teacher and student models differs, leading to data loss and reduced precision due to approximate estimation processes, especially in the FITNets method which requires multiple layers for hint layer correspondence.
Innovation Solution
A method that sets a downscaling layer in the teacher model with filter parameters identical to the student model, allowing for single-stage downscaling processing, reducing data loss and improving precision by grouping layers for more efficient parameter utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If FITNets technique is used to enable KD training with different layer scales, then training flexibility is improved, but data loss increases due to two approximate estimation processing steps
Solution Approach 1:
The patent extracts the downscaling operation from the student model side and places it in the teacher model side. By setting a downscaling layer in the teacher model with the same number of filters and filter kernel as the student model's layer to be trained, the method directly obtains scaled-down feature maps without needing approximate estimation, thus reducing data loss while maintaining training flexibility
Solution Approach 2:
The patent performs downscaling processing in advance in the teacher model before knowledge distillation. By pre-computing the downscaled feature maps with exact parameters matching the student model, the method eliminates the need for subsequent approximate estimation steps, reducing information loss while enabling cross-scale training
2Speed
If KD technique is used for model compression, then processing speed is improved, but training precision deteriorates when layer scales differ
Solution Approach 1:
The patent changes the parameter configuration by setting the downscaling layer's filter number and filter kernel to exactly match the student model's layer parameters. This parameter alignment enables direct feature map extraction without approximation, maintaining training precision while achieving model compression speed benefits
3Adaptability or versatility
If FITNets uses hint layer for cross-scale training, then layer scale adaptability is improved, but computational complexity increases due to multiple processing stages
Solution Approach 1:
The patent merges the downscaling operation with the teacher model's existing layer structure. By integrating the downscaling layer into the teacher model with matching parameters, the method eliminates the need for separate hint layers and multiple approximate estimation steps, reducing computational complexity while maintaining cross-scale adaptability
Data Source
AI summary
A training and application method for a neural network model is provided. The training method determines the first network model to be trained and sets a downscaling layer for at least one layer in the first network model, wherein the number of filters and filter kernel of the downscaling layer are identical to those of layers to be trained in the second network model. Filter parameters of the downscaling layer are transmitted to the second network model as training information. By this training method, training can also be performed even when the scale of the layer for training in the first network model is different from that of the layers to be trained in the second network model, and the amount of lost data is small.