Neural Network Feature Map Expansion for Many-to-One Knowledge Injection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing knowledge distillation methods for deep neural networks (DNNs) suffer from information loss due to feature dimension mismatch and inefficient one-to-one knowledge transfer, leading to suboptimal accuracy and computational inefficiency.
Innovation Solution
Implementing many-to-one knowledge injection through two layers in the target neural network, where the first layer expands feature maps and the second layer converts them to match the dimensions of the teacher network, allowing for improved knowledge transfer and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If one-to-one knowledge transfer is used between teacher and student networks, then the training process is simple, but information loss occurs due to feature dimension mismatch
Solution Approach 1:
The patent divides the feature map transformation process into multiple segments by introducing intermediate layers between the teacher and student networks. These intermediate layers segment the knowledge transfer process, allowing gradual dimension adaptation rather than direct one-to-one mapping, thereby reducing information loss while managing complexity.
Solution Approach 2:
The patent introduces intermediate neural network layers as mediators between the teacher and student networks. These intermediary layers perform feature dimension transformation and knowledge distillation, enabling efficient information transfer while adapting feature dimensions without direct one-to-one mapping.
2Adaptability or versatility
If feature dimension alignment is performed through transforms, then dimension mismatch is resolved, but information loss increases due to feature dimension reduction
Solution Approach 1:
The patent changes the dimensional transformation approach by introducing new feature dimensions through intermediate layers rather than reducing existing dimensions. This allows the student network to learn transformed features in an expanded dimensional space, preserving information while achieving dimension compatibility.
Solution Approach 2:
The patent performs preliminary feature transformation and dimension adaptation in intermediate layers before the final student network processing. This preliminary action prepares the features in a way that maintains information content while ensuring compatibility with the student network's expected input dimensions.
3Measurement precision
If deep neural networks are used to achieve high accuracy, then classification performance improves, but computation cost increases significantly
Solution Approach 1:
The patent creates a simplified copy of the teacher network's knowledge through the student network using knowledge distillation. The student network copies essential patterns and features from the teacher network but with reduced computational complexity, achieving comparable accuracy with lower computation cost.
Solution Approach 2:
The patent changes key parameters of the neural network through knowledge distillation, transferring optimized weight parameters and feature representations from the teacher network to the student network. This parameter transfer enables the student network to achieve high accuracy with fewer computational operations.
Data Source
AI summary
A target neural network can be trained with a support neural network through many-to-one knowledge injection. The many-to-one knowledge injection is facilitated by two layers inserted into the target neural networks. The first layer converts a target OFM in the target neural network into an expanded feature map having more channels. The second layer converts the expanded feature map to a new feature map having the same dimensions as the target OFM. The expanded feature map can be divided into segments, each of which has the same number of channels as a support OFM in the support neural network so that the knowledge in the support OFM can be injected into each of the segment through a many-to-one injection. To train the target neural network, parameters inside the target neural network are modified to minimize a feature distance between the expanded feature map and the support OFM.