Knowledge distillation method based on reverse feature fusion and classifier reuse

Abstract

The present invention relates to the technical field of computers. Disclosed is a knowledge distillation method based on reverse feature fusion and classifier reuse. The present invention at least comprises the following steps: S1: acquiring an original data set, and dividing same into a training set and a test set; S2: preprocessing the original data set; S3: using the training set to pre-train a teacher network model, and saving the trained teacher network model; S4: introducing a decoupled attention projection module and a reverse feature fusion module into a student network by freezing pre-trained weights of a teacher network, and reusing a teacher classifier and a feature distance loss function to train a student model; and S5: during an inference stage, retaining a student network architecture and parameters of a teacher classifier layer. By means of the design of the method, the present invention can overcome the problem that a large discrepancy in intermediate feature representations between a teacher and a student hinders the student network from fully learning implicit knowledge in intermediate layer features of the teacher network. Moreover, by reusing the teacher classifier, an output distribution closer to that of the teacher is obtained.

Description

Knowledge distillation method based on inverse feature fusion and classifier reuse

[0001] This invention relates to the field of computer technology, specifically to a knowledge distillation method based on inverse feature fusion and classifier reuse. Background Technology

[0002] In recent years, the rapid development of deep neural networks (DNNs) has greatly promoted the progress of computer vision, and they have been widely used in tasks such as image classification, object detection, and image segmentation. However, powerful network performance is usually accompanied by a large number of parameters and high computational costs, making it difficult to deploy on resource-constrained devices. Therefore, how to reduce computational costs while ensuring model performance has become a challenging problem. Knowledge distillation provides an effective solution by transferring knowledge from a pre-trained large model to a lightweight model. Existing knowledge distillation methods are broadly divided into offline knowledge distillation and online knowledge distillation. Offline knowledge distillation adopts a two-stage training approach: first, a large teacher model is pre-trained, and then the extracted knowledge is transferred to a smaller student model to help the student learn the complex knowledge in the teacher model. Online knowledge distillation, on the other hand, adopts a single-stage training approach, directly optimizing the target model by continuously updating knowledge during training, allowing the student model to fully utilize the rich information from multiple outputs.

[0003] In the knowledge distillation method, there is a significant capability gap between the teacher model and the student model trained from scratch, resulting in a large difference in the intermediate feature representation between the teacher and student models. This makes it difficult for the student network to fully learn the implicit knowledge of the intermediate layer features of the teacher network.

[0004] Therefore, a new solution is needed to address the above problems. Summary of the Invention

[0005] The purpose of this invention is to provide a knowledge distillation method based on inverse feature fusion and classifier reuse, so as to overcome the problem that there is a large gap between the intermediate feature representations of teachers and students in knowledge distillation. This problem makes it difficult for the student network to fully learn the implicit knowledge of the intermediate layer features of the teacher network.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a knowledge distillation method based on inverse feature fusion and classifier reuse, comprising at least the following steps:

[0007] S1: Obtain the original dataset and divide it into training and test sets;

[0008] S2: Preprocess the original dataset;

[0009] S3: Use the training set to pre-train the teacher network model and save the trained teacher network model;

[0010] S4: By freezing the pre-trained weights of the teacher network, a decoupled attention projection module and an inverse feature fusion module are introduced into the student network, and the teacher classifier and feature distance loss function are reused to train the student model.

[0011] S5: During the inference phase, the parameters of the student network architecture and the teacher classifier layer are preserved.

[0012] Furthermore, S4 includes at least the following steps:

[0013] S4.1: Freeze the pre-trained weights of the teacher network model;

[0014] S4.2: Build decoupled attention projection modules at each stage of the student network to match the features of students at each stage with the feature dimensions of the teacher classifier;

[0015] S4.3: Use the reverse feature fusion module on the student network model to reversely fuse the features of each stage of deep and shallow layers layer by layer;

[0016] S4.4: The teacher classifier is reused to train the fused features, and the feature distance loss function is used as the distillation loss to train the student model.

[0017] Furthermore, the decoupled attention projection module includes a decoupled attention module and a projection module.

[0018] Furthermore, the decoupling attention module is encapsulated into a basic unit, for the first... Original features of the layer The data is fed into two paths for processing. One path performs average pooling on each channel within the height space, and the other path performs average pooling on each channel within the width space.

[0019] Then, one-dimensional convolutional layers are applied to the two paths respectively to enhance the original feature information in the decoupled horizontal and vertical directions;

[0020] Subsequently, group normalization was used to reduce interference between the two channels, enhance the positional information of the feature space dimension, and reduce the number of parameters.

[0021] Original features Horizontal attention Score and Vertical Attention The decoupling expression for fractions is:

[0022]

[0023] in, and These represent two one-dimensional convolutional layers for different paths. Used to represent group normalization This is used to represent the non-linear activation function Sigmoid, which modifies the attention scores obtained from the two paths with the features respectively. Multiply;

[0024] Features after decoupling the attention module The output expression is:

[0025] .

[0026] Furthermore, the projection module is encapsulated into a basic unit, which includes two... And an adaptive convolutional layer;

[0027] The two Convolutional layers are used to adjust the number of channels to the same number as the deep features; adaptive convolutional layers are used based on... The convolution kernel and stride are adaptively adjusted to match the spatial resolution of the student classifier with that of the teacher classifier. The projected features... Used for subsequent inverse feature fusion and teacher classifier reuse;

[0028] Features after projection module The output expression is:

[0029]

[0030] Design projection function This is used to transform the student's features at each stage after the attention module has been decoupled to the same spatial size as the teacher's deep features, where:

[0031]

[0032] in The space size is The weighted matrix.

[0033] Furthermore, S4.3 includes at least the following steps:

[0034] The first Features of the layer after projection and the fusion features of the next level To merge;

[0035] Then, the fused features are fed into two paths for processing, one of which contains two convolutional kernels. One path uses convolutional layers to capture local information, while another path uses additional global pooling layers to capture global information.

[0036] The feature information captured by the two paths is activated using the Sigmoid function to obtain weight scores, which are then compared with the feature values. , Multiplication, using parameters This is used to adjust the importance of deep and shallow features, ultimately resulting in the fused features at this level. ;

[0037] The overall network hierarchy is The deep feature map is used as the fused feature of the first input, that is ;

[0038] The feature expression of the subsequent reverse fusion output is:

[0039]

[0040] in It is the fusion function from the feature fusion module, and the final fused feature is represented as: .

[0041] Furthermore, S4.4 includes at least the following steps:

[0042] The pre-trained teacher's classifier layer parameters are reused in the fusion features, allowing the student model to perform inference by directly reusing the pre-trained teacher classifier. Feature alignment loss becomes the sole source of gradient generation. The total network hierarchy is Deep feature map of the teacher model;

[0043] Construct the feature distance loss function:

[0044] .

[0045] Furthermore, the parameters of the student network architecture and the teacher classifier layer that are retained during the inference phase in S5 are specifically as follows:

[0046] During the student inference phase, the teacher network model retains only the classifier layer parameters and the student network architecture portion, with almost no additional cost increase.

[0047] Compared with the prior art, the beneficial effects of the present invention are:

[0048] This invention overcomes the problem that significant differences in feature representation between teachers and students make it difficult for student networks to fully learn the implicit knowledge of the intermediate layers of the teacher network. Furthermore, by reusing the teacher classifier, it obtains an output distribution that more closely approximates the teacher's. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 is a flowchart of an embodiment of the present invention;

[0051] Figure 2 is a framework diagram of the overall knowledge distillation method according to an embodiment of the present invention;

[0052] Figure 3 is a diagram of the decoupled attention projection module according to an embodiment of the present invention;

[0053] Figure 4 is a diagram of the reverse feature fusion module according to an embodiment of the present invention;

[0054] Figure 5 is a graph showing the impact of the teacher classifier and label joint guidance on performance in an embodiment of the invention.

[0055] Figure 6 is a heatmap of the student fusion feature visualization analysis according to an embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0057] Please refer to Figure 1. The knowledge distillation method based on inverse feature fusion and classifier reuse includes at least the following steps:

[0058] Step 1: Obtain the original dataset and divide it into training and test sets;

[0059] In this embodiment, the original dataset obtained is the CIFAR-100 dataset, which contains 60,000 RGB images of size 32 × 32 in 100 categories. The training set and test set are divided into 50k and 10k images, respectively.

[0060] Step 2: Preprocess the original dataset;

[0061] In this embodiment, all images in the original dataset are processed and normalized using channel mean and standard deviation, and standard data augmentation operations are employed.

[0062] Step 3: Pre-train the teacher model using the training set and save the trained teacher model;

[0063] In this embodiment, the teacher model includes ResNet1 10×2, ResNet3 2×4, The experimental parameters for training the teacher model were set as follows: Batch Size = 64, Epoch = 240. The initial learning rate was set to 0.05, and the gradient learning rate method was used throughout, with the learning rate divided by 10 at Epochs of 150, 180, and 210. The optimizer used was the mechanical gradient descent method, with weight decay at [value missing]. Its momentum is 0.9.

[0064] Step 4: Freeze the pre-trained weights of the teacher network model, build decoupled projection modules at each stage of the student network; use the inverse feature fusion module to fuse features at each stage layer by layer; reuse the teacher classifier to fuse features, and use the feature distance loss function to train the student model.

[0065] In this embodiment, the student models include ResNet, WRN, ShuffleNet, and VGG. In the experimental parameters for training the student models, the initial learning rate for ShuffleNet and MobileNet is 0.02, while the initial learning rate for the other models is 0.05. Other parameter settings are the same as those for training the teacher models.

[0066] Step 4 includes:

[0067] Step 4-1: Freeze the pre-training weights of the teacher network model;

[0068] Step 4-2: Build decoupled attention projection modules at each stage of the student network to match the features of students at each stage with the feature dimensions of the teacher classifier.

[0069] Step 4-3: Use the reverse feature fusion module on the student network model to reverse fuse the features of each stage of the deep and shallow layers layer by layer.

[0070] Step 4-4: Reuse the teacher classifier to the fused features and train the student model using the feature distance loss function;

[0071] Specifically, during the training of the student model, the pre-training weights of the teacher model obtained in step 3 are frozen to ensure that the weights of the teacher model do not change during the training process.

[0072] Secondly, decoupled attention projection modules are built at each stage of the student network model. These modules consist of a decoupled attention module and a projection module. The decoupled attention module is encapsulated as a basic unit, for example, at the [stage name missing]. Original features of the layer The data is processed via two paths: one path performs average pooling on each channel in the height space, and the other path performs average pooling on each channel in the width space. Then, one-dimensional convolutional layers are applied to both paths to enhance the original feature information in the decoupled horizontal and vertical directions. Subsequently, group normalization is used to reduce interference between the two channels, enhancing the positional information in the feature space dimension and reducing the number of parameters. Original features Horizontal attention Score and Vertical Attention The decoupling expression for fractions is:

[0073]

[0074]

[0075] in, One-dimensional convolutional layers used to represent the two paths respectively Used to represent group normalization This is used to represent the non-linear activation function Sigmoid, which modifies the attention scores obtained from the two paths with the features respectively. Multiplication; Features after decoupling the attention module The output expression is:

[0076]

[0077] The projection module is encapsulated into a basic unit, containing two... And an adaptive convolutional layer. Two of them... Convolutional layers are used to adjust the number of channels to the same number as the deep features; adaptive convolutional layers are used based on... The convolution kernel and stride are adaptively adjusted to match the spatial resolution of the student classifier with that of the teacher classifier. The projected features... Used for subsequent inverse feature fusion and teacher classifier reuse; features after projection module The output expression is:

[0078]

[0079] Design projection function This is used to transform the student's features at each stage, after the attention module has been decoupled, into a spatial size similar to the teacher's deep features. ,in The space size is The weighted matrix.

[0080] Therefore, compared to directly using simple A feature projector constructed using convolutional kernels or linear methods achieves feature dimension matching between student features and the teacher classifier. The addition of a decoupled attention module enhances the expressive power of the original features by focusing on key spatial channel information, improving the efficiency of the projection module and achieving more accurate alignment with the teacher classifier. The decoupled attention module decouples the original features into horizontal and vertical feature vectors in the spatial dimension, capturing rich spatial information in both directions of the feature map, thus more efficiently enhancing important regions of features at each stage. It also merges the unique characteristics of the feature maps in both directions, improving the accuracy of attention prediction in each direction and enhancing the expressive power of the original features. The projection module uses adaptive convolutional layers to adaptively change the convolutional stride and kernel size according to the spatial resolution of the original features at each stage, ensuring that each feature is accurately projected to the same spatial size as the teacher's deep features. The enhanced expressive power of the original features further improves the alignment efficiency of the projection module, achieving efficient matching with the teacher classifier. This promotes the efficient fusion of multi-stage features with different scales and semantics.

[0081] As shown in Figure 2, in the reverse feature fusion module, the first step is to... Features of the layer after projection and the fusion features of the next level The features are then fused. The fused features are processed through two paths: one path contains two convolutional kernels. Convolutional layers capture local information; another path uses an additional global pooling layer to capture global information. The features captured by both paths are activated using the sigmoid function to obtain weight scores, which are then compared with the feature values... Multiplication, using parameters This is used to adjust the importance of deep and shallow features, ultimately resulting in the fused features at this level. .

[0082] The overall network hierarchy is The deep feature map is used as the fused feature of the first input, that is The expression for the reverse feature fusion module is:

[0083]

[0084] Therefore, the designed reverse feature fusion module realizes the fusion of shallow texture and deep abstract features, integrates feature information of adjacent stages, builds a bridge for information to flow from deep to shallow, enriches students' feature information, enhances feature expression, thereby shortening the feature difference between teachers and students, and enabling teachers to more efficiently impart knowledge to students' networks.

[0085] Furthermore, as shown in Figure 2, the teacher classifier is reused to fuse features, and the student model is trained using the feature distance loss function, specifically as follows:

[0086] The pre-trained teacher's classifier layer parameters are reused in the fusion features, allowing the student model to perform inference by directly reusing the pre-trained teacher classifier. Feature alignment loss becomes the sole source of gradient generation. The feature distance loss function is constructed as follows:

[0087]

[0088] Therefore, the designed reuse teacher classifier and feature distance loss function, by reusing the powerful discriminative classifier of the teacher model to the student model, enable students to imitate the teacher's representation space, better learn the teacher's representative knowledge, and improve the students' predictive performance. The richer fused features obtained through the inverse feature fusion module, compared with relying solely on single-layer features, can improve reuse efficiency, enhance the consistency between teacher and student models, and further narrow the performance gap.

[0089] Step 5: During the inference phase, retain the parameters of the student network architecture and the teacher classifier layer.

[0090] During the student inference phase, the teacher network model retains only the classifier layer parameters and the student network architecture portion, with almost no additional cost increase.

[0091] To verify the effectiveness of the proposed knowledge distillation method, the experimental results are shown in Tables 1 and 2.

[0092] Table 1 shows the experimental results of CIFAR-100 compared to the knowledge distillation method under the same teacher-student structure.

[0093] Table 2. Experimental results of CIFAR-100 compared to the knowledge distillation method under different teacher-student structures.

[0094] The experimental results in Table 1 demonstrate that the knowledge distillation method for inverse feature fusion and classifier reuse in this invention exhibits broad effectiveness. Compared to the baseline student model trained with cross-entropy in Vanilla KD, the accuracy of most teacher-student combinations is improved by at least 4%. Compared to state-of-the-art (SOTA) methods, it achieves optimal performance when using both homogeneous and heterogeneous architectures for both teachers and students. For example, it outperforms the SOTA method SimKD by 0.73% when using a combination of ResNet-32x4 as the teacher model and ResNet-8x4 as the student model.

[0095] Next, ablation experiments were conducted on the attention module, where -OA indicates that features were not projected through the decoupled attention module. The experiments demonstrated that the decoupled attention module effectively captured rich spatial information in both the horizontal and vertical directions, enhanced the expressive power of the original input features, achieved efficient matching with the teacher classifier, and improved the alignment efficiency of the projection module. Higher accuracy was achieved, proving the feasibility of this invention.

[0096] Table 3 compares the accuracy of RCRKD before and after using the decoupled attention module on CIFAR-100.

[0097] Next, ablation experiments were conducted on the inverse feature fusion module and classifier reuse, where +RFF indicates that features were only inversely fused, and +CR indicates that the student network model was reused by the teacher classifier. Experimental results show that the student model benefits from the fusion of shallow and deep features, significantly reducing the intermediate feature gap between teachers and students. Furthermore, the richer fused features improve the efficiency of classifier reuse, both of which effectively enhance the student model's recognition performance, achieving higher accuracy and demonstrating the feasibility of this invention.

[0098] Table 4 compares the accuracy of RCRKD classifier reuse and RFF module before and after using the CIFAR-100.

[0099] Figure 5 illustrates the impact of joint guidance from the teacher classifier and label information on student model performance under different weight combinations. In all settings, students performing significantly worse than those using inverse feature fusion and classifier reuse, even when simultaneously receiving the teacher classifier and label guidance, achieve performance close to that of the teacher simply by reusing the teacher classifier. This indicates a conflict between the different types of supervisory signals provided by the teacher classifier and the labels, making it difficult to integrate knowledge into the student model through joint training. Experimental results show that the student model can significantly bridge the performance gap between teachers and students by directly reusing the teacher classifier, without needing additional real labels or soft label guidance from the teacher.

[0100] As shown in Figure 6, to further evaluate the effectiveness of the inverse feature fusion method, we use heatmap visualization to highlight regions considered important for predicting the corresponding labels. Figure 3 illustrates that our proposed inverse feature fusion module successfully guides the student network's attention to detail-related regions closer to the teacher at each stage. Furthermore, the fused features successfully combine attentional spatial information from deep abstract features and shallow texture features, highlighting the effectiveness of the inverse feature fusion module in enhancing students' feature representation ability and reducing the differences between teachers and students. In contrast, the student baseline model sometimes treats the correct regions as background while focusing attention on spatially adjacent objects.

[0101] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A knowledge distillation method based on inverse feature fusion and classifier reuse, characterized in that: At least the following steps are included: S1: Obtain the original dataset and divide it into training and test sets; S2: Preprocess the original dataset; S3: Use the training set to pre-train the teacher network model and save the trained teacher network model; S4: By freezing the pre-trained weights of the teacher network, a decoupled attention projection module and an inverse feature fusion module are introduced into the student network, and the teacher classifier and feature distance loss function are reused to train the student model. S5: During the inference phase, the parameters of the student network architecture and the teacher classifier layer are preserved.

2. The knowledge distillation method based on inverse feature fusion and classifier reuse according to claim 1, characterized in that: The S4 includes at least the following steps: S4.1: Freeze the pre-trained weights of the teacher network model; S4.2: Build decoupled attention projection modules at each stage of the student network to match the features of students at each stage with the feature dimensions of the teacher classifier; S4.3: Use the reverse feature fusion module on the student network model to reversely fuse the features of each stage of deep and shallow layers layer by layer; S4.4: The teacher classifier is reused to train the fused features, and the feature distance loss function is used as the distillation loss to train the student model.

3. The knowledge distillation method based on inverse feature fusion and classifier reuse according to claim 2, characterized in that: The decoupled attention projection module includes a decoupled attention module and a projection module.

4. The knowledge distillation method based on inverse feature fusion and classifier reuse according to claim 3, characterized in that: The decoupled attention module is encapsulated into a basic unit, for the first... Original features of the layer The data is fed into two paths for processing. One path performs average pooling on each channel within the height space, and the other path performs average pooling on each channel within the width space. Then, one-dimensional convolutional layers are applied to the two paths respectively to enhance the original feature information in the decoupled horizontal and vertical directions; Subsequently, group normalization was used to reduce interference between the two channels, enhance the positional information of the feature space dimension, and reduce the number of parameters. Original features Horizontal attention Score and Vertical Attention The decoupling expression for the fraction is: ； in, and These represent two one-dimensional convolutional layers for different paths. Used to represent group normalization This is used to represent the non-linear activation function Sigmoid, which modifies the attention scores obtained from the two paths with the features respectively. Multiply; Features after decoupling the attention module The output expression is: 。 5. The knowledge distillation method based on inverse feature fusion and classifier reuse according to claim 4, characterized in that: The projection module is encapsulated into a basic unit, which includes two... And an adaptive convolutional layer; The two Convolutional layers are used to adjust the number of channels to the same number as the deep features; adaptive convolutional layers are based on... The convolution kernel and stride are adaptively adjusted to match the spatial resolution of the student classifier with that of the teacher classifier. The projected features... Used for subsequent inverse feature fusion and teacher classifier reuse; Features after projection module The output expression is: ； Design projection function This is used to transform the student's features at each stage after the attention module has been decoupled to the same spatial size as the teacher's deep features, where: ； in The space size is The weighted matrix.

6. The knowledge distillation method based on inverse feature fusion and classifier reuse according to claim 5, characterized in that: S4.3 includes at least the following steps: The first Features of the layer after projection and the fusion features of the next level To merge; Then, the fused features are fed into two paths for processing, one of which contains two convolutional kernels. One path uses convolutional layers to capture local information, while another path uses additional global pooling layers to capture global information. The feature information captured by the two paths is activated using the Sigmoid function to obtain weight scores, which are then compared with the feature values. ， Multiplication, using parameters This is used to adjust the importance of deep and shallow features, ultimately resulting in the fused features at this level. ； The overall network hierarchy is The deep feature map is used as the fused feature of the first input, that is ； The feature expression of the subsequent reverse fusion output is: ； in It is the fusion function from the feature fusion module, and the final fused feature is represented as: 。 7. The knowledge distillation method based on inverse feature fusion and classifier reuse according to claim 6, characterized in that: S4.4 includes at least the following steps: The pre-trained teacher's classifier layer parameters are reused in the fusion features, allowing the student model to perform inference by directly reusing the pre-trained teacher classifier. Feature alignment loss becomes the sole source of gradient generation. The total network hierarchy is Deep feature map of the teacher model; Construct the feature distance loss function: 。 8. The knowledge distillation method based on inverse feature fusion and classifier reuse according to claim 1, characterized in that: Specifically, in S5, the parameters of the student network architecture and the teacher classifier layer are retained during the inference phase as follows: During the student inference phase, the teacher network model retains only the classifier layer parameters and the student network architecture.

Images