Heterogeneous knowledge distillation method and device
By determining the contributions of teacher and student models in heterogeneous knowledge distillation, using attention mechanisms and matrix transformations for ranking and grouping, and training student models with alignment loss functions, the problem of low knowledge transfer efficiency between heterogeneous models is solved, achieving reduced computational complexity and improved performance.
Patent Information
- Application Number
- PCT/CN2025/094251
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-05-12
- Publication Date
- 2026-11-19
AI Technical Summary
Existing knowledge distillation techniques mainly focus on homogeneous models, lacking knowledge transfer between heterogeneous models, especially knowledge distillation between Transformer models and other models, resulting in high computational complexity and low efficiency.
By determining the contributions of the teacher and student models, processing units are sorted and grouped based on attention mechanisms and matrix transformations, and the student model is trained using an alignment loss function to achieve heterogeneous knowledge distillation.
It improves the efficiency of knowledge transfer between heterogeneous models, reduces computational complexity, and enhances the performance of student models.
Smart Images

Figure CN2025094251_19112026_PF_FP_ABST
Abstract
Description
Methods and apparatus for heterogeneous knowledge distillation Technical Field
[0001] This disclosure relates to the field of knowledge distillation, and more specifically to methods and apparatus for heterogeneous knowledge distillation. Background Technology
[0002] In recent years, the Transformer model has been widely used in many fields. However, the time complexity of the Transformer model is proportional to the square of the length of the input sequence. As the length of the input sequence increases, the computational complexity increases significantly, leading to high computational overhead and resource consumption when processing long sequence tasks.
[0003] Currently, various sequence models (e.g., Mamba model, linear attention model, etc.) have been proposed to improve the above problems and reduce computational complexity, but these models lack a rich ecosystem.
[0004] Therefore, there is a need to utilize the Transformer model, which has a rich ecosystem, as a teacher model to transfer knowledge from the teacher model to the sequence model.
[0005] However, existing knowledge distillation techniques mostly focus on knowledge transfer between homogeneous models (such as between Transformer models), while there is a lack of research on knowledge distillation between heterogeneous models (such as between Transformer models and Mamba models).
[0006] Therefore, there is a need to achieve knowledge distillation between student and teacher models when the student model is heterogeneous to the teacher model, while ensuring computational efficiency and improving performance. Summary of the Invention
[0007] A brief overview of this disclosure is provided below to provide a basic understanding of certain aspects of it. It should be understood that this overview is not an exhaustive summary of the disclosure. It is not intended to identify key or essential parts of the disclosure, nor is it intended to limit its scope. Its purpose is merely to present certain concepts in a simplified form as a prelude to the more detailed description that follows.
[0008] According to one aspect of this disclosure, a method for heterogeneous knowledge distillation is provided, comprising: a teacher model-related contribution determination process, wherein, for a plurality of processing units in the input sequence of each layer of a trained teacher model, a teacher model-related contribution of each processing unit to each layer of the teacher model is determined; a sorting and grouping process, wherein the plurality of processing units are sorted based on the teacher model-related contribution of each processing unit and grouped according to the sorting result; a student model-related contribution determination process, wherein, based on information related to the grouping of processing units, a plurality of processing units of a student model heterogeneous with the teacher model are grouped, and a student model-related contribution equivalent to the teacher model-related contribution of each processing unit to each layer of the student model is determined; a contribution alignment process, wherein an alignment loss function is determined based on the teacher model-related contribution and the student model-related contribution of the processing units, wherein the alignment loss function is used to align the teacher model-related contribution and the student model-related contribution; and a training process, wherein the student model is trained based on the alignment loss function.
[0009] According to one aspect of this disclosure, the teacher model is a sequence model, and the student model is a sequence model heterogeneous with the teacher model.
[0010] According to one aspect of this disclosure, when the teacher model is a Transformer model, the teacher model-related contribution of the processing unit is determined based on the attention weights of the processing units under the Transformer attention mechanism and at least one in the output feature map of the processing unit; when the teacher model is an efficient sequence model, the teacher model-related contribution of the processing unit is determined based on the matrix transformation form associated with the teacher model and at least one in the output feature map of the processing unit.
[0011] According to one aspect of this disclosure, when the student model is a Transformer model, the student model-related contribution of the processing unit is determined based on the attention weights of the processing unit under the Transformer attention mechanism and at least one in the output feature map of the processing unit; when the student model is an efficient sequence model, the student model-related contribution of the processing unit is determined based on the matrix transformation form associated with the student model and at least one in the output feature map of the processing unit.
[0012] According to one aspect of this disclosure, an alignment loss function is determined based on one of the following, such that the distributions of student model-related contributions and teacher model-related contributions of processing units are consistent: determining the alignment loss function based on a comparison between the sums of student model-related contributions of processing units in each group of the student model; or determining the alignment loss function based on the Kullback-Leibler Divergence between student model-related contributions and teacher model-related contributions of processing units in the input sequence; or determining the alignment loss function based on the cross-entropy loss of processing units in each group of the student model.
[0013] According to one aspect of this disclosure, the processing unit can be divided into two or more groups.
[0014] According to one aspect of this disclosure, efficient sequence models include the Mamba model or a linear attention model.
[0015] According to one aspect of this disclosure, training a student model based on an alignment loss function includes: determining whether the consistency between the student model's relevant contribution and the teacher model's relevant contribution in each group processing unit is greater than a predetermined threshold; if the consistency is greater than the predetermined threshold, grouping continues for the teacher model in the ranking grouping process, and the student model's relevant contribution determination process, contribution alignment process, and training process are repeated; if the consistency is less than or equal to the predetermined threshold, determining whether a predetermined training objective is met; if the predetermined training objective is not met, the teacher model's relevant contribution determination process, ranking grouping process, student model's relevant contribution determination process, contribution alignment process, and training process are repeated to train the student model; if the predetermined training objective is met, the method ends.
[0016] According to one aspect of this disclosure, the method is used for an AI architecture that includes an encoder.
[0017] According to one aspect of this disclosure, an apparatus for heterogeneous knowledge distillation is disclosed, comprising: a teacher model-related contribution determination unit configured to: determine the teacher model-related contribution of each processing unit to each layer of the teacher model for a plurality of processing units in the input sequence of each layer of a trained teacher model; a sorting and grouping unit configured to sort the plurality of processing units based on the teacher model-related contribution of each processing unit and group the plurality of processing units according to the sorting result; a student model-related contribution determination unit configured to: group the plurality of processing units of a student model heterogeneous with the teacher model based on information related to the grouping of processing units, and determine the student model-related contribution of each processing unit to each layer of the student model that is equivalent to the teacher model-related contribution; a contribution alignment unit configured to: determine an alignment loss function based on the teacher model-related contribution and the student model-related contribution of the processing units, wherein the alignment loss function is used to align the teacher model-related contribution and the student model-related contribution; and a training unit configured to train a student model based on the alignment loss function.
[0018] According to one aspect of this disclosure, the contribution alignment unit is configured to determine an alignment loss function based on one of the following, such that the distributions of student model-related contributions and teacher model-related contributions of the processing unit are consistent:
[0019] The alignment loss function is determined by comparing the sum of the student model-related contributions of the processing units in each group of the student model; or
[0020] The alignment loss function is determined based on the Kullback-Leibler divergence between the student model's relevant contribution and the teacher model's relevant contribution in the input sequence; or
[0021] The alignment loss function is determined by the cross-entropy loss of the processing units in each group of the student model.
[0022] According to one aspect of this disclosure, the apparatus for heterogeneous knowledge distillation is used in an AI architecture that includes an encoder.
[0023] According to one aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a computer, enables the implementation of the method for heterogeneous knowledge distillation according to this disclosure.
[0024] These and other advantages of this disclosure will become more apparent from the following detailed description of preferred embodiments of the disclosure in conjunction with the accompanying drawings. Attached Figure Description
[0025] To further illustrate the above and other advantages and features of this disclosure, a more detailed description of specific embodiments of this disclosure is provided below with reference to the accompanying drawings. The drawings, together with the following detailed description, are included in and form a part of this specification. Elements having the same function and structure are indicated by the same reference numerals. It should be understood that these drawings only depict typical examples of this disclosure and should not be construed as limiting the scope of this disclosure.
[0026] In the attached diagram:
[0027] Figure 1 schematically illustrates examples of isomorphic and heterogeneous models;
[0028] Figure 2 schematically illustrates isomorphic knowledge distillation and heteromorphic knowledge distillation;
[0029] Figure 3A shows an exemplary flowchart of a method for heterogeneous knowledge distillation according to an embodiment of the present disclosure;
[0030] Figure 3B further illustrates an exemplary process of step S306 in Figure 3A;
[0031] Figure 4 illustrates how to obtain the average equivalent attention vector of a processing unit for a representative processing unit;
[0032] Figure 5 shows a block diagram of the structure of an apparatus for heterogeneous knowledge distillation according to an embodiment of the present disclosure; and
[0033] Figure 6 shows a block diagram of an exemplary structure of a general-purpose personal computer in which methods and / or networks according to embodiments of the present disclosure can be implemented.
[0034] While this disclosure is readily subject to various modifications and substitutions, specific embodiments thereof have been shown by way of example in the accompanying drawings and are described in detail herein. However, it should be understood that the description of specific embodiments herein is not intended to limit this disclosure to the specific forms disclosed, but rather, this disclosure is intended to cover all modifications, equivalents, and substitutions falling within the spirit and scope of this disclosure. It should be noted that throughout the drawings, corresponding reference numerals indicate corresponding parts. Detailed Implementation
[0035] This disclosure will now be described more fully with reference to the accompanying drawings. The following description is merely exemplary and is not intended to limit the disclosure, its application, or its uses.
[0036] Example embodiments are provided so that this disclosure will become exhaustive and will fully convey its scope to those skilled in the art. Numerous specific details, such as examples of particular components, apparatus, and methods, are set forth to provide a detailed understanding of embodiments of this disclosure. It will be apparent to those skilled in the art that the specific details are not required, and that the example embodiments may be implemented in many different forms, none of which should be construed as limiting the scope of this disclosure. In some example embodiments, well-known processes, well-known structures, and well-known techniques are not described in detail.
[0037] It should also be noted that, in order to avoid obscuring this disclosure with unnecessary details, only the equipment structure and / or processing steps closely related to the solution according to this disclosure are shown in the accompanying drawings, while other details that are not closely related to this disclosure are omitted.
[0038] Different models are called homogeneous models if they share the same architecture, and heterogeneous models if they have different architectures. Figure 1 illustrates examples of homogeneous and heterogeneous models.
[0039] As shown in Figure 1, two Transformer models or two Mamba models using the same architecture are isomorphic even if some parameters (such as the number of layers and dimensions) differ. Similarly, two ResNet models with the same structure are also isomorphic, even if they have different numbers of layers, namely 50 layers (ResNet-50) and 18 layers (ResNet-18).
[0040] Referring back to Figure 1, models with different architectures are heterogeneous models. For example, one is a Transformer model and the other is a Mamba model; or, one is a ResNet model and the other is a Transformer model; or, one is a ResNet model and the other is a Mamba model, etc.
[0041] Knowledge distillation is a technique that transfers knowledge from a large, complex model (the teacher model) to a smaller model (the student model). Depending on whether the teacher and student models are isomorphic or heterogeneous, knowledge distillation can be divided into isomorphic knowledge distillation and heterogeneous knowledge distillation.
[0042] Figure 2 schematically illustrates isomorphic knowledge distillation and heterogeneous knowledge distillation. As shown in Figure 2, isomorphic knowledge distillation refers to knowledge distillation when the teacher model and the student model are isomorphic, while heterogeneous knowledge distillation refers to knowledge distillation when the teacher model and the student model are heterogeneous.
[0043] Traditional heterogeneous knowledge distillation methods typically align the distribution of features or output predictions (e.g., the distribution of Logits) of different models (such as Transformer and CNN models), but neglect the alignment at the attention level between models.
[0044] This disclosure proposes attention-based knowledge distillation between heterogeneous models to achieve attention alignment across different architectural models.
[0045] The following analysis of different models will discuss the attention or equivalent attention of each model.
[0046] The Transformer model is based on a multi-head attention mechanism. The attention mechanism of the Transformer model can be expressed as Equation (1).
[0047] Where Q is the query vector, representing the current position where attention needs to be calculated, Q = xW q , x is the input sequence, W q To query the weight matrix;
[0048] K is the key vector, representing the identifiers at other locations, used together with Q to calculate the correlation, K = xW k W k This is the key weight matrix;
[0049] V is a value vector representing the actual information, V = xW v W v The value weight matrix.
[0050] Efficient sequence models refer to models that model input sequences. Examples of efficient sequence models include the Mamba model and the linear attention model. The Mamba model is a linear sequence modeling model based on the Selective State Space (SSM) mechanism.
[0051] The selective state-space mechanism will be discussed further below.
[0052] Equations (2)-(3) below are the discrete formulas for the selective state space of the Mamba model. Wherein, equation (2) represents the state update of the Mamba model, and equation (3) represents the output of the Mamba model. t =A t h t-1 +B t x t (2)
[0053] In equation (2), h tLet h be the hidden state at time t. t-1 Let A be the hidden state of the time preceding time t-1. t It is a state transition matrix, representing the dependence of the current state on the state at the previous time step, B. t It is an input mapping matrix, representing the input x that controls the current time t. t The hidden state h at time t t The impact.
[0054] In equation (3), y t The output at the current time t, To output the mapping matrix C t The transpose of is used to map the hidden state to the output space.
[0055] Equation (2) above can be rewritten as:
[0056] In equation (4), h0 = B0x0, where Combining equations (3) and (4), we can derive:
[0057] In summary, the output y of the Mamba model can be expressed as equation (5). y = SSM(A,B,C)(x) = Mx (5)
[0058] In equation (5), M is the equivalent attention weight matrix, where the element in the j-th row and i-th column of M can be expressed as equation (6).
[0059] Equation (6) above can also be expressed as Where 0≤i≤j≤L, and L is the length of the input sequence x.
[0060] For other efficient sequence models besides the Mamba model, such as the linear attention model, the model output can also be represented as a matrix transformation as shown in equation (7): y = QK T V = xW q (xW k ) T xW v =Mx (7)
[0061] From the above equations (1), (5) and (7), it can be seen that for the output of Transformer models and efficient sequence models (e.g., Mamba model, linear attention model, etc.) under different architectures, the output of the model can be represented as a certain matrix transformation form. For Mamba model and linear attention model, the equivalent attention of efficient sequence model can be calculated based on the global equivalent attention matrix M.
[0062] Based on the above, the method for heterogeneous knowledge distillation according to the present disclosure will be described below with reference to FIG3A. FIG3A shows an exemplary flowchart of the method for heterogeneous knowledge distillation according to an embodiment of the present disclosure.
[0063] The method for heterogeneous knowledge distillation begins at step S301 in Figure 3A.
[0064] In step S302, for each layer of the trained teacher model, the contribution of each processing unit to the teacher model at each layer is determined for multiple processing units in the input sequence. Here, the processing unit is the basic unit (i.e., a token) of the input sequence. The teacher model can be a Transformer model or an efficient sequence model. It should be understood that this disclosure is not limited to these; the teacher model can also be other sequence models besides Transformer models and efficient sequence models. For example, it can be the RWKV model mentioned in the paper "RWKV: Reinventing RNNs for the Transformer Era" published on arXiv in 2023 by Bo Peng, Eric Alcaide, Quentin Anthony, et al., or the TTT model mentioned in the paper "Learning to (Learn at Test Time): RNNs with Expressive Hidden States" published on arXiv in 2024 by Yu Sun, Xinhao Li, Karan Dalal, et al., etc.
[0065] Specifically, when the teacher model is a Transformer model, the teacher model-related contribution of the processing unit can be determined based on the attention weight of the processing unit under the Transformer attention mechanism, or based on the output feature map of the processing unit, or based on both the attention weight and the output feature map of the processing unit.
[0066] When the teacher model is an efficient sequence model, the teacher model-related contribution of the processing unit can be determined based on the matrix transformation form related to the teacher model, or based on the output feature map of the processing unit. Alternatively, the teacher model-related contribution of the processing unit can be determined based on both the matrix transformation form related to the teacher model and the output feature map of the processing unit.
[0067] In step S303, multiple processing units are sorted based on the teacher model-related contribution of each processing unit, and the multiple processing units are grouped according to the sorting results.
[0068] In step S304, based on information related to the grouping of processing units, multiple processing units of the student model that are heterogeneous with the teacher model are grouped, and the student model-related contribution of each processing unit to each layer of the student model that is equivalent to the contribution of the teacher model is determined.
[0069] The student model is a heterogeneous model from the teacher model. The student model can be a Transformer model, an efficient sequence model, or other sequence models heterogeneous to the teacher model. For example, the student model can be the RWKV model mentioned in the 2023 arXiv paper "RWKV: Reinventing RNNs for the Transformer Era" by Bo Peng, Eric Alcaide, Quentin Anthony, et al., or the TTT model mentioned in the 2024 arXiv paper "Learning to (Learn at Test Time): RNNs with Expressive Hidden States" by Yu Sun, Xinhao Li, Karan Dalal, et al., etc. For instance, when the teacher model is a Transformer model, the student model can be a Mamba model or a linear attention model; when the teacher model is a Mamba model, the student model can be a linear attention model or a Transformer model; when the teacher model is a linear attention model, the student model can be a Transformer model or a Mamba model. It should be understood that the above description is only for illustrative purposes regarding teacher and student models and is not intended to impose limitations.
[0070] When the student model is a Transformer model, the contribution of the processing unit to the student model can be determined based on the attention weights of the processing unit under the Transformer attention mechanism, or based on the output feature map of the processing unit, or based on both the attention weights and the output feature map of the processing unit. When the student model is an efficient sequence model, the contribution of the processing unit to the student model can be determined based on the matrix transformation form related to the student model, or based on the output feature map of the processing unit, or based on both the matrix transformation form related to the student model and the output feature map of the processing unit.
[0071] In step S305, an alignment loss function is determined based on the teacher model relevance contribution and the student model relevance contribution of the processing unit, wherein the alignment loss function is used to align the teacher model relevance contribution and the student model relevance contribution. The alignment loss function can be determined based on one of the following to ensure a consistent distribution between the student model relevance contribution and the teacher model relevance contribution of the processing unit: determining the alignment loss function based on a comparison of the sums of the student model relevance contributions of the processing units in each group of the student model; or determining the alignment loss function based on the Kullback-Leibler divergence between the student model relevance contribution and the teacher model relevance contribution of the processing units in the input sequence; or determining the alignment loss function based on the cross-entropy loss of the processing units in each group of the student model. It should be understood that this disclosure is not limited thereto, and other implementations of alignment loss functions are also included within the scope of this disclosure.
[0072] In step S306, the student model is trained based on the alignment loss function. Then the method ends.
[0073] The steps for training the student model based on the alignment loss function shown in Figure 3A (i.e., step S306 in Figure 3A) will be described in detail below with reference to Figure 3B.
[0074] Figure 3B shows a flowchart detailing the steps of training a student model based on the alignment loss function in Figure 3A according to an embodiment of the present invention.
[0075] In step S3061, it is determined whether the consistency between the student model-related contribution and the teacher model-related contribution of each group processing unit is greater than a predetermined threshold. If the consistency is greater than the predetermined threshold, the method for heterogeneous knowledge distillation of this disclosure proceeds to step S3062. If the consistency is less than or equal to the predetermined threshold, the method for heterogeneous knowledge distillation of this disclosure proceeds to step S3063.
[0076] In step S3062, the teacher model grouping is further grouped, that is, the grouping obtained in step S303 of Figure 3A is further grouped, and the processing of step S304 and the steps after step S304 in the method for heterogeneous knowledge distillation is repeated.
[0077] In step S3063, it is determined whether a predetermined training objective is met. If the predetermined training objective is not met, step S302 and the steps following step S302 in the method for heterogeneous knowledge distillation are repeated to train the student model. If the predetermined training objective is met, the method ends. According to the embodiments of this disclosure, the predetermined training objective may be determining whether the alignment loss function during training meets predetermined conditions. If the alignment loss function meets predetermined conditions, it can be determined that the training objective has been achieved, or the training has reached the maximum number of iterations. It should be understood that this disclosure is not limited to this, and other implementations of the training objective are also included within the scope of this disclosure.
[0078] It should be understood that in step S306 of FIG3A, the above-mentioned steps S3061 and S3062 may be optional. According to the embodiment of the present invention, in step S306, the processing of step S3063 can be performed directly.
[0079] The following details the model-related contributions of the computational processing units in the above-described method for heterogeneous knowledge distillation. It should be noted that since both the teacher and student model-related contributions can be calculated using the processing described here, for clarity, the following descriptions of model-related contributions will not differentiate between teacher and student models.
[0080] First, calculate the attention weights or equivalent attention weights of the input sequence. Let x be the input sequence. in The length is L.
[0081] For the Transformer model, the attention weights are obtained according to formula (1). Where R represents the real number space, B is the batch size, H is the number of attention heads, and L... Q L is the length of the query sequence. KGiven the length of the key sequence, under the self-attention mechanism, L Q =L K =L, B×H×L Q ×L K The dimension of the real number space.
[0082] For efficient sequence models (e.g., the Mamba model), the equivalent attention weights obtained according to formula (6) Where R represents the real number space, and L×L is the dimension of the real number space. m This represents the feature dimension of the processing unit in the Mamba model.
[0083] Then, the average attention weight of each processing unit in the input sequence for the final output is calculated.
[0084] For the Transformer model, the i-th processing unit in the input sequence Average attention weight It can be expressed as equation (8):
[0085] Where b∈B, h∈H, q∈L Q , k∈L K A b,h,q,k This represents the attention weights of the b-th sample, the h-th attention head, the q-th position of the query sequence, and the k-th position of the key sequence within a batch size B.
[0086] For the Mamba model, the equivalent attention weight M at each position can be calculated according to equation (6). b,d,r,c ,in, D m This represents the feature dimension of the processing unit in the Mamba model.
[0087] Similar to the Transformer model, the average equivalent attention weight of each processing unit in the input sequence can be calculated according to Equation (9).
[0088] Where b is the batch processing dimension, d is the feature dimension, r is the row of the M matrix, and c is the column of the M matrix.
[0089] Since M is calculated in equation (9) above... b,d,r,c The complexity is too high. Therefore, according to the following equation (10), the equivalent attention vector M corresponding to the Fth processing unit (token) in the input sequence is used. b,d,F To characterize Compared to the square matrix form of Mb,d,r,c M b,d,F We only need to calculate the equivalent attention vector for the row corresponding to the Fth processing unit.
[0090] In equation (10), 0 ≤ F ≤ L.
[0091] Figure 4 illustrates how to obtain the average equivalent attention vector of a processing unit for a representative processing unit. As shown in Figure 4, a globally representative processing unit can be selected based on the Mamba model architecture and task requirements. The black block in Figure 4 represents the selected F-th processing unit as the globally representative processing unit. For a Mamba model with a bidirectional encoder structure (i.e., Bi-Mamba architecture), a summation processing unit (CLS Token) can be selected. The equivalent attention weights corresponding to the forward and backward branches of the encoder structure are calculated using forward calculation and backward calculation, respectively. Finally, the equivalent attention weight results of the forward and backward calculations are concatenated and transposed to obtain the average equivalent attention vector of all processing units for the representative processing unit.
[0092] For Mamba models without an encoder structure, or in scenarios without a summation processing unit (CLS Token), the last processing unit in the input sequence is selected as the representative processing unit by default. We use the equivalent attention weight vector of a processing unit relative to other processing units, where the attention weight of the processing unit relative to itself can be set to 1.0 after normalization. Therefore, we can also obtain the equivalent attention weight of each input processing unit.
[0093] Furthermore, for linear attention models, QK is easily obtained. T Similarly, the average attention weight of the processing unit can be calculated using formula (5).
[0094] Based on the average attention weights of the processing units obtained above, the attention weight or equivalent attention weight of the k-th processing unit can be obtained.
[0095] For an input sequence comprising L (L≥1) processing units, there are two ways to calculate the correlation contribution of the k-th processing unit (k∈L). In this way.
[0096] Method 1: The relevant contribution can be calculated directly using attention weights or equivalent attention weights. The relevant contribution of the k-th processing unit (k∈L) It can be obtained by the following formula (11).
[0097] Method 2: The correlation contribution can be calculated by multiplying the attention weights or equivalent attention weights element-wise with the current layer's output feature map. The correlation contribution of the k-th processing unit (k∈L) is then calculated. It can be obtained through the following equation (12), where, It is the output feature map of the kth processing unit. This represents the average attention weight of the processing unit.
[0098] In equations (11)-(12) above, when the teacher model or student model is a Transformer model, Let be the average attention weight of the k-th processing unit. This is relevant when the teacher or student model is an efficient sequence model. Let be the equivalent attention weight of the k-th processing unit, which is a matrix transformation related to the model. It should be understood that the calculation of the correlation contribution of the processing units in this invention is not limited to this; according to embodiments of this invention, the correlation contribution of the processing units can also be determined based on the output feature map of the processing units.
[0099] The following further details the grouping process related to the above methods for heterogeneous knowledge distillation.
[0100] For the input sequence, determine the teacher model-related contribution of each processing unit in the input sequence, sort the teacher model-related contributions of the processing units in descending order, and divide the sorted results into n groups, denoted as .
[0101] The student model learns from the grouping of the teacher model, resulting in the grouping of the student model.
[0102] The student model determines the relevant contribution of each processing unit in the input sequence, sorts the relevant contributions of the processing units in descending order, and divides the sorted results into n groups, denoted as nn.
[0103] Optionally, the crossover ratio (IoU) can be calculated based on whether the consistency between the student model's relevant contribution and the teacher model's relevant contribution within each group is greater than a predetermined threshold. For example, according to an embodiment of the present invention, the crossover ratio (IoU) can be calculated using equation (13). index To measure consistency.
[0104] in, express and The number of elements in the intersection of the two.
[0105] If the intersection-combination ratio index IoU index If the value exceeds a predetermined threshold, it indicates a higher degree of consistency between the student model's contribution and the teacher model's contribution in the processing unit. Therefore, further grouping of the teacher model is necessary. If the Intersection over Union (IoU) index is also high... index If the value is less than or equal to a predetermined threshold, it indicates a low degree of consistency between the student model's contribution and the teacher model's contribution in the processing unit, and further grouping of the teacher model is unnecessary. For example, for... Re-sort the teacher model-related contributions of the processing units corresponding to each element, and then continue grouping, for example, into m groups, where m≥2. Then perform operations similar to those after the grouping above. It should be recognized that n and m do not necessarily have to be equal.
[0106] To better understand the scheme of this disclosure, let's take an input sequence x = (x1, x2, x3, x4, x5, x6, x7, x8) comprising 8 processing units as an example. For this input sequence, the teacher model determines the teacher model's relevant contribution to each processing unit and ranks them. For example, the ranking result is as follows: The teacher's contribution to the model for the k-th processing unit (1≤k≤8):
[0107] Based on the ranking of the teacher model's contribution to the processing units, the teacher model divides the processing units in the input sequence into two groups. Processing units x5, x2, x3, and x4 with higher teacher model contribution belong to one group, while processing units x1, x6, x7, and x8 with relatively lower teacher model contribution belong to the other group. According to the embodiments of this disclosure, for each group, the grouping result can be recorded by simply recording the sequence number of the processing unit in the input sequence. For example, for the grouping of the teacher model... in, It should be recognized that the embodiments of the present invention are not limited thereto, and the teacher model can also divide the processing units into other numbers of groups, for example, into n groups, n≥2.
[0108] Then, according to the corresponding grouping of the teacher model The processing unit in the input sequence obtains the grouping learned by the student model.
[0109] The student model determines the relevant contribution of each processing unit and ranks them. For example, the ranking results are as follows: The contribution of the student model to the k-th processing unit (1≤k≤8):
[0110] The student models are grouped based on their ranking of contributions to each processing unit. For example, the processing units are grouped according to their ranking of contributions.
[0111] The cross-union index IoU, used to represent the consistency between the student model's relevant contribution and the teacher model's relevant contribution of the processing unit, is calculated according to the above formula (13). index Intersection over Union (IoU) index index The higher the value, the greater the consistency between the student model's contribution and the teacher model's contribution in the processing unit. In this example, the length L of the input sequence is 8.
[0112] If the calculated crossover ratio index IoU index If the value exceeds a predetermined threshold (e.g., 0.5), the teacher model needs to be further subdivided. This is especially important if the Intersection over Union (IoU) index is also considered. index If the values are less than or equal to a predetermined threshold, further grouping of the teacher model is not required. For example, [the following text is incomplete and requires further context: "will"]. The teacher model-related contributions of the processing units corresponding to the serial numbers are re-sorted and then divided into m groups, where m ≥ 2. Then, operations similar to those performed after the grouping are executed. It should be understood that the above-mentioned further subdivision of the teacher model groups based on the consistency between the student model-related contributions and the teacher model-related contributions of the processing units is not necessary. According to the embodiments of the present invention, the above-mentioned subdivision of the groups can be ignored. Furthermore, the number of groups in the initial grouping and the subsequent grouping of the teacher model need not be the same; that is, n and m need not be equal in the above description.
[0113] The following further explains the details related to the alignment loss function in the above methods for heterogeneous knowledge distillation. The alignment loss function aligns the relevant contributions of the teacher model and the student model. Different quantification methods can be used to determine the alignment loss function based on the relevant contributions of the teacher model and the student model of the processing unit.
[0114] In this scheme, the alignment loss must be added to different groups within the same level of grouping; that is, the number of elements in each group that should have the alignment loss is equal. Grouping at different levels can be treated as multiple alignment losses superimposed.
[0115] According to embodiments of this disclosure, the alignment loss function can be determined based on a comparison between the sums of the student model-related contributions of processing units in each group of the student model. Specifically, the alignment loss function L... attn As shown in equation (14):
[0116] in, represent Total number of elements in the group. represent The mean contribution of elements within the group, where δ represents the penalty term. δ can be a minimum value, for example, δ can be 10. -6 .
[0117] According to embodiments of this disclosure, the alignment loss function can also be determined based on the KL divergence (Kullback-Leibler Divergence) between the student model correlation contribution and the teacher model correlation contribution of the processing units in the input sequence. Specifically, the alignment loss function L is determined as shown in equation (15). attn L attn =KL(Pt||Ps) (15)
[0118] Where KL() represents the KL divergence function, and Pt and Ps represent the distributions of the teacher model and the student model, respectively. The distribution of the teacher model's contribution to the teacher model group can be used as Pt, and the distribution of the student model's contribution to the student model group can be used as Ps. The alignment loss function is shown in equation (16).
[0119] In formula (16), N represents the number of groups. This represents the number of elements in the i-th group.
[0120] According to embodiments of this disclosure, the alignment loss function can also be determined using the cross-entropy loss of the processing units in each group of the student model.
[0121] The Boolean mask, represented by equation (17), can be obtained based on the group that appears first in the teacher model, i.e., the group with the highest contribution to the teacher model.
[0122] in, This represents the group with the highest attention contribution in the teacher model grouping. For example, the teacher model is divided into two groups. but for The Boolean mask has the same dimension as the length of the input sequence, as shown in equation (17). In this context, the location of the processing unit is set to True, and vice versa. At this point, the alignment loss function L... attn It can be expressed as equation (18):
[0123] Where y mask =I(Mask), y mask ∈R LR is a real number field, L is the number of processing units in the input sequence, and I(Mask) is used to process the MASK obtained by equation (17) into a mask with the same dimension as the input processing units. σ represents the relevant contribution of the i-th processing unit in the student model, and σ is the sigmoid activation function.
[0124] Figure 5 shows a block diagram of the structure of an apparatus 500 for heterogeneous knowledge distillation according to an embodiment of the present disclosure. As shown in Figure 5, the apparatus 500 for heterogeneous knowledge distillation may include a teacher model related contribution determination unit 510, a sorting and grouping unit 520, a student model related contribution determination unit 530, a contribution alignment unit 540, and a training unit 550.
[0125] The processing corresponding to the heterogeneous knowledge distillation method shown in Figures 3A-3B can be implemented by the device 500. Specifically, the heterogeneous knowledge distillation device 500 in Figure 5 can include a teacher model contribution determination unit 510, a sorting and grouping unit 520, a student model contribution determination unit 530, a contribution alignment unit 540, and a training unit 550 to implement the processing in steps S302-S306 of Figure 3A. Details regarding each unit in the heterogeneous knowledge distillation device 500 have been given in detail in the description of the method in Figures 3A-3B and will not be repeated here.
[0126] The various components and units in the aforementioned apparatus for heterogeneous knowledge distillation can be configured via software, firmware, hardware, or a combination thereof. Specific means or methods of configuration are well known to those skilled in the art and will not be elaborated upon here. When implemented via software or firmware, the programs constituting the software are installed from a storage medium or network onto a computer with a dedicated hardware architecture (e.g., the general-purpose computer 500 shown in Figure 5). When various programs are installed, the computer is capable of performing various functions.
[0127] The inventors conducted experiments on the UCF-Crime dataset in the video anomaly detection task. The comparison results of the key indicators of each experiment are shown in Table 1 below. It can be found that the performance is significantly improved by utilizing the heterogeneous knowledge distillation method disclosed in this paper.
[0128] Table 1 Comparison Results of Key Indicators
[0129] Experiment 1 is characterized by relying solely on a simple task classification; Experiment 2 is characterized by optimizing the model's original output Logits loss and classification loss; and Experiment 3 is characterized by optimizing the knowledge distillation, Logits loss, and classification loss based on the attention mechanism disclosed in this publication. It can be seen that Experiment 3, through cross-model attention alignment, solves the problem of traditional methods neglecting attention adaptation, resulting in improvements in key metrics such as precision, recall, F1 score, and area under the receiver operating characteristic curve (ROC AUC).
[0130] Figure 6 shows a block diagram of an exemplary structure of a general-purpose personal computer in which methods and / or networks according to embodiments of the present disclosure can be implemented. As shown in Figure 6, a central processing unit (CPU) 501 performs various processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. Data required when the CPU 601 performs various processes, etc., is also stored as needed in RAM 603. The CPU 601, ROM 602, and RAM 603 are connected to each other via bus 604. An input / output interface 605 is also connected to bus 604.
[0131] The following components are connected to input / output interface 605: input device 606 (including keyboard, mouse, etc.), output device 607 (including display, such as cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.), storage device 608 (including hard disk, etc.), and communication device 609 (including network interface card, such as LAN card, modem, etc.). Communication device 609 performs communication processing via a network such as the Internet. Drive 610 may also be connected to input / output interface 605 as needed. Removable media 611, such as disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 610 as needed, so that computer programs read from them can be installed into storage device 608 as needed.
[0132] When the above series of processes are implemented through software, the program constituting the software is installed from a network such as the Internet or a storage medium such as removable medium 611.
[0133] Those skilled in the art will understand that such storage media are not limited to the removable medium 611 shown in FIG. 6, which stores programs and is distributed separately from the device to provide programs to users. Examples of removable media 611 include magnetic disks (including floppy disks (registered trademark)), optical disks (including optical disc read-only memory (CD-ROM) and digital versatile disks (DVD)), magneto-optical disks (including mini-disk (MD) (registered trademark)), and semiconductor memories. Alternatively, the storage medium may be ROM 602, a hard disk included in storage device 608, etc., which stores programs and is distributed to users along with the device containing them.
[0134] This disclosure also provides corresponding computer program code and a computer program product storing machine-readable instruction code. When these instruction codes are read and executed by a machine, the methods described above according to embodiments of this disclosure can be performed.
[0135] Accordingly, the storage medium configured to carry the aforementioned program product storing machine-readable instruction code is also included in this disclosure. Storage media include, but are not limited to, floppy disks, optical disks, magneto-optical disks, memory cards, memory sticks, etc.
[0136] Based on the above description, the embodiments of this disclosure provide the following technical solutions, but are not limited thereto.
[0137] Finally, it should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Furthermore, unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0138] While embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, it should be understood that the embodiments described above are merely illustrative of the present disclosure and do not constitute a limitation thereof. Those skilled in the art can make various modifications and alterations to the above embodiments without departing from the spirit and scope of the present disclosure. Therefore, the scope of the present disclosure is defined only by the appended claims and their equivalents.
Claims
1. A method for heterogeneous knowledge distillation, comprising the steps of: a. determining, for each processing unit in an input sequence of each layer of a trained teacher model, a teacher model related contribution of each processing unit to each layer of the teacher model, b. sorting the plurality of processing units based on the teacher model related contribution of each processing unit and grouping the plurality of processing units according to the result of the sorting; c. based on information related to the grouping of processing units, determining, for a plurality of processing units of a student model that is heterogeneous to the teacher model, a student model related contribution of each processing unit to each layer of the student model that is equivalent to the teacher model related contribution, d. determining an alignment loss function based on the teacher model related contribution and the student model related contribution of the processing units, wherein the alignment loss function is used to align the teacher model related contribution and the student model related contribution; and e. training the student model based on the alignment loss function. The teacher model is a sequence model, and the student model is a sequence model that is heterogeneous to the teacher model.
2. The method of claim 1, wherein, 3. The method of claim 1, wherein, when the teacher model is a Transformer model, the teacher model related contribution of a processing unit is determined based on at least one of an attention weight of the processing unit under a Transformer attention mechanism and an output feature map of the processing unit, when the teacher model is an efficient sequence model, the teacher model related contribution of a processing unit is determined based on at least one of a matrix transformation form related to the teacher model and an output feature map of the processing unit.
4. The method of claim 3, wherein, when the student model is the Transformer model, the student model related contribution of a processing unit is determined based on at least one of an attention weight of the processing unit under a Transformer attention mechanism and an output feature map of the processing unit, when the student model is the efficient sequence model, the student model related contribution of a processing unit is determined based on at least one of a matrix transformation form related to the student model and an output feature map of the processing unit. The alignment loss function is determined based on one of the following, such that the distribution between the student model related contribution and the teacher model related contribution of a processing unit is consistent:
5. The method of claim 4, wherein, the alignment loss function is determined based on a comparison between the sum of the student model related contribution of the processing units in each group of the student model; or the alignment loss function is determined based on the KL divergence between the student model related contribution and the teacher model related contribution of the processing units in the input sequence; or the alignment loss function is determined based on the cross-entropy loss of the processing units in each group of the student model.
6. The method of claim 1, wherein the processing units can be divided into two or more groups. The efficient sequence model includes a Mamba model or a linear attention model.
7. The method of claim 1, wherein, 8. The method of any one of claims 1 to 7, wherein, training the student model based on the alignment loss function comprises: determining whether a consistency degree between the student model related contribution degree and the teacher model related contribution degree of the processing unit in each group is greater than a predetermined threshold, if the consistency degree is greater than the predetermined threshold, continuing grouping for the group of step b, and repeating steps c to e, if the consistency degree is less than or equal to the predetermined threshold, determining whether a predetermined training target is met, if the predetermined training target is not met, repeating the steps a to e to train the student model, and if the predetermined training target is met, ending the method.
9. The method of any one of claims 1 to 7, wherein, The method is used for an AI architecture comprising an encoder.
10. An apparatus for heterogeneous knowledge distillation, comprising: a teacher model related contribution degree determination unit configured to determine, for a plurality of processing units in an input sequence of each layer of a trained teacher model, a teacher model related contribution degree of each processing unit to each layer of the teacher model; a sorting and grouping unit configured to sort the plurality of processing units based on the teacher model related contribution degree of each processing unit and group the plurality of processing units according to the sorting result; a student model related contribution degree determination unit configured to group a plurality of processing units of a student model heterogeneous to the teacher model based on information related to the grouping of the processing units, and determine a student model related contribution degree equivalent to the teacher model related contribution degree of each processing unit to each layer of the student model; a contribution degree alignment unit configured to determine an alignment loss function based on the teacher model related contribution degree and the student model related contribution degree of the processing units, wherein the alignment loss function is used to align the teacher model related contribution degree and the student model related contribution degree; and a training unit configured to train the student model based on the alignment loss function.
11. The apparatus of claim 10, wherein, The teacher model is a sequence model, and the student model is a sequence model heterogeneous to the teacher model.
12. The apparatus of claim 10, wherein, When the teacher model is a Transformer model, the teacher model related contribution degree of the processing unit is determined based on at least one of an attention weight of the processing unit under a Transformer attention mechanism and an output feature map of the processing unit, When the teacher model is an efficient sequence model, the teacher model related contribution degree of the processing unit is determined based on at least one of a matrix transformation form related to the teacher model and an output feature map of the processing unit.
13. The apparatus of claim 12, wherein, When the student model is the Transformer model, the student model related contribution degree of the processing unit is determined based on at least one of an attention weight of the processing unit under a Transformer attention mechanism and an output feature map of the processing unit, When the student model is the efficient sequence model, the student model related contribution degree of the processing unit is determined based on at least one of a matrix transformation form related to the student model and an output feature map of the processing unit.
14. The apparatus of claim 13, wherein, The contribution alignment unit is configured to determine the alignment loss function based on one of: a comparison between a sum of student model related contributions of processing units in each grouping of the student model, and a KL divergence between student model related contributions and teacher model related contributions of processing units in the input sequence, and a cross-entropy loss of processing units in respective groupings of the student model.
15. The apparatus of claim 10, wherein the processing units can be grouped into two or more groups.
16. The apparatus of claim 10, wherein, The high-efficiency sequence model comprises a Mamba model or a linear attention model.
17. The apparatus of any one of claims 10 to 16, wherein, The apparatus is used in an AI architecture comprising an encoder.
18. A computer readable storage medium having stored thereon a computer program, which, when executed by a computer, is capable of implementing the method of any one of claims 1-9.
