Method of adjusting model and related device

Abstract

Embodiments of the present application provide a method of adjusting a model and related devices for adjusting a model. This method includes: obtaining a gating network logit value of each expert network in a first MoE layer corresponding to each token in T tokens based on the T tokens and a target MoE model; obtaining К selected expert networks corresponding to each token in the T tokens based on the logit value of each expert network in the first MoE layer corresponding to each token in the T tokens; obtaining an adjusted first MoE layer by pruning at least one expert network in the first MoE layer based on the К selected expert networks corresponding to each token in the T tokens; obtaining an adjusted target MoE model based on the adjusted first MoE layer. The proposed technique is applied for each MoE model layer in layer-by-layer manner and adjusts a MoE model to reduce a cost of resources required for deploying the MoE model.

Description

METHOD OF ADJUSTING MODEL AND RELATED DEVICETECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of computers, and more particularly, to a method of adjusting a model, and related devices for adjusting a model.BACKGROUND

[0002] A mixture-of-experts (MoE) technique, compared to a traditional Large Language Model (LLM), increases the number of parameters of a model while controlling cost and latency. A MoE model includes at least one MoE layer, each MoE layer of the MoE model includes a gating network and expert networks. Each expert network in the MoE layer is used to handle a specific task. The gating network is used to output a probability of choosing each expert network to process a token. At least one expert network in the MoE layer is activated to process a token when the token is input into the MoE layer. In other words, at least one expert network in the MoE layer is not activated to process a token and still requires some storage resources.

[0003] Therefore, how to adjust the MoE model to reduce a cost of resources required for deploying the MoE model is a challenge.SUMMARY

[0004] Embodiments of the present application provide a method of adjusting a model and related devices for adjusting a model. The technical solution adjusts a MoE model to reduce a cost of resources required for deploying the MoE model.

[0005] According to a first aspect, an embodiment of this application provides a method of adjusting a model. This method includes: based on T tokens and a target MoE model, obtaining a gating network logit value of each expert network in a first MoE layer corresponding to each token in the T tokens, where the first MoE layer belongs to at least one MoE layer in the target MoE model, the first MoE layer includes N1expert networks and the gating network, the logit value ofeach expert network corresponding to each token is used to represent a probability of activating each expert network to process each token, T is a positive integer, and Ni is a positive integer greater than 1; based on the logit value of each expert network in the first MoE layer corresponding to each token, obtaining K selected expert networks corresponding to each token in the T tokens, where the K selected expert networks corresponding to each token are such that a value of a first loss function is maximized, the value of the first loss function is determined based on a first penalty value and a logit value of each selected expert network in the K selected expert networks corresponding to each token, K is a positive integer less than Ni, based on the number of selected expert networks, and / or, based on a logit value of each initially activated expert network in K initially activated expert networks corresponding to each token and the logit value of each expert network in the K selected expert networks corresponding to each token, the first penalty value is determined, and the logit value of each initially activated expert network corresponding to each token is greater than or equal to the logit value of each expert network in the N1expert networks corresponding to each token other than the K initially activated expert networks; based on the K selected expert networks corresponding to each token in the T tokens, obtaining an adjusted first MoE layer by pruning at least one expert network in the first MoE layer, where the number of expert networks in the adjusted first MoE layer is less than N1; and obtaining an adjusted target MoE model based on the adjusted first MoE layer, where the adjusted target MoE model includes the adjusted first MoE layer.

[0006] According to the method above, the K selected expert networks corresponding to each token in the T tokens are determined by utilizing the logit value of each expert network in the first MoE layer corresponding to each token in T tokens, thereby pruning the first MoE layer and obtaining the adjusted target MoE model. Due to the fact that the number of expert networks in the adjusted first MoE layer is less than the number of expert networks in the first MoE layer, a cost of resources required for deploying the adjusted target MoE model is reduced.

[0007] In a possible design, the value of the first loss function includes a difference between the first penalty value and a sum of logit values of the K selected expert networks corresponding to each token in the T tokens.

[0008] According to the method above, the larger the logit value of the K selected expert networks corresponding to each token in T tokens, the larger the value of the first loss function.The smaller the first penalty value, the larger the first loss function. Therefore, according to this method, the logit value of each selected expert network in the K selected expert networks corresponding to each token in the T tokens can be maximized, thereby reducing the impact on the performance of the adjusted target MoE model.

[0009] In a possible design, the first penalty value is determined based on the number of selected expert networks, and the first penalty value is a first value; or, the first penalty value is determined based on the logit value of each initially activated expert network in the K initially activated expert networks corresponding to each token and the logit value of each expert network in the K selected expert networks corresponding to each token in the T tokens, and the first penalty value is a second value; or, the first penalty value is determined based on the number of selected expert networks, the logit value of each initially activated expert network in the K initially activated expert networks corresponding to each token and the logit value of each expert network in the K selected expert networks corresponding to each token in the T tokens, and the first penalty value is a sum of a first value and a second value. Among them, the first value is positively correlated with the number of selected expert networks, and the second value is positively correlated with a difference between the logit value of each initially activated expert network in the K initially activated expert networks corresponding to each token and the logit value of each expert network in the K selected expert networks corresponding to each token in the T tokens.

[0010] In a possible design, N3 selected expert networks are obtained by removing at least one duplication on the K selected expert networks corresponding to each token in the T tokens, and N3is a positive integer less than N1. The N3selected expert networks include the K selected expert networks corresponding to each token in T tokens, and the N3selected expert networks do not include the same expert network. That is, the number of selected expert networks is N3. The N3selected expert networks are also referred to as N3retained expert networks.

[0011] According to the method above, the fewer the number of selected expert networks, the smaller the first penalty value, and the larger the value of the first loss function. The smaller the difference between the logit value of the kthinitially activated expert network corresponding to each token and the logit value of the kthselected expert network, the smaller the first penalty value, and the larger the value of the first loss function. Therefore, according to this method, the number of selected expert networks is controlled, thereby reducing the cost of resources required fordeploying the adjusted target MoE model.

[0012] In a possible design, the first value includes a product of a first penalty factor and the number of selected expert networks, and the first penalty factor is a preset value; or, the first value includes a dot product of a second penalty factor and a first vector, the second penalty factor includes an N1-dimensional vector, the first vector includes an N1-dimensional binary vector, and the first vector is used to represent the number of selected expert networks.

[0013] According to the method above, the impact of the number of selected expert networks on the first penalty value can be controlled by determining the first value through a penalty factor and the number of selected expert networks, thereby making it easier to control the number of selected expert networks.

[0014] In a possible design, the second value includes a sum of differences between logit values of the K initially activated expert networks corresponding to each token and logit values of the K selected expert networks corresponding to each token in the T tokens.

[0015] According to the method above, the smaller the difference between the logit value of the kthinitially activated expert network corresponding to each token and the logit value of the kthselected expert network corresponding to each token, the smaller the second value, and the larger the value of the first loss function. Therefore, according to this method, it is encouraged to choose the initially activated expert network as the selected expert network as much as possible, thereby reducing the impact on the performance of the adjusted target MoE model.

[0016] In a possible design, a first retained expert network belongs to the adjusted first MoE layer, each pruned expert network in at least one pruned expert network belongs to the N1expert networks other than expert networks in the adjusted first MoE layer, and the method further includes: updating a weight of the first retained expert network based on a weight of each pruned expert network in the at least one pruned expert network and the weight of the first retained expert network, where the updated weight of the first retained expert network is determined by a weighted sum of the weight of each pruned expert network in the at least one pruned expert network and the weight of the first retained expert network.

[0017] According to the method above, the knowledge of each pruned expert network is retained by merging the weights of each pruned expert in the at least one pruned expert network and the first retained expert network, thereby improving the performance of the adjusted MoEmodel.

[0018] In a possible design, a difference between a first logit value of the first retained expert network corresponding to a first token and a first logit value of a first pruned expert network corresponding to the first token is less than or equal to a first preset threshold, the first token belongs to the T tokens, and the first pruned expert network belongs to the at least one pruned expert network.

[0019] According to the method above, the difference in logit values between the pruned expert network and the retained expert network used to merge weights is less than or equal to a preset threshold. Therefore, the processing effect obtained by using the retained expert network and the pruned expert network to process the same token is similar, thereby reducing the impact on the performance of the adjusted target MoE model.

[0020] In a possible design, a second MoE layer belongs to the first MoE model, the second MoE layer includes N2expert networks, N2is a positive integer greater than 1, and the obtaining an adjusted target MoE model based on the adjusted first MoE layer includes: obtaining an adjusted second MoE layer, where the number of expert networks in the adjusted second MoE layer is less than or equal to N2; and obtaining the adjusted target MoE model based on the adjusted first MoE layer and the adjusted second MoE layer, where the number of expert networks in the adjusted first MoE layer and the number of expert networks in the adjusted second MoE layer are different.

[0021] According to the method above, by pruning MoE layers in the target MoE model, expert networks included in at least two adjusted MoE layers are different in number, that is, compression ratios of at least two MoE layers are different. Different compression ratios can improve the performance of the adjusted target MoE model.

[0022] According to a second aspect, an embodiment of this application provides a device for adjusting a model. This device includes: an obtaining unit, configured to: based on T tokens and a target MoE model, obtain a gating network logit value of each expert network in a first MoE layer corresponding to each token in the T tokens, where the first MoE layer belongs to at least one MoE layer in the target MoE model, the first MoE layer includes N1expert networks and the gating network, the logit value of each expert network corresponding to each token is used to represent a probability of activating each expert network to process each token, T is a positive integer, and Ni is a positive integer greater than 1; and a processing unit, configured to: based on the logit valueof each expert network in the first MoE layer corresponding to each token, obtain K selected expert networks corresponding to each token, where the K selected expert networks corresponding to each token are such that a value of a first loss function is maximized, the value of the first loss function is determined based on a first penalty value and a logit value of each selected expert network in the K selected expert networks corresponding to each token, K is a positive integer less than N1, based on the number of selected expert networks, and / or, based on a logit value of each initially activated expert network in K initially activated expert networks corresponding to each token and the logit value of each expert network in the K selected expert networks corresponding to each token, the first penalty value is determined, and the logit value of each initially activated expert network corresponding to each token is greater than or equal to the logit value of each expert network in the N1expert networks corresponding to each token other than the K initially activated expert networks; based on the K selected expert networks corresponding to each token in the T tokens, obtain an adjusted first MoE layer by pruning at least one expert network in the first MoE layer, where the number of expert networks in the adjusted first MoE layer is less than N1; and obtain an adjusted target MoE model based on the adjusted first MoE layer, where the adjusted target MoE model includes the adjusted first MoE layer.

[0023] In a possible design, the value of the first loss function includes a difference between the first penalty value and a sum of logit values of the K selected expert networks corresponding to each token in the T tokens.

[0024] In a possible design, the first penalty value is determined based on the number of selected expert networks, and the first penalty value is a first value; or, the first penalty value is determined based on the logit value of each initially activated expert network in the K initially activated expert networks corresponding to each token and the logit value of each expert network in the K selected expert networks corresponding to each token in the T tokens, and the first penalty value is a second value; or, the first penalty value is determined based on the number of selected expert networks, the logit value of each initially activated expert network in the K initially activated expert networks corresponding to each token and the logit value of each expert network in the K selected expert networks corresponding to each token in the T tokens, and the first penalty value is a sum of a first value and a second value. Among them, the first value is positively correlated with the number of selected expert networks, and the second value is positivelycorrelated with a difference between the logit value of each initially activated expert network in the K initially activated expert networks corresponding to each token and the logit value of each expert network in the K selected expert networks corresponding to each token in the T tokens.

[0025] In a possible design, N3 selected expert networks are obtained by removing at least one duplication on the K selected expert networks corresponding to each token in the T tokens, and N3is a positive integer less than N1. The N3selected expert networks include the K selected expert networks corresponding to each token in T tokens, and the N3selected expert networks do not include the same expert network. That is, the number of selected expert networks is N3. The N3selected expert networks are also referred to as N3retained expert networks.

[0026] In a possible design, the first value includes a product of a first penalty factor and the number of selected expert networks, and the first penalty factor is a preset value; or, the first value includes a dot product of a second penalty factor and a first vector, the second penalty factor includes an N1-dimensional vector, the first vector includes an N1-dimensional binary vector, and the first vector is used to represent the number of selected expert networks.

[0027] In a possible design, the second value includes a sum of differences between logit values of the K initially activated expert networks corresponding to each token and logit values of the K selected expert networks corresponding to each token in the T tokens.

[0028] In a possible design, a first retained expert network belongs to the adjusted first MoE layer, each pruned expert network in at least one pruned expert network belongs to the N1expert networks other than expert networks in the adjusted first MoE layer, and the processing unit is configured to update a weight of the first retained expert network based on a weight of each pruned expert network in the at least one pruned expert network and the weight of the first retained expert network, where the updated weight of the first retained expert network is determined by a weighted sum of the weight of each pruned expert network in the at least one pruned expert network and the weight of the first retained expert network.‘

[0029] In a possible design, a difference between a first logit value of the first retained expert network corresponding to a first token and a first logit value of a first pruned expert network corresponding to the first token is less than or equal to a first preset threshold, the first token belongs to the T tokens, and the first pruned expert network belongs to the at least one pruned expert network.

[0030] In a possible design, a second MoE layer belongs to the first MoE model, the second MoE layer includes N2expert networks, N2is a positive integer greater than 1, and the processing unit is configured to: obtain an adjusted second MoE layer, where the number of expert networks in the adjusted second MoE layer is less than or equal to N2; and obtain the adjusted target MoE model based on the adjusted first MoE layer and the adjusted second MoE layer, where the number of expert networks in the adjusted first MoE layer and the number of expert networks in the adjusted second MoE layer are different.

[0031] According to a third aspect, an embodiment of this application provides a computing device cluster, including at least one computing device, where the computing device includes a processor and a memory coupled with the processor, where the memory is configured to store a computer program, and the processor is configured to invoke and run the computer program stored in the memory, so that the computing device executes the method in any one of the first aspect or any possible design of the first aspect.

[0032] According to a fourth aspect, an embodiment of this application provides a computer program product including instructions, where when the computer program product is run on a server, the server is enabled to perform the method in any one of the first aspect or any possible design of the first aspect.

[0033] According to a fifth aspect, an embodiment of this application provides a computer readable storage medium including instructions, where when run on a server, the computer readable storage medium enables the server to perform the method in any one of the first aspect or any possible design of the first aspect.DESCRIPTION OF DRAWINGS

[0034] FIG. 1 is a schematic block diagram of a MoE model.

[0035] FIG. 2 is a schematic diagram of a method of processing a token by a MoE layer in a MoE model.

[0036] FIG. 3 is a schematic diagram of a method of adjusting a model according to an embodiment of this application.

[0037] FIG. 4 is a schematic diagram of interchangeability of expert networks according to anembodiment of this application.

[0038] FIG. 5 is a schematic diagram of K selected expert networks corresponding to a token according to an embodiment of this application.

[0039] FIG. 6 is a schematic diagram of K selected expert networks corresponding to another token according to an embodiment of this application.

[0040] FIG. 7 is a schematic diagram of a method of adjusting a model according to another embodiment of this application.

[0041] FIG. 8 is a schematic diagram of a method of adjusting a model according to another embodiment of this application.

[0042] FIG. 9 is a schematic diagram of a device for adjusting a model according to an embodiment of this application.

[0043] FIG. 10 is a schematic block diagram of a computing device according to an embodiment of this application.

[0044] FIG. 11 is a schematic block diagram of a computing device cluster according to an embodiment of this application.

[0045] FIG. 12 is a schematic block diagram of a computing device 1000A and a computing device 1000B connected by a network according to an embodiment of this application.DESCRIPTION OF EMBODIMENTS

[0046] The following describes the technical solutions in the present application with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without making creative labor shall fall within the scope of protection of the present application.

[0047] The present application will present aspects, embodiments, or features around systems that include multiple devices, components, modules, etc. It should be understood and appreciated that the individual systems may include additional devices, components, modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in connection with the accompanying drawings. In addition, combinations of these options may be used.

[0048] In addition, in the embodiments of the present application, the word "exemplarily" and the phrase "as an example" are used to indicate for example, illustration or description. Any embodiment or design solution described as "exemplarily" in this application should not be construed as being superior to or more advantageous than other embodiments or design solutions. Rather, the use of the word "example" is intended to present the concept in a specific manner.

[0049] The phrases "in some possible embodiments", "in some possible application scenarios", etc., appearing in various places in this description, do not necessarily refer to the same embodiments, but rather mean "one or more, but not all, embodiments" unless otherwise specifically emphasized. Unless otherwise specifically emphasized, the terms "including", "comprising", "having", and variations thereof all mean "including but not limited to".

[0050] In the present application, "at least one" refers to one or more, and "multiple" refers to two or more, "and / or", describing the association of the associated objects, indicates that three relationships can exist. For example, A and / or B can mean A alone, both A and B, and B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following associated objects are in an "or" relationship.

[0051] The application scenarios described in the present application embodiments are intended to illustrate the technical solutions of the present application embodiments more clearly and do not constitute a limitation to the technical solutions provided by the present application embodiments. It is known to those of ordinary skill in the art that the technical solutions provided by the present application embodiments are equally applicable to similar technical problems as the system architecture evolves and new application scenarios emerge.

[0052] The technical solution in the embodiments of the present application is applied to computing devices, such as servers, hosts, personal computers, laptops, desktop computers, etc. The server is a local server or a cloud server. This technical solution is applied to a cloud management platform when the server is a cloud server.

[0053] FIG. 1 is a schematic block diagram of a MoE model. As shown in FIG. 1, a MoE model 100 includes at least one MoE layer, and each MoE layer includes a gating network and expert networks. The gating network is used to output a probability of choosing each expert network in the MoE layer to process a token. The expert network is used to process a token and output processed data. The functions of each expert network in the MoE layer are different, that is,ioeach expert network is used to process a specific task.

[0054] For example, the MoE model 100 includes a MoE layer 110 and / or a MoE layer 120. The MoE layer 110 includes a gating network 111, an expert network 112, and an expert network 113. The MoE layer 120 includes a gating network 121, an expert network 122, and an expert network 123. The MoE layer 110 and / or the MoE layer 120 can also include more expert networks.

[0055] In some embodiments, at least two MoE layers in the MoE model 100 are connected. An output of one of the at least two MoE layers is used as an input of the other MoE layer of the at least two MoE layers. For example, the MoE layer 110 is connected to the MoE layer 120, and an output of the MoE layer 110 is used as an input of the MoE layer 120.

[0056] In some embodiments, a method of processing a token by a MoE layer is shown in FIG.2. FIG. 2 is a schematic diagram of a method of processing a token by a MoE layer in a MoE model. As shown in FIG. 2, a token 210 is fed into a gating network 220 in a MoE layer 200. Outputs of the gating network 220 includes a probability of each expert network in the MoE layer 200, and the probability of each expert network is used to represent the probability of choosing this expert network to process the token 210. The token 210 is fed into at least one expert network in the MoE layer 200 to obtain an output of each expert network.

[0057] For example, the output of the MoE layer 200 is obtained according to following formulas:

[0058] (1)

[0059] G(x)i = Softmax(TopK(x · Wg)i) (2)

[0060] TopK(x · Wg)i = (x · Wg)i, if (x · Wg)i is among the top − K coordinate of logits x · WgI— oo, otherwise

[0061] Among them, x is a token, Ei(x) represents an output of the ithexpert network in N expert networks in the MoE layer 200, and G (x) i represents a probability of the ithexpert network obtained by the gating network 220, i=0,..., N-l. is a summation symbol. Softmax( ) represents a normalized exponential function. Wg represents a weight matrix of the gating network 220, and (x · Wg)i is a score of the ithexpert network corresponding to x and is obtained by the gating network 220. The higher the score, the higher the correlation between x and the ithexpert network, and the more likely the ithexpert network processes x effectively. A score of each expert networknin the top K expert networks belongs to top K among scores of the N expert networks, and the scores of the N expert networks include each score of each expert network in the N expert networks. TopK(x · Wg)i = (x · Wg)i when the ithexpert network belongs to the top K expert networks, otherwise TopK(x · Wg)i = (x · Wg)i = −∞ (go to zero after softmax).

[0062] For example, when the MoE layer 200 includes an expert network 230 and an expert network 240, the token 210 is input into the expert network 230, and an output of the expert network 230 is obtained as an output 231. The token 210 is input into the expert network 240, and an output of the expert network 240 is obtained as an output 241. The token 210 is input into the gating network 220 to obtain a probability of the expert network 230 (i.e. probability 221) and a probability of the expert network 240 (i.e. probability 222). The output 231 is multiplied by the probability 221 to obtain an output 232, and the output 241 is multiplied by the probability 222 to obtain an output 242. The output 232 and the output 242 are added to obtain an output 250 of the MoE layer 200.

[0063] For example, when three or more expert networks are included in the MoE layer 200, the probabilities of the expert network 230 and the expert network 240 are greater than or equal to probabilities of expert networks other than the expert network 230 and the expert network 240 in the MoE layer 200, that is, the probabilities of the expert network 230 and the expert network 240 are top 2 among the probabilities of the expert networks in the MoE layer 200.

[0064] FIG. 3 is a schematic diagram of a method of adjusting a model according to an embodiment of this application. The method in FIG. 3 is applied by a computing device. The method in FIG. 3 includes the following steps.

[0065] Step 310: obtaining a gating network logit value of each expert network in a first MoE layer corresponding to each token in T tokens based on the T tokens and a target MoE model.

[0066] Optionally, before step 310, the computing device obtains a calibration dataset and a target MoE model. The calibration dataset includes T tokens, where T is a positive integer. The target MoE model includes at least one MoE layer, each MoE layer includes a gating network and expert networks. The gating network and expert networks are described in FIG. 1 and FIG. 2.

[0067] For example, the calibration dataset includes a Wikitext dataset.

[0068] For example, the target MoE model is a pre-trained model, which means that the target MoE model has good processing performance when processing at least one token. For example,the target MoE model is an LLM.

[0069] For example, the target MoE model includes a first MoE layer, which includes a gating network and Ni expert networks, and Ni is a positive integer greater than 1.

[0070] In some embodiments, the computing device inputs each token in T tokens into the first MoE layer to obtain a logit value of each expert network in the first MoE layer corresponding to each token. The logit value of each expert network corresponding to each token is used to represent a probability of choosing the expert network to process the token.

[0071] For example, a logit value Lt iof the ithexpert network in the first MoE layer corresponding to the tthtoken in the T tokens is determined according to following formula:

[0072] Lt,i= Softmax ((t · Wg)i) (4)

[0073] Among them, t is the tthtoken in the T tokens. Wgrepresents a weight matrix of the gating network in the first MoE layer, (t · Wg)iis a score of the ithexpert network in the N1expert networks corresponding to t and is obtained by the gating network in the first MoE layer, and i=0,..., Ni- 1. The higher the score, the higher the correlation between t and the ithexpert network, and the more likely the ithexpert network processes t effectively. Softmax( ) represents a normalized exponential function.

[0074] In some embodiments, when the tthtoken is processed by an expert network A, if a difference between a logit value of the expert network A corresponding to the tthtoken and a logit value of an expert network B corresponding to the tthtoken is less than or equal to a first preset threshold, the tthtoken can also be processed by the expert network B. In other words, the impact of the expert network B processing the tthtoken on the processing effect is relatively small. Therefore, based on this, step 320 can be used to maximize selected expert networks’ logit values corresponding to T tokens with constraints on of the number of selected expert networks. The first preset threshold is greater than or equal to 0, and the value of the first preset threshold is not limited in the embodiments of this application. For example, the first preset threshold is 0.1.

[0075] For example, FIG. 4 is a schematic diagram of interchangeability of expert networks according to an embodiment of this application. FIG. 4 shows a heatmap of expert network’s substitutability in a pre-trained Mixtral-8x7B model. In FIG. 4, a horizontal axis represents the number of tokens, a vertical axis represents identities of expert networks, and darker shades in thisheatmap correspond to the number of expert networks able to process a token with a maximum logit value variation of 0.1 or less instead of selected ones. By analyzing the calibration dataset of 80000 tokens, it was found that in 70.5% of cases expert networks can be interchanged. Almost every token can be processed by other expert networks with a logit value change of 0.1 or less.

[0076] Step 320, obtaining K selected expert networks corresponding to each token in the T tokens based on the logit value of each expert network in the first MoE layer corresponding to each token in T tokens.

[0077] In some embodiments, the computing device determines K selected expert networks for each token from Ni expert networks based on the logit value of each expert network in the first MoE layer corresponding to each token in T tokens. K is a positive integer less than Ni. K is a preset value, and the specific value of K is not limited in the embodiments of this application.

[0078] In some embodiments, the K selected expert networks corresponding to each token in the T tokens are such that a value of a first loss function is maximized, and the value of the first loss function is determined based on a first penalty value and a logit value of each selected expert network in the K selected expert networks corresponding to each token in the T tokens. The first penalty value is determined based on the number of selected expert networks, and / or, the first penalty value is determined based on a logit value of each initially activated expert network in K initially activated expert networks corresponding to each token in the T tokens and the logit value of each expert network in the K selected expert networks corresponding to each token in the T tokens. The logit value of each initially activated expert network in the K initially activated expert networks corresponding to each token is greater than or equal to the logit value of each expert network in the Ni expert networks corresponding to each token other than the K initially activated expert networks. That is, the logit values of the K initially activated expert networks corresponding to each token is top K among the logit values of the Ni expert networks corresponding to each token.

[0079] In some embodiments, the value of the first loss function includes a difference between the first penalty value and a sum of logit values of the K selected expert networks corresponding to each token in the T tokens. For example, the first loss function Fi is determined according to following formulas:

[0080] Fi = ZU Et=o SXo’1~ P (5)

[0081] Xtki £ {0,1} (6)

[0082] ∀t ∈ {0,1,..., T - 1}, i ∈ {0,1,..., N - 1} max(∑K-1k=0Xkt,i) ≤ 1 (7)

[0083] ∀t ∈ {0,1,..., T - 1}, k ∈ {0,1,..., K - 1} ∑N-1i=0Xkt,i= 1 (8)

[0084] ∀t ∈ {0,1,..., T - 1}, k ∈ {0,1,..., K - 2} ∑N-1i=0Lt,i· Xk+1t,i≤ ∑N-1i=0Lt,i· Xkt,i(9)

[0085] Among them, Lt iis described in formula (4). Xktrepresents whether the ithexpert network in the Ni expert networks is the kthselected expert network in the K selected expert networks corresponding to the t* token. t=0,..., T-l. i=0, Ni-1. k=0, K-l. Xktsatisfies formulas (6), (7), and (8). The formula (6) represents that a value of Xkt,iis 0 or 1. When the ithexpert network is the kthselected expert network corresponding to the tthtoken, Xkt= 1. When the ithexpert network is not the kthselected expert network corresponding to the tthtoken, Xkt= 0. The formula (7) represents that for the tthtoken in the T tokens and the ithexpert network in the Ni expert networks, when the values of t and i are fixed, the ithexpert network belongs to the K selected expert networks corresponding to the tthtoken, or the ithexpert network does not belong to the K selected expert networks corresponding to the tthtoken. The formula (8) represents that for the tthtoken in the T tokens and the kthselected expert network in the K selected expert networks, when the values of t and k are fixed, the kthselected expert network corresponding to the tthtoken belongs to the Ni expert networks. Lt iand Xktsatisfy the formula (9). The formula (9) represents that for the tthtoken in the T tokens and the kthselected expert network in the K selected expert networks, when the values of t and k are fixed, a logit value of the (k+l)thselected expert network corresponding to the tthtoken is less than or equal to a logit value of the kthselected expert network. In formula (5), P represents the first penalty value.

[0086] In some embodiments, the computing device selects K expert networks for each token as K selected expert networks to determine the value of Fi. The computing device determines the maximum value of Fi from multiple values of Fi. The computing device determines K selected expert networks corresponding to each token when the value of Fi is maximized.

[0087] In some embodiments, when the first penalty value is determined based on the number of selected expert networks, the first penalty value is a first value. Alternatively, when the first penalty value is determined based on the logit value of each initially activated expert network in the K initially activated expert networks corresponding to each token and the logit value of each expert network in the K selected expert networks corresponding to each token in the T tokens, the first penalty value is a second value. Alternatively, when the first penalty value is determined based on the number of selected expert networks, the logit value of each initially activated expert network in the K initially activated expert networks corresponding to each token and the logit value of each expert network in the K selected expert networks corresponding to each token in the T tokens, the first penalty value is a sum of a first value and a second value. Among them, the first value is positively correlated with the number of selected expert networks. In other words, the fewer the number of selected expert networks, the smaller the first penalty value. The more the number of selected expert networks, the larger the first penalty value. The second value is positively correlated with a difference between the logit value of each initially activated expert network in the K initially activated expert networks corresponding to each token and the logit value of each expert network in the K selected expert networks corresponding to each token in the T tokens. In other words, the smaller the difference between the logit value of the kthinitially activated expert network corresponding to each token and the logit value of the kthselected expert network, the smaller the first penalty value. The larger the difference between the logit value of the kthinitially activated expert network corresponding to each token and the logit value of the kthselected expert network, the larger the first penalty value.

[0088] In some embodiments, the first value includes a product of a first penalty factor and the number of selected expert networks, and the first penalty factor is a preset value. Alternatively, the first value includes a dot product of a second penalty factor and a first vector, the second penalty factor includes an N1-dimensional vector, the first vector includes an N1-dimensional binary vector, and the first vector is used to represent the number of selected expert networks.

[0089] For example, the computing device removes at least one duplication on the K selected expert networks corresponding to each token in T tokens, and determines N3 selected expert networks. N3 is a positive integer less than Ni. That is, the N3 selected expert networks include the K selected expert networks corresponding to each token in T tokens, and the N3 selected expertnetworks do not include the same expert network. That is, the number of selected expert networks is N3. The N3 selected expert networks are also referred to as N3 retained expert networks.

[0090] For example, when the first value includes the product of the first penalty factor and the number of selected expert networks, the first value P1= p1N3. Among them, p1represents the first penalty factor.

[0091] For example, when the first value includes the dot product of the second penalty factor and the first vector, the first value P1is determined according to following formula:

[0092] P1= p2maxt∑K-1k=0Xk(10)

[0093] Among them, Xkis a two-dimensional matrix of size T x Nl, and the ithelement in the tthrow of Xkrepresents whether the ithexpert network in the Ni expert networks is the kthselected expert network in the K selected expert networks corresponding to the t* token in the T tokens. For example, when the ithelement in the tthrow of Xkis ai, the ithexpert network is the kthselected expert network corresponding to the tthtoken. When the ithelement in the tthrow of Xkis a2, the ithexpert network is not the kthselected expert network corresponding to the tthtoken.represents whether the ithexpert network belongs to the K selected expert networks corresponding to the tthtoken. For example, when the ithelement in the tthrow of ∑K-1k=0Xkis a1, the ithexpert network belongs to the K selected expert networks corresponding to the tthtoken. When the ithelement in the tthrow of ∑K-1k=0Xkis a2, the ithexpert network does not belong to the K selected expert networks corresponding to the tthtoken. maxt∑K-1k=0Xkis an N1-dimensional row vector or column vector. When maxt∑K-1k=0Xkis a row vector, the ithelement in the first row of maxt∑K-1k=0Xkrepresents whether the ithexpert network belongs to the K selected expert networks corresponding to each token in the T tokens. When maxt∑K-1k=0Xkis a column vector, the first element in the ithrow of maxt∑K-1k=0Xkrepresents whether the ithexpert network belongs to the K selected expert networks corresponding to each token in the T tokens. For example, when the ithelement of maxt∑K-1k=0Xk= a1, the ithexpert network belongs to the K selected expert networks corresponding to each token in the T tokens. When the ithelement of maxt∑K-1k=0Xk= a2, the ithexpert network does not belong to the K selected expert networks corresponding to eachtoken in the T tokens. p2is the second penalty factor, and p2is a preset vector. p2is an Ni-dimensional vector. p2maxt∑K-1k=0Xkrepresents the dot product of p2and maxt∑K-1k=0Xk. a1anda2are preset values, a1≠a2. For example, a1= 1, a2= 0.

[0094] For example, assuming T=3, Ni=8, K=2, X°, X1and p2are shown in formula (11) to (13), and according to formula (10), following formulas (14) to (16) are obtained:0 1 0 0 0 0 O’

[0095] X° = 0 1 0 0 0 0 0 0 (11).0 0 0 0 0 1 0 0.0 0 0 0 0 1 0 O’

[0096] 0 0 0 0 0 0 1 0 (12).1 0 0 0 0 0 0 0.

[0097] P2 = [&1 b2^3 b4b5b6b7b8] (13)0 1 0 0 0 1 0 O’

[0098] ∑2k=0Xk= 0 1 0 0 0 0 1 0 (14).1 0 0 0 0 1 0 0.

[0099] maxt∑2k=0Xk= [1 1 0 0 0 1 1 0] (15)

[0100] P1= p2maxt∑2k=0Xk

[0101] = [b1b2b3b4b5b6b7b8][1 1 0 0 0 1 1 0]

[0102] = 1 × b1+ 1 × b2+ 0 × b3+ 0 × b4+ 0 × b5+ 1 × b6+ 1 × b7+ 0 × b8

[0103] = b1+ b2+ b6+ b7(16)

[0104] In some embodiments, the second value includes a sum of differences between logit values of the K initially activated expert networks corresponding to each token and logit values of the K selected expert networks corresponding to each token in the T tokens. For example, the second value P2is determined according to following formula:

[0105] = (I’)

[0106] Among them, Lt iand Xktare described above. topk(Lt) represents the logit value of the kthinitially activated expert network in the K initially activated expert networks corresponding to the tthtoken in the T tokens. The logit value of each initially activated expert network in the K initially activated expert networks corresponding to each token is greater than or equal to the logit values of the Ni expert networks corresponding to each token other than the K initially activated expert networks.topfc(Lt) -1represents the difference betweenthe logit value of the kthinitially activated expert network corresponding to the tthtoken and the logit value of the kthselected expert network corresponding to the tthtoken in the T tokens.

[0107] In some embodiments, the first penalty value includes the sum of the first value and the second value, the first value is determined based on the formula (10), the second value is determined based on the formula (17), the first loss function Fi is determined according to formula (18), and the K selected expert networks corresponding to each token in T tokens are determined according to formula (19):

[0108] F, = ZK S& O1S£o1Mu ~ P2 max Sjfzj X“ - SEfttopM - VM) (18)

[0109] MAX (∑K-1k=0∑T-1t=0∑N-1i=0Lt,iXkt,i− p2maxt∑K-1k=0Xk− ∑K-1k=0∑T-1t=0(topk(Lt) −(19)

[0110] Among them, MAX ) represents a function of taking a maximum value. According to formulas (18) and (19), it can be seen that in order to maximize the value of Fi, the sum of the logit values of the K selected expert networks corresponding to each token in the T tokens should be as large as possible, but the number of selected expert networks should be as small as possible, so that the first value is small, and the difference between the kthinitially activated expert network corresponding to each token and the kthselected expert network corresponding to each token should be as small as possible, so that the second value is small. In other words, by performing element-wise multiplication of Lt,iand Xkt,i, the logit values of the activated expert networks across all tokens are obtained. To identify expert networks with higher average importance, the value of Fi is maximized. Additionally, formulas (18) and (19) include two penalty terms: the first value is used to reduce the number of selected expert networks, and the second value is used to encourage the use of initially activated expert networks across all tokens, leveraging the interchangeability property. That is, formulas (18) and (19) achieve a balance between retaining critical expert networks and minimizing the total number of expert networks.

[0111] For example, a schematic diagram of the K selected expert networks corresponding to the token is shown in FIG. 5 and FIG. 6. In FIG. 5, a token 510 is input into a gating network 520, and the gating network 520 belongs to the first MoE layer. The first MoE layer includes Ni expertnetworks, i.e. Ni=8. The gating network 520 outputs logit values of 8 expert networks corresponding to the token 510, which include a logit value 521 to a logit value 528. A logit value of the first expert network among the 8 expert networks is a logit value 521. A logit value of the second expert network among the 8 expert networks is a logit value 522. A logit value of the third expert network among the 8 expert networks is a logit value 523. A logit value of the fourth expert network among the 8 expert networks is a logit value 524. A logit value of the fifth expert network among the 8 expert networks is a logit value 525. A logit value of the sixth expert network among the 8 expert networks is a logit value 526. A logit value of the seventh expert network among the 8 expert networks is a logit value 527. A logit value of the eighth expert network among the 8 expert networks is a logit value 528. A length of a box corresponding to each logit value in FIG. 5 represents a size of each logit value. From FIG. 5, it can be seen that the logit values 528, 527, 525, and 522 are top 4 in 8 logit values shown in FIG. 5.

[0112] For example, as shown in FIG. 5, when K=4, expert networks corresponding to the logit value 522, 526, 527 and 528 belong to four selected expert networks corresponding to the token 510. Among them, the logit values 528, 527, and 522 are relatively large, and average logit values of expert networks corresponding to the logit values 528, 527 and 522 are relatively large. An average logit value of an expert network is a ratio of a sum of logit values of expert networks corresponding to each token in T tokens to T. The logit value 526 is relatively small, but an average logit value of an expert network corresponding to the logit value 526 is relatively large. Expert networks corresponding to the logit value 521, 523, 524, and 525 do not belong to the four selected expert networks corresponding to the token 510. Among them, the logit values 521, 523, and 524 are relatively small, and average logit values of expert networks corresponding to the logit values 521, 523 and 524 are relatively small. The logit value 525 is relatively large, but an average logit value of an expert network corresponding to the logit value 525 is relatively small. From FIG. 5, it can be seen that the K selected expert networks corresponding to each token determined in step 320 include 2 expert networks with larger logit values in the expert networks corresponding to each token. Therefore, pruning the first MoE layer based on the K selected expert networks corresponding to each token reduces the cost of resources required for deploying the adjusted target MoE model and the impact on the performance of the adjusted target MoE model.

[0113] For example, In FIG. 6, a token 610 is input into a gating network 520, and the gatingnetwork 520 is described in FIG. 5. The first MoE layer includes N1expert networks, i.e. N1=8. The gating network 520 outputs logit values of 8 expert networks corresponding to the token 610, which include a logit value 621 to a logit value 628. The logit value of the first expert network among the 8 expert networks is a logit value 621. The logit value of the second expert network among the 8 expert networks is a logit value 622. The logit value of the third expert network among the 8 expert networks is a logit value 623. The logit value of the fourth expert network among the 8 expert networks is a logit value 624. The logit value of the fifth expert network among the 8 expert networks is a logit value 625. The logit value of the sixth expert network among the 8 expert networks is a logit value 626. The logit value of the seventh expert network among the 8 expert networks is a logit value 627. The logit value of the eighth expert network among the 8 expert networks is a logit value 628. A length of a box corresponding to each logit value in FIG. 6 represents a size of each logit value. From FIG. 6, it can be seen that the logit values 627, 625, 628, and 624 are top 4 in 8 logit values shown in FIG. 6.

[0114] For example, as shown in FIG. 6, when K=4, expert networks corresponding to the logit value 622, 626, 627 and 628 belong to four selected expert networks corresponding to the token 610. Among them, the logit values 628 and 627 are relatively large, and average logit values of expert networks corresponding to the logit values 628 and 627 are relatively large. The logit values 622 and 626 are relatively small, but average logit values of expert networks corresponding to the logit values 622 and 626 are relatively large. Expert networks corresponding to the logit values 621, 623, 624, and 625 do not belong to the four selected expert networks corresponding to the token 610. Among them, the logit values 621 and 623 are relatively small, and average logit values of expert networks corresponding to the logit values 621 and 623 are relatively small. The logit value 624 is relatively large, but an average logit value of an expert network corresponding to the logit value 624 is relatively small. The logit value 625 is relatively large, but an expert network corresponding to the logit value 625 can be replaced by an expert network corresponding to the logit value 628. For example, a difference between the logit value 628 and the logit value 625 is less than or equal to the first preset threshold. From FIG. 6, it can be seen that the K selected expert networks corresponding to each token determined in step 320 include expert networks with larger logit values in the expert networks corresponding to each token. Therefore, pruning the first MoE layer based on the K selected expert networks corresponding to each token reduces the costof resources required for deploying the adjusted target MoE model and the impact on the performance of the adjusted target MoE model.

[0115] For example, when the expert network corresponding to the logit value 625 is replaced by the expert network corresponding to the logit value 628, a weight of the expert network corresponding to the logit value 625 and a weight of the expert network corresponding to the logit value 628 are merged. The specific implementation method is shown in FIG. 7.

[0116] Step 330, obtaining an adjusted first MoE layer by pruning at least one expert network in the first MoE layer based on the K selected expert networks corresponding to each token in the T tokens.

[0117] The computing device removes at least one expert network of the N1expert network based on the N3selected expert networks. N3is a positive integer less than N1. The N3selected expert networks include the K selected expert networks corresponding to each token in T tokens, and the N3 selected expert networks do not include the same expert network. The computing device retains the N3 selected expert networks in the first MoE layer and prunes at least one expert network in the first MoE layer other than the N3 selected expert networks to obtain the adjusted first MoE layer. The adjusted first MoE layer includes the N3 selected expert networks.

[0118] Step 340, obtaining an adjusted target MoE model based on the adjusted first MoE layer.

[0119] The computing device replaces the first MoE layer in the target MoE model with the adjusted first MoE layer to obtain the adjusted target MoE model.

[0120] In some embodiments, the target MoE model includes a second MoE layer including N2 expert networks, and N2 is a positive integer greater than 1. The second MoE layer is different from the first MoE layer. The computing device utilizes steps 310-330 to determine N4 selected expert networks in the second MoE layer, and prunes the second MoE layer to obtain an adjusted second MoE layer. The adjusted second MoE layer includes N4 selected expert networks, and N4< N2.

[0121] For example, the number of expert networks in the adjusted second MoE layer is different from the number of expert networks in the adjusted first MoE layer. Due to the different number of expert networks in at least two MoE layers in the adjusted target MoE model, each layer has a different compression ratio, thereby improving the performance of the compressed model.

[0122] In some embodiments, the computing device replaces the second MoE layer in thetarget MoE model with the adjusted second MoE layer to obtain the adjusted target MoE model.

[0123] In the method in FIG. 3, the K selected expert networks corresponding to each token in the T tokens are determined by utilizing the logit value of each expert network in the first MoE layer corresponding to each token in T tokens, thereby pruning the first MoE layer and obtaining the adjusted target MoE model. Due to the fact that the number of expert networks in the adjusted first MoE layer is less than the number of expert networks in the first MoE layer, a cost of resources required for deploying the adjusted target MoE model is reduced. Moreover, through the first loss function, the logit values of the selected expert networks can be maximized to avoid affecting the performance of the adjusted target MoE model, and the number of expert networks in the adjusted MoE layer is controlled to further reduce the cost of resources required for deploying the adjusted target MoE model. That is, this method achieves a balance between retaining important expert networks and minimizing the number of expert networks.

[0124] For example, when the target MoE model is a Mixtal-8x7B-Instruct model, data obtained by pruning the target MoE model using different methods are shown in Table 1:

[0125] Table 1number ofnumber of method compression retained expert accuracy droptokens networksmethod 1 1.31 6 13.97% 2048 method 2 1.31 6 1.02% 262144 method 3 1.31 2-8 0.75% 2048 method 1 1.93 4 17.76% 2048 method 2 1.93 4 3.07% 262144method 3 1.92 2-8 2.90% 2048

[0126] Among them, the method 1 is a method based on frequency. According to the method 1, an adjusted target MoE model is obtained by retaining most frequently activated expert networks in a first MoE layer of a target MoE model and pruning other expert networks in the first MoE layer of the target MoE model. According to the method 2, an adjusted target MoE model is obtained by retaining r expert networks in each MoE layer of the target MoE model and pruning other expert networks in each MoE layer of the target MoE model, and the r expert networks in the first MoE layer minimizes a difference between an output of the first MoE layer and an output of an adjusted first MoE layer. The method 3 is the method provided in the embodiments of this application. The compression in table 1 is a ratio of an amount of resources required to store theadjusted target MoE model to an amount of resources required to store the target model. The number of retained expert networks in table 1 represents the number of expert networks included in each adjusted MoE layer of the adjusted target MoE model. The accuracy drop in table 1 is a ratio of a difference between a performance of the target MoE model and a performance of the adjusting target MoE model to the performance of the target MoE model. The calculation method for the performance of the model is a zero-shot performance evaluation on Winogrande dataset. The number of tokens in table 1 represents the number of tokens required for pruning a model using different methods. From Table 1, it can be seen that under similar compression ratios, the number of expert networks in each layer of the adjusted target MoE model obtained according to the method in the embodiments of the present application can be the same or different. The adjusted target MoE model obtained according to the method in the embodiments of the present application has a minimum of 2 expert networks retained and achieves the lowest metric degradation in the accuracy drop.

[0127] In some embodiments, a specific implementation of step 330 is shown in FIG. 7. FIG.7 is a schematic diagram of a method of adjusting a model according to another embodiment of this application. The method in FIG. 7 is applied by a computing device. The method in FIG. 7 includes the following steps.

[0128] Step 710, obtaining N3retained expert networks in the first MoE layer based on the K selected expert networks corresponding to each token in the T tokens.

[0129] The computing device determines the K selected expert networks corresponding to each token in T tokens according to step 320 in FIG. 3, and removes at least one duplication on the K selected expert networks corresponding to each token in T tokens to obtain the N3retained expert networks in the first MoE layer. The N3retained expert networks include the K selected expert networks corresponding to each token in the T tokens, and the N3retained expert networks do not include the same expert network. N3is a positive integer less than N1.

[0130] Step 720, obtaining at least one set of pruned expert networks corresponding to at least one of the N3retained expert networks.

[0131] The computing device obtains at least one set of pruned expert networks corresponding to at least one of the N3retained expert networks, and the set of pruned expert networks includes at least one pruned expert network belonging to expert networks of the first MoE layer other thanthe N3retained expert networks.

[0132] For example, the N3retained expert networks include a first retained expert network, the expert networks in the first MoE layer other than the N3retained expert networks include a first pruned expert network, and the first pruned expert network belongs to a set of pruned expert networks corresponding to the first retained expert network.

[0133] In some embodiments, a difference between a first logit value of the first retained expert network and a first logit value of the first pruned expert network is less than or equal to the first preset threshold. For example, the first logit value of the first retained expert network is less than or equal to a sum of the first logit value of the first pruned expert network and the first preset threshold, alternatively, the first logit value of the first pruned expert network is less than or equal to a sum of the first logit value of the first retained expert network and the first preset threshold. The first logit value of the first retained expert network is determined based on the logit value of the first retained expert network corresponding to at least one token in the T tokens, and the first logit value of the first pruned expert network is determined based on the logit value of the first pruned expert network corresponding to the at least one token in the T tokens. The first preset threshold is greater than or equal to 0, and the value of the first preset threshold is not limited in the embodiments of this application. For example, the first preset threshold is 0.1.

[0134] For example, the first logit value of the first retained expert network is the logit value of the first retained expert network corresponding to a first token, the first logit value of the first pruned expert network is the logit value of the first pruned expert network corresponding to the first token, and the first token belongs to the T tokens. Alternatively, the first logit value of the first retained expert network is an average logit value of the first retained expert network, and the first logit value of the first pruned expert network is an average logit value of the first pruned expert network. The average logit value of the first retained expert network includes a ratio of a sum of logit values of the first retained expert network corresponding to each token in the T tokens to T, and the average logit value of the first pruned expert network includes a ratio of a sum of logit values of the first pruned expert network corresponding to each token in the T tokens to T.

[0135] In some embodiments, a difference between a first logit value of the first retained expert network and a first logit value of each pruned expert network in the set of pruned expert networks corresponding to the first retained expert network is less than or equal to the first preset threshold.The first logit value of each pruned expert network is similar to the first logit value of the first pruned expert network.

[0136] Step 730, updating a weight of each retained expert network in the at least one retained expert network based on the weight of each retained expert network and at least one weight of at least one pruned expert network corresponding to each retained expert network.

[0137] In some embodiments, the computing device updates a weight of the first retained expert network based on a weight of each pruned expert network in the set of pruned expert networks corresponding to the first retained expert network and the weight of the first retained expert network. A weight of the updated first retained expert network is a weighted sum of the weight of the first retained expert network and the weight of each pruned expert network corresponding to the first retained expert network. For example, a weight W- of the updated first retained expert network is determined according to following formula:

[0138] Wj′ = α0Wj+ ∑m=1MαmWm(20)

[0139] Among them, Wjrepresents a weight matrix in the first retained expert network. The set of pruned expert networks corresponding to the first retained expert network includes M pruned expert networks, and M is a positive integer.represents a weight matrix of the mthpruned expert network in the M pruned expert networks corresponding to the first retained expert network. α0represents a weighting coefficient of Wj, and αmrepresents a weighting coefficient ofα0+ ∑m=1Mαm= 1. The specific values of a0and amin the embodiments of this application are not limited.

[0140] For example, when only the first pruned expert network is included in the set of pruned expert networks corresponding to the first retained expert network, the computing device updates the weight of the first retained expert network based on its weight matrix Wjand a weight of the first pruned expert network W1. The weight matrix Wj′ of the updated first retained expert network is determined according to following formula: Wj′ = α0Wj+ α1W1, α0+ α1= 1.

[0141] Step 740, obtaining the adjusted first MoE layer by pruning at least one expert network in the first MoE layer.

[0142] The computing device obtains the adjusted first MoE layer by pruning at least oneexpert network in the first MoE layer other than the N3retained expert networks. The adjusted first MoE layer includes the N3retained expert networks. At least one of the N3 retained expert networks has a weight matrix updated according to steps 720-730.

[0143] In some embodiments, after step 740, the computing device continues to execute step 340.

[0144] In some embodiments, the computing device prunes at least one MoE layer in the target MoE model and updates weight matrices of some or all of the expert networks in each adjusted MoE layer to obtain the adjusted target MoE model.

[0145] In the method in FIG. 7, the computing device saves knowledge of the pruned expert network by merging the weight matrix of the pruned expert network with the weight matrix of the retained expert network, thereby improving the performance of the adjusted target MoE model.

[0146] In some embodiments, a method of adjusting a target MoE model is shown in FIG. 8. FIG. 8 is a schematic diagram of a method of adjusting a model according to another embodiment of this application. The method in FIG. 8 is applied by a computing device. The method in FIG. 8 includes the following steps.

[0147] Step 810, determining the qthMoE layer in the target MoE model.

[0148] The computing device obtains the target MoE model, and the target MoE model is described in FIG.3 and FIG.7. The target MoE model includes Q MoE layers, where Q is a positive integer. Each of the Q MoE layers includes a gating network and multiple expert networks. The gating network and expert networks are described in FIG. 1 to FIG. 7. The computing device determines the qthMoE layer from the Q MoE layers, q=1,..., Q.

[0149] Step 820, obtaining at least one retained expert network in the qthMoE layer. The specific implementation method is described in steps 310 and 320 in FIG. 2, and will not be repeated here.

[0150] Step 830, updating at least one weight of at least one retained expert network in the qthMoE layer. The specific implementation method is described in steps 720 and 730 in FIG. 7, which will not be repeated here.

[0151] Step 840, obtaining an adjusted q111MoE layer by pruning at least one expert network in the qthMoE layer. The specific implementation method is described in step 740 of FIG. 7, which will not be repeated here.

[0152] Step 850, determining whether q is less than Q.

[0153] The computing device determines whether q is less than Q. When q is less than Q, the computing device executes step 860. When q is greater than or equal to Q, the computing device executes step 870.

[0154] Step 860, updating q.

[0155] The computing device updates q, and the updated q is a sum of q before the update and 1. After updating q, the computing device repeats steps 810-850 based on the updated q until q equals Q.

[0156] Step 870, obtaining an adjusted target MoE model.

[0157] The computing device prunes each MoE layer in the target MoE model, and merges the weights of the pruned expert network and the retained expert network in each layer to obtain the adjusted target MoE model. The adjusted target MoE model includes Q adjusted MoE layers. Expert networks in at least two of the adjusted MoE layers are the same or different in number.

[0158] In some embodiments, the number of expert networks included in one or more of the Q adjusted MoE layers is the same as the number of expert networks included in this MoE layer before pruning.

[0159] In the method of FIG. 8, the computing device prunes each MoE layer in the target MoE model and retains the knowledge of the pruned expert network, thereby reducing the cost of resources required for deploying the adjusted target MoE model and avoiding affecting the performance of the adjusted target MoE model, thereby achieving lossless (< 0.5%) metric drop with a high compression ratio.

[0160] For example, when the target MoE model is a Mixtal-8x7B-Instruct model, data obtained by pruning the target MoE model using the method in FIG. 8 are shown in Table 2:

[0161] Table 2number ofnumber of method compression retained expert accuracy droptokens networksmethod 3 1.30 2-8 0.29 2048 method 3 1.51 2-8 0.31 2048method 3 1.96 2-8 2.60 2048

[0162] Among them, the compression, the number of retained expert networks, the accuracy drop, and the number of tokens in table 2 are the same as those in table 1, and will not be repeatedhere. From tables 1 and 2, it can be seen that using the method in FIG. 8 results in less performance degradation of the adjusted target MoE model at similar compression ratios. For example, at compression ratios of 1.3 and 1.51, the method in FIG. 8 achieves almost lossless (<0.5%) metric drop. From tables 1 and 2, it can also be seen that the method in FIG. 8 achieves better performance when the number of tokens used is relatively small.

[0163] FIG. 9 is a schematic diagram of a device 900 for adjusting a model according to an embodiment of this application. As shown in FIG. 9, the device 900 for adjusting a model includes: an obtaining unit 910 and a processing unit 920. The device 900 for adjusting a model implements the method as shown in FIG. 3, FIG. 7 and FIG. 8. The device 900 for adjusting a model is applied to a computing device.

[0164] The obtaining unit 910 is configured to obtain a logit value of each expert network in a first MoE layer corresponding to each token in T tokens based on the T tokens and a target MoE model obtain a project code. The obtaining unit 910 performs the step 310 in FIG. 3.

[0165] The processing unit 920 is configured to: obtain K selected expert networks corresponding to each token in the T tokens based on the logit value of each expert network in the first MoE layer corresponding to each token in T tokens; obtain an adjusted first MoE layer by pruning at least one expert network in the first MoE layer based on the K selected expert networks corresponding to each token in the T tokens; obtain an adjusted target MoE model based on the adjusted first MoE layer. The processing unit 920 performs the steps 320-340 in FIG. 3.

[0166] In some embodiments, the processing unit 920 performs the steps 710-740 in FIG. 7.

[0167] In some embodiments, the processing unit 920 performs the steps 810-870 in FIG. 8.

[0168] Among them, both the obtaining unit 910 and the processing unit 920 are implemented through software or hardware. The processing unit 920 is taken as an example to introduce the implementation of the processing unit 920. Similarly, the implementation of the obtaining unit 910 refers to the implementation of the processing unit 920.

[0169] When the unit is an example of a software functional unit, the processing unit 920 includes code running on a computational instance. Among them, the computational instance includes at least one of physical hosts (computing devices), virtual machines, and containers. Furthermore, the above computational instance can be one or more. For example, the processing unit 920 includes code running on multiple hosts / virtual machines / containers. It should be notedthat multiple hosts / virtual machines / containers used to run the code are distributed in the same region or in different regions. Furthermore, multiple hosts / virtual machines / containers used to run the code are distributed within the same availability zone (AZ) or across different AZs, each of which includes a data center or multiple geographically close data centers. Typically, a region includes multiple AZs.

[0170] Similarly, multiple hosts / virtual machines / containers used to run the code are distributed within the same virtual private cloud (VPC) or across multiple VPCs. Among them, usually one VPC is set within a region, and cross regional communication between two VPCs within the same region, as well as VPCs from different regions, requires a communication gateway to be set up within each VPC to achieve interconnection between VPCs.

[0171] When the unit is an example of a hardware functional unit, the processing unit 920 includes at least one computing device, such as a server. Alternatively, the processing unit 920 also is a device implemented using application specific integrated circuits (ASIC) or programmable logic devices (PLD). Among them, the above-mentioned PLD is a complex programmable logic device (CPLD), field programmable gate array (FPGA), general array logic (GAL), or any combination thereof.

[0172] The multiple computing devices included in the processing unit 920 are distributed in the same region or in different regions. The multiple computing devices included in the processing unit 920 are distributed within the same AZ or across different AZs. Similarly, the multiple computing devices included in the processing unit 920 are distributed within the same VPC or across multiple VPCs. Among them, the multiple computing devices are any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0173] Therefore, the units of each example described in the embodiments of the present application are implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians may use different methods to achieve the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0174] It should be noted that the device provided in the above embodiments only provides examples of the division of various functional units when executing the above methods. Inpractical applications, the above functions can be assigned to different functional units according to needs, that is, the internal structure of the device can be divided into different functional units to complete all or part of the functions described above. For example, the obtaining unit 910 can be used to perform any step in the above method, and the processing unit 920 can be used to perform any step in the above method. The steps responsible for implementing the obtaining unit 910 and the processing unit 920 can be specified as needed. The obtaining unit 910 and the processing unit 920 respectively implement different steps in the above methods to achieve all the functions of the above device.

[0175] In addition, the device and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments mentioned above, which will not be repeated here.

[0176] The method provided in the embodiments of the present application may be executed by a computing device, which may also be referred to as a computer system. This includes a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on top of the operating system layer. This hardware layer includes hardware such as processing units, memory, and memory control units, followed by a detailed explanation of the hardware’s functions and structure. This operating system is any one or more computer operating systems that implement business processing through processes, such as Linux operating system, Unix operating system, Android operating system, iOS operating system, or Windows operating system. This application layer includes applications such as browsers, contacts, word processing software, instant messaging software, etc. And, alternatively, the computer system can be a handheld device such as a smartphone, or a terminal device such as a personal computer, which is not specifically limited by the present application, as long as it can be implemented through the methods provided in the embodiments of the present application. The execution subject of the method provided in the embodiments of the present application can be a computing device, or a functional module in the computing device that can call and execute the program.

[0177] FIG. 10 is a schematic block diagram of a computing device according to an embodiment of this application. The computing device 1000 can be a server, a computer, or other device with computing power. The computing device 1000 shown in FIG. 10 includes at least one processor 1010 and a memory 1020.

[0178] It should be understood that embodiments of the present application do not limit the number of processors and memory in the computing device 1000.

[0179] The processor 1010 executes instructions in the memory 1020 to enable the computing device 1000 to implement the method provided in the embodiments of the present application. Alternatively, the processor 1010 executes instructions in the memory 1020 to enable the computing device 1000 to implement the various functional modules provided in the embodiments of the present application, thereby implementing the methods provided in the embodiments of the present application.

[0180] Optionally, the computing device 1000 also includes a communication interface 1030. The communication interface 1030 uses transceiver modules such as but not limited to network interface cards and transceivers to achieve communication between the computing device 1000 and other devices or communication networks.

[0181] Optionally, the computing device 1000 also includes a system bus 1040, where the processor 1010, the memory 1020, and the communication interface 1030 are respectively connected to the system bus 1040. The processor 1010 can access the memory 1020 through the system bus 1040, for example, the processor 1010 can read and write data or execute code in the memory 1020 through the system bus 1040. The system bus 1040 is either a peripheral component interconnect express (PCI) bus or an extended industry standard architecture (EISA) bus. The system bus 1040 is divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in FIG. 10, but it does not mean that there is only one bus or one type of bus.

[0182] One possible implementation is that the function of the processor 1010 is mainly to interpret instructions (or code) of computer programs and process data in computer software. Among them, the instructions of the computer program and the data in the computer software can be stored in the cache of the memory 1020 or the processor 1010.

[0183] Optionally, the processor 1010 may be an integrated circuit chip with signal processing capabilities. As an example rather than a limitation, the processor 1010 is a general-purpose processor, digital signal processor (DSP), ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. Among them, general-purpose processors are microprocessors, etc. For example, the processor 1010 is a centralprocessing unit (CPU).

[0184] The memory 1020 can provide running space for processes in the computing device 1000, for example, storing computer programs (specifically, program code) used to generate processes in the memory 1020. After the computer program is run by the processor and generates a process, the processor allocates corresponding storage space for the process in the memory 1020. Furthermore, the above storage space further includes text segments, initialization data segments, bit initialization data segments, stack segments, heap segments, and so on. The memory 1020 stores data generated during the operation of the process, such as intermediate data or process data, in the storage space corresponding to the above process.

[0185] Optionally, the memory is used to temporarily store operational data in the processor 1010 and data exchanged with external memory such as hard drives. As long as the computer is running, the processor 1010 will transfer the data that needs to be processed to memory for processing, and then transmit the result after the operation is completed.

[0186] As an example rather than a limitation, the memory 1020 may be either volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Among them, non-volatile memory is read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), flash, or electrically EPROM (EEPROM). Volatile memory is a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus DRAM (DRDRAM). It should be noted that the memory 1020 of the system and method described in this application is intended to include but not limited to these and any other suitable types of memory.

[0187] The structure of the computing device 1000 listed above is only for illustrative purposes, and this application is not limited to it. The computing device 1000 in the embodiments of this application includes various hardware in computer systems in prior art. For example, the computing device 1000 also includes other storage devices besides memory 1020, such as disk storage, etc. Technicians in this field should understand that the computing device 1000 may also include other devices necessary for normal operation. Meanwhile, according to specific needs, technical personnel in this field should understand that the above-mentioned computing device1000 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the above-mentioned computing device 1000 may only include the devices necessary to implement the embodiments of the present application, without necessarily including all the devices shown in FIG. 10.

[0188] The embodiment of this application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server. In some embodiments, the computing device may also be a terminal device such as a desktop computer, laptop, or smartphone.

[0189] As shown in FIG. 11, the computing device cluster includes at least one computing device 1000. The memory 1020 in one or more computing devices 1000 in a computing device cluster may contain the same instructions for executing the above method.

[0190] In some possible implementations, the memory 1020 in one or more computing devices 1000 within the computing device cluster may also hold partial instructions for executing the above method. In other words, a combination of one or more computing devices 1000 can jointly execute the instructions of the above method.

[0191] It should be noted that the memory 1020 in different computing devices 1000 in the computing device cluster can store different instructions, which are used to perform some of the functions of the above-mentioned devices. That is to say, the instructions stored in the memory 1020 of different computing devices 1000 can realize the function of one or more modules in the above-mentioned device.

[0192] In some possible implementations, one or more computing devices in a computing device cluster can be connected through a network. Among them, the network can be a wide area network, a local area network, or the like. FIG. 12 illustrates a possible implementation approach. As shown in FIG. 12, two computing devices 1000A and 1000B are connected through a network. Specifically, the computing devices 1000A and 1000B are connected to the network through their communication interfaces.

[0193] It should be understood that the functionality of the computing device 1000A shown in FIG. 12 can also be accomplished by multiple computing devices 1000. Similarly, the functionality of computing device 1000B can also be accomplished by multiple computing devices 1000.

[0194] An embodiment of this application provides a computer program product includinginstructions, which can run on a computing device cluster or be stored in any available medium. When it is run by a computing device cluster, the computing device cluster is made to execute the methods provided above, or the computing device cluster is made to implement the functions of the devices provided above.

[0195] An embodiment of this application provides a computer readable storage medium including instructions. The computer readable storage medium is any available medium that computing devices can store, or a data storage device such as a data center containing one or more available media. The available media can be magnetic media (such as floppy disks, hard drives, and magnetic tapes), optical media (such as digital video disc (DVD)), or semiconductor media (such as solid-state drives), etc. The computer readable storage medium includes instructions. When the instructions are run on a computer device cluster, the computer device cluster executes the methods provided above.

[0196] An embodiment of this application provides a chip system, where the chip system includes a memory and a processor, the memory is configured to store a computer program, and the processor is configured to invoke the computer program from the memory and run the computer program, so that a server on which a chip is disposed performs the methods provided above.

[0197] A person of ordinary skill in the art may be aware that, in combination with the examples described in the embodiments disclosed in this specification, units and algorithm steps can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed by hardware or software depends on particular applications and design constraints of the technical solutions. A person skilled in the art may use different methods to implement the described functions for each particular application, but it should not be considered that the implementation goes beyond the scope of this application.

[0198] It may be clearly understood by a person skilled in the art that, for the purpose of convenient and brief description, for a detailed working process of the foregoing system, apparatus, and unit, refer to a corresponding process in the foregoing method embodiment. Details are not described herein again.

[0199] In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus, and method may be implemented in other manners. For example,the described apparatus embodiment is merely an example. For example, the unit division is merely logical function division and may be other division in actual implementation. For example, a plurality of units or components may be combined or integrated into another system, or some features may be ignored or not performed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections may be implemented through some interfaces. The indirect couplings or communication connections between the apparatuses or units may be implemented in electronic, mechanical, or other forms.

[0200] The units described as separate parts may be or may not be physically separate, and parts displayed as units may be or may not be physical units, may be located in one position, or may be distributed on a plurality of network units. Some or all of the units may be selected based on actual requirements to achieve the objectives of the solutions of the embodiments.

[0201] In addition, functional units in the embodiments of this application may be integrated into one processing unit, or each of the units may exist alone physically, or two or more units are integrated into one unit.

[0202] When the functions are implemented in a form of a software functional unit and sold or used as an independent product, the functions may be stored in a computer readable storage medium. Based on such an understanding, the technical solutions in this application essentially, or the part contributing to the prior art, or some of the technical solutions may be implemented in a form of a software product. The computer software product is stored in a storage medium, and includes several instructions for instructing a computer device (which may be a personal computer, a server, a network device, or the like) to perform all or some of the steps of the methods described in the embodiments of this application. The foregoing storage medium includes: any medium that can store program code, such as a USB flash drive, a removable hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc.

[0203] The foregoing descriptions are merely specific implementations of this application, but are not intended to limit the protection scope of this application. Any variation or replacement readily figured out by a person skilled in the art within the technical scope disclosed in this application shall fall within the protection scope of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

Claims

CLAIMSWhat is claimed is:

1. A method of adjusting a model, comprising:obtaining a gating network logit value of each expert network in a first mixture of experts (MoE) layer corresponding to each token in T tokens based on the T tokens and a target MoE model, wherein the first MoE layer belongs to at least one MoE layer in the target MoE model, the first MoE layer comprises N₁ expert networks and the gating network, the logit value of each expert network corresponding to each token is used to represent a probability of activating each expert network to process each token, T is a positive integer, and N₁ is a positive integer greater than 1;obtaining K selected expert networks corresponding to each token in the T tokens based on the logit value of each expert network in the first MoE layer corresponding to each token in the T tokens, wherein the K selected expert networks corresponding to each token in the T tokens are such that a value of a first loss function is maximized, the value of the first loss function is determined based on a first penalty value and a logit value of each selected expert network in the K selected expert networks corresponding to each token in the T tokens, K is a positive integer less than N₁, the first penalty value is determined based on the number of selected expert networks, and / or, the first penalty value is determined based on a logit value of each initially activated expert network in K initially activated expert networks corresponding to each token in the T tokens and the logit value of each expert network in the K selected expert networks corresponding to each token in the T tokens, and the logit value of each initially activated expert network in the K initially activated expert networks corresponding to each token is greater than or equal to the logit value of each expert network in the N₁ expert networks corresponding to each token other than the K initially activated expert networks;obtaining an adjusted first MoE layer by pruning at least one expert network in the first MoE layer based on the K selected expert networks corresponding to each token in the T tokens, wherein the number of expert networks in the adjusted first MoE layer is less than N₁; andobtaining an adjusted target MoE model based on the adjusted first MoE layer, wherein the adjusted target MoE model comprises the adjusted first MoE layer.

2. The method according to claim 1, wherein the value of the first loss function comprises a difference between the first penalty value and a sum of logit values of the K selected expert networks corresponding to each token in the T tokens.

3. The method according to claim 1 or 2, wherein the first penalty value is determined based on the number of selected expert networks, and the first penalty value is a first value; orthe first penalty value is determined based on the logit value of each initially activated expert network in the K initially activated expert networks corresponding to each token and the logit value of each expert network in the K selected expert networks corresponding to each token in the T tokens, and the first penalty value is a second value; orthe first penalty value is determined based on the number of selected expert networks, the logit value of each initially activated expert network in the K initially activated expert networks corresponding to each token and the logit value of each expert network in the K selected expert networks corresponding to each token in the T tokens, and the first penalty value is a sum of a first value and a second value; andamong them, the first value is positively correlated with the number of selected expert networks, and the second value is positively correlated with a difference between the logit value of each initially activated expert network in the K initially activated expert networks corresponding to each token and the logit value of each expert network in the K selected expert networks corresponding to each token in the T tokens.

4. The method according to claim 3, wherein the first value comprises a product of a first penalty factor and the number of selected expert networks, and the first penalty factor is a preset value; orthe first value comprises a dot product of a second penalty factor and a first vector, the second penalty factor comprises an N₁-dimensional vector, the first vector comprises an N₁-dimensional binary vector, and the first vector is used to represent the number of selected expert networks.

5. The method according to claim 3 or 4, wherein the second value comprises a sum of differences between logit values of the K initially activated expert networks corresponding to each token and logit values of the K selected expert networks corresponding to each token in the T tokens.

6. The method according to any one of claims 1-5, wherein a first retained expert networkbelongs to the adjusted first MoE layer, each pruned expert network in at least one pruned expert network belongs to the N1expert networks other than expert networks in the adjusted first MoE layer, and the method further comprises:updating a weight of the first retained expert network based on a weight of each pruned expert network in the at least one pruned expert network and the weight of the first retained expert network, wherein the updated weight of the first retained expert network is determined by a weighted sum of the weight of each pruned expert network in the at least one pruned expert network and the weight of the first retained expert network.

7. The method according to claim 6, wherein a difference between a first logit value of the first retained expert network corresponding to a first token and a first logit value of a first pruned expert network corresponding to the first token is less than or equal to a first preset threshold, the first token belongs to the T tokens, and the first pruned expert network belongs to the at least one pruned expert network.

8. The method according to any one of claims 1-7, wherein a second MoE layer belongs to the first MoE model, the second MoE layer comprises N₂ expert networks, N₂ is a positive integer greater than 1, and the obtaining an adjusted target MoE model based on the adjusted first MoE layer comprises:obtaining an adjusted second MoE layer, wherein the number of expert networks in the adjusted second MoE layer is less than or equal to N₂; andobtaining the adjusted target MoE model based on the adjusted first MoE layer and the adjusted second MoE layer, wherein the number of expert networks in the adjusted first MoE layer and the number of expert networks in the adjusted second MoE layer are different.

9. A device for adjusting a model, comprising:an obtaining unit, configured to obtain a gating network logit value of each expert network in a first mixture of experts (MoE) layer corresponding to each token in T tokens based on the T tokens and a target MoE model, wherein the first MoE layer belongs to at least one MoE layer in the target MoE model, the first MoE layer comprises N₁ expert networks and the gating network, the logit value of each expert network corresponding to each token is used to represent a probability of activating each expert network to process each token, T is a positive integer, and N₁ is a positive integer greater than 1;a processing unit, configured to obtain K selected expert networks corresponding to each token in the T tokens based on the logit value of each expert network in the first MoE layer corresponding to each token in the T tokens, wherein the K selected expert networks corresponding to each token in the T tokens are such that a value of a first loss function is maximized, the value of the first loss function is determined based on a first penalty value and a logit value of each selected expert network in the K selected expert networks corresponding to each token in the T tokens, K is a positive integer less than Ni, the first penalty value is determined based on the number of selected expert networks, and / or, the first penalty value is determined based on a logit value of each initially activated expert network in K initially activated expert networks corresponding to each token in the T tokens and the logit value of each expert network in the K selected expert networks corresponding to each token in the T tokens, and the logit value of each initially activated expert network in the K initially activated expert networks corresponding to each token is greater than or equal to the logit value of each expert network in the N₁ expert networks corresponding to each token other than the K initially activated expert networks;the processing unit, also configured to obtain an adjusted first MoE layer by pruning at least one expert network in the first MoE layer based on the K selected expert networks corresponding to each token in the T tokens, wherein the number of expert networks in the adjusted first MoE layer is less than N₁; andthe processing unit, also configured to obtain an adjusted target MoE model based on the adjusted first MoE layer, wherein the adjusted target MoE model comprises the adjusted first MoE layer.

10. The device according to claim 9, wherein the value of the first loss function comprises a difference between the first penalty value and a sum of logit values of the K selected expert networks corresponding to each token in the T tokens.

11. The device according to claim 9 or 10, wherein the first penalty value is determined based on the number of selected expert networks, and the first penalty value is a first value; orthe first penalty value is determined based on the logit value of each initially activated expert network in the K initially activated expert networks corresponding to each token and the logit value of each expert network in the K selected expert networks corresponding to each token in the T tokens, and the first penalty value is a second value; orthe first penalty value is determined based on the number of selected expert networks, the logit value of each initially activated expert network in the K initially activated expert networks corresponding to each token and the logit value of each expert network in the K selected expert networks corresponding to each token in the T tokens, and the first penalty value is a sum of a first value and a second value; andamong them, the first value is positively correlated with the number of selected expert networks, and the second value is positively correlated with a difference between the logit value of each initially activated expert network in the K initially activated expert networks corresponding to each token and the logit value of each expert network in the K selected expert networks corresponding to each token in the T tokens.

12. The device according to claim 11, wherein the first value comprises a product of a first penalty factor and the number of selected expert networks, and the first penalty factor is a preset value; orthe first value comprises a dot product of a second penalty factor and a first vector, the second penalty factor comprises an N₁-dimensional vector, the first vector comprises an N₁-dimensional binary vector, and the first vector is used to represent the number of selected expert networks.

13. The device according to claim 11 or 12, wherein the second value comprises a sum of differences between logit values of the K initially activated expert networks corresponding to each token and logit values of the K selected expert networks corresponding to each token in the T tokens.

14. The device according to any one of claims 9-13, wherein a first retained expert network belongs to the adjusted first MoE layer, each pruned expert network in at least one pruned expert network belongs to the N1expert networks other than expert networks in the adjusted first MoE layer, and the processing unit is configured to:update a weight of the first retained expert network based on a weight of each pruned expert network in the at least one pruned expert network and the weight of the first retained expert network, wherein the updated weight of the first retained expert network is determined by a weighted sum of the weight of each pruned expert network in the at least one pruned expert network and the weight of the first retained expert network.

15. The device according to claim 14, wherein a difference between a first logit value of thefirst retained expert network corresponding to a first token and a first logit value of a first pruned expert network corresponding to the first token is less than or equal to a first preset threshold, the first token belongs to the T tokens, and the first pruned expert network belongs to the at least one pruned expert network.

16. The device according to any one of claims 9-15, wherein a second MoE layer belongs to the first MoE model, the second MoE layer comprises N₂ expert networks, N₂ is a positive integer greater than 1, and the processing unit is configured to:obtain an adjusted second MoE layer, wherein the number of expert networks in the adjusted second MoE layer is less than or equal to N2, andobtain the adjusted target MoE model based on the adjusted first MoE layer and the adjusted second MoE layer, wherein the number of expert networks in the adjusted first MoE layer and the number of expert networks in the adjusted second MoE layer are different.

17. A computing device cluster, comprising at least one computing device, wherein the computing device comprises a processor and a memory coupled with the processor, wherein the memory is configured to store a computer program, and the processor is configured to invoke and run the computer program stored in the memory, so that the computing device executes the method according to any one of claims 1-8.

18. A computer program product comprising instructions, wherein when the computer program product is run on a server, the server is enabled to perform the method according to any one of claims 1-8.

19. A computer readable storage medium storing instructions, wherein when run on a server, the computer readable storage medium enables the server to perform the method according to any one of claims 1-8.

Images