Method and apparatus for generating emulator for pre-trained large model

Abstract

A method and apparatus for generating an emulator for a pre-trained large model. The emulator is used for performing cross-domain fine-tuning on a first large model comprising N functional units. The method comprises: performing H rounds of first update, an h-th round of first update comprising: on the basis of importance scores, in the h-th round of first update, of functional units in the first large model subjected to an (h-1)-th round of cropping, cropping, from a second large model subjected to an (h-1)-th round of first update, a plurality of functional units and PEFT modules corresponding thereto, wherein functional units comprised in the second large model subjected to the (h-1)-th round of first update are the same as functional units comprised in the first large model subjected to the (h-1)-th round of cropping; using the second large model subjected to an h-th round of cropping as a teacher model and the first large model subjected to the (h-1)-th round of cropping as a teacher model, updating parameters in the PEFT modules comprised in the second large model subjected to the h-th round of cropping; and finally, on the basis of the second large model subjected to H rounds of first update, generating the emulator corresponding to the first large model.

Description

Methods and apparatus for generating simulators from pre-trained large models

[0001] This application claims priority to Chinese patent application filed on January 21, 2025, with application number 202510100580.X and entitled "Method and Apparatus for Generating Simulators of Pre-trained Large Models", the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments in this specification belong to the field of computer technology, and in particular relate to a method and apparatus for generating a simulator for a pre-trained large model. Background Technology

[0003] For privacy protection purposes, cross-domain fine-tuning of pre-trained large models may be necessary. During this process, the model provider can use lossy compression techniques to compress the pre-trained large model and generate an emulator corresponding to it. This emulator is then passed to the data provider. The data provider can then use this emulator and its own training dataset to train an adapter corresponding to the large model. This adapter is then passed to the model provider. The model provider can then plug the adapter into the pre-trained large model to obtain a target large model capable of performing downstream tasks, thus completing the cross-domain fine-tuning process.

[0004] In the process of generating a simulator, multiple network layers can be trimmed from a pre-trained large model. The trimmed large model is used as the student model, and the untrimmed large model is used as the teacher model. Knowledge distillation is then performed on the trimmed large model to obtain the simulator that will be passed on to the data provider. However, in this implementation, it is necessary to directly update a large number of parameters included in the trimmed large model in a relatively large number of iterations to obtain a simulator with satisfactory performance. That is, it requires a large amount of computing resources, storage resources, and time resources to obtain a simulator with satisfactory performance. Summary of the Invention

[0005] The purpose of this invention is to provide a method and apparatus for generating simulators from pre-trained large models.

[0006] Firstly, a method is provided for generating a simulator for a pre-trained large model. The simulator is used to perform cross-domain fine-tuning on a pre-trained first large model, which includes N functional units. The method includes: performing H rounds of first updates. Any h-th round of first updates includes: based on the importance scores of each functional unit in the first large model pruned in h-1 rounds in the h-th round of first updates, pruning several functional units from a second large model that has undergone h-1 rounds of first updates and performing parameter-efficient fine-tuning on their corresponding parameters. The tuning (PEFT) module obtains a second large model after the h-th round of trimming. The functional units included in the second large model after the h-1-th round of first update are the same as those included in the first large model after the h-1-th round of trimming. The first input sample is processed using the second large model after the h-th round of trimming to obtain a first processing result. Based on the first processing result and the second processing result, the parameters in each PEFT module included in the second large model after the h-th round of trimming are updated, wherein the second processing result is obtained by processing the first input sample using the first large model after the h-1-th round of trimming. Based on the second large model after the H-th round of first update, a simulator corresponding to the first large model is generated.

[0007] Secondly, an apparatus for generating a simulator for a pre-trained large model is provided. The simulator is used to perform cross-domain fine-tuning on a pre-trained first large model, which includes N functional units. The apparatus includes: an update processing unit for performing H rounds of first updates. Any h-th round of first updates includes: based on the importance scores of each functional unit in the h-1 round of pruning in the h-th round of first updates, pruning several functional units and their corresponding PEFT modules from a second large model that has undergone h-1 rounds of first updates, obtaining a second large model that has undergone h-th rounds of pruning, and then performing h-1 rounds of first updates. The updated second-large model includes the same functional units as the first-large model after h-1 rounds of pruning; the first input sample is processed using the second-large model after h-1 rounds of pruning to obtain a first processing result; based on the first processing result and the second processing result, the parameters in each PEFT module of the second-large model after h-1 rounds of pruning are updated, wherein the second processing result is obtained by processing the first input sample using the first-large model after h-1 rounds of pruning; the simulator generation unit is used to generate a simulator corresponding to the first-large model based on the second-large model after H rounds of first updates.

[0008] Thirdly, a computing device is provided, including a memory and a processor, wherein the memory stores computer programs / instructions, and the processor executes the computer programs / instructions to implement the method described in the first aspect.

[0009] Fifthly, a computer-readable storage medium is provided having a computer program / instructions stored thereon, wherein when the computer program / instructions are executed in a computing device, the computing device performs the method described in the first aspect.

[0010] In the technical solution provided in the embodiments of this specification: by progressively executing the pruning and knowledge distillation of the second major model, which serves as the student model, and during the knowledge distillation of the second major model, only the parameters in the PEFT modules that have not been pruned at the current moment are updated. This allows for obtaining a simulator with satisfactory performance with relatively low computational, storage, and time resources. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments in this specification, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 is a schematic diagram illustrating the relationship between large models provided in an embodiment of this specification.

[0013] Figure 2 is a schematic diagram of the process of performing the first update of the hth round on the second large model, provided by an example in the embodiments of this specification.

[0014] Figure 3 is a schematic diagram of the process of performing the kth round of the second update on the second large model, provided by an example in the embodiments of this specification.

[0015] Figure 4 is a schematic diagram of a progressive cropping large model provided as an example in the embodiments of this specification;

[0016] Figure 5 is a schematic diagram of the structure of a device for generating a pre-trained large model simulator provided in the embodiments of this specification. Detailed Implementation

[0017] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0018] For privacy protection purposes, it may be necessary to perform cross-domain fine-tuning on the pre-trained large model (hereinafter referred to as the first large model). During the cross-domain fine-tuning of the first large model, the model provider can use lossy compression techniques to generate an emulator corresponding to the source model. The emulator will be passed to the data provider. The data provider can then train an adapter corresponding to the first large model based on the emulator and its own training dataset. The adapter will be passed to the model provider. The model provider can then plug the emulator into the first large model to obtain a second large model that can be used to perform downstream tasks.

[0019] During cross-domain fine-tuning, on the one hand, the model provider is unaware of the training dataset held by the data provider, the data provider is unaware of the first-largest model provided by the model provider, and also unaware of the second-largest model ultimately generated by the data provider for performing downstream tasks, thus meeting privacy protection requirements. On the other hand, the simulator is obtained by lossily compressing the first-largest model, resulting in a relatively small parameter scale. The data provider only needs to consume relatively few computing, storage, and time resources to train an adapter that meets the performance requirements. Furthermore, the simulator is obtained by compressing the first-largest model; the adapter trained based on the simulator cannot have the same performance as the second-largest model obtained through cross-domain fine-tuning, which also reflects the protection of the security of the first and second-largest models.

[0020] The key challenge of cross-domain fine-tuning lies in how to effectively perform lossy compression of the first large model; that is, the key challenge of cross-domain fine-tuning lies in how to obtain a simulator corresponding to the first large model.

[0021] In one possible implementation, multiple network layers can be trimmed from a first large model. The trimmed first large model is used as the student model, and the undripped first large model is used as the teacher model. Knowledge distillation is then performed on the trimmed first large model to obtain a simulator that will be passed to the data provider. For example, the parameters of the first two and last two network layers of the first large model can be kept unchanged, while the remaining network layers are trimmed in equal increments. For instance, for network layers 3 through 10, only layers 3, 5, 7, and 9 are retained, while layers 4, 6, 8, and 10 are trimmed. The undripped first large model is used as the teacher model, and the trimmed first large model is used as the student model. Knowledge distillation is performed on the student model, causing it to mimic the input and output behavior of the teacher model. The student model that has undergone knowledge distillation will then be used as the simulator.

[0022] In this implementation, a simulator with satisfactory performance needs to be obtained by updating a large number of parameters in the trimmed first model through a large number of iterations, which consumes a lot of computing resources, storage resources and time resources.

[0023] This specification provides a method and apparatus for generating a simulator from a pre-trained large model. This simulator is used for cross-domain fine-tuning of a pre-trained first large model, which includes N functional units. It can first perform H rounds of first updates, where each h-th round of first update includes: based on the importance scores of each functional unit in the h-1 round-pruned first model in the h-th round of first update, pruning several functional units and their corresponding PEFT modules from the h-1 round-pruned second large model to obtain the h-th round-pruned second large model. The functional units included in the h-1 round-pruned second large model are the same as those included in the h-1 round-pruned first large model. The h-th round-pruned second large model is then used to process a first input sample to obtain a first processing result. Based on the first and second processing results, the parameters in each PEFT module of the h-th round-pruned second large model are updated. The second processing result is obtained by processing the first input sample using the h-1 round-pruned first large model. Finally, based on the H rounds of first updates of the second large model, a simulator corresponding to the first large model is generated.

[0024] In this way, by progressively pruning and distilling the knowledge of the second major model, which serves as the student model, and updating only the parameters in the PEFT modules that have not been pruned at the current moment during the knowledge distillation process, a simulator with satisfactory performance can be obtained with relatively low computational, storage, and time resources.

[0025] The relationship between the first unpruned model and the second unpruned model will be explained first.

[0026] The first pre-trained model serves as the source model for cross-domain fine-tuning. Without pruning, the first model typically includes multiple stacked Transformer modules, which can be further decomposed into multi-head attention (MHA) modules and multi-layer perceptron (MLP) modules.

[0027] From the multiple Transformer modules included in the first main model, N functional units that allow lossy compression can be identified. Specifically, if coarse-grained pruning is required, these N functional units can be the N Transformer modules belonging to the multiple Transformer modules; if fine-grained pruning is required, these N functional units can include the N / 2 MHA modules and N / 2 MLP modules included in the N / 2 Transformer modules belonging to the multiple Transformer modules.

[0028] The MHA and MLP modules included in the same Transformer module may have different importance in the first large model. Using coarse-grained pruning with the Transformer module as the pruning granularity may result in the pruning of relatively important MHA or MLP modules, which will ultimately lead to a significant decrease in the performance of the subsequent simulator. Therefore, fine-grained pruning is preferred.

[0029] Currently, various efficient parameter fine-tuning methods are provided for fine-tuning large pre-trained models, including adapter-based methods, reparameterization-based methods, and prompt-based methods, aiming to minimize the number of trainable parameters. All of the aforementioned efficient parameter fine-tuning methods require setting multiple PEFT modules for the large pre-trained model. The fine-tuning process is completed by updating the parameters of these multiple PEFT modules. Based on this, N PEFT modules corresponding to the N functional units can be configured. Then, referring to Figure 1, a second large model corresponding to the first large model can be formed using the first large model and the N PEFT modules corresponding to the N functional units.

[0030] The following example, which specifically includes N low-rank adaptation (LORA) modules, illustrates the relationship between the N PEFT modules in the second-largest model and the N functional units included in the first-largest model.

[0031] First, any nth LORA module among the N LORA modules corresponds to the nth functional unit among the N functional units included in the first large model. Here, it is assumed that the parameter matrix of the nth functional unit is W. Pn Then the nth LORA module includes the parameter matrix a Mn and b Mn , for a Mn and b Mn The result W obtained by multiplication Mn , and the parameter matrix W of the nth functional unit Pn The number of rows and columns is the same. Among them, WMn This represents the parameter matrix W of the nth functional unit during the update of the second major model. Pn The increments of each parameter included; therefore, in updating the second largest model, only a needs to be updated. Mn and b Mn The parameters in the table are sufficient; there is no need to update the parameter matrix W of the nth functional unit. Pn The included large number of parameters; when the iteration termination condition is met, it can be based on a Mn and b Mn Calculate W Mn The parameter matrix W of the nth functional unit Pn Updated to W Pn +W Mn That's all.

[0032] Furthermore, this section continues to assume that in the first large model, independent of the second large model, the input of the (n+1)th functional unit out of N functional units is... It is the output of the nth functional unit. In the second major model, which corresponds to the first major model, the corresponding functional units are connected in parallel. The input of the (n+1)th functional unit... and the input of the (n+1)th PEFT module All should be the output of the nth functional unit. With the output of the nth PEFT module The result of the merging, i.e. and In essence, it can be

[0033] Understandably, when using N PEFT modules and the first large model to form a second large model corresponding to the first large model, the parameter matrices of all N PEFT modules need to be initialized. For example, for any nth LORA module, the parameter matrix a... Mn and b Mn It is necessary to modify the parameter matrix a. Mn and b Mn Perform initialization to ensure that a is initialized. Mn and b Mn In the resulting increment matrix, each element has a value of 0 or less than a preset value that approaches 0.

[0034] After obtaining the second major model corresponding to the first major model, a method for generating a simulator from a pre-trained model, as provided in the embodiments of this specification, can be executed based on the first and second major models. This method enables the gradual pruning and knowledge distillation of the second major model to generate a simulator for cross-domain fine-tuning from the pre-trained first major model.

[0035] The method for generating a simulator from a pre-trained model provided in the embodiments of this specification can be divided into two generation stages. In the first generation stage, an H-round first update can be performed on the second major model. In the second generation stage, a simulator corresponding to the first major model can be generated based on the second major model after the H-round first update. For example, an H-1-round second update can be performed on the second major model after the H-1-round second update, and then the simulator corresponding to the first major model can be determined based on the second major model after the H-1-round second update.

[0036] The following example describes the process of performing an arbitrary h-th round first update on the second large model during the first generation phase.

[0037] Figure 2 is a schematic diagram illustrating the process of performing the first update of the h-th round on the second largest model, as exemplarily provided in the embodiments of this specification.

[0038] Referring to Figure 2, any h-th round of first update may include some or all of the following steps S201 to S209.

[0039] In step S201, the importance score of each functional unit in the first large model after the h-1th round of pruning is determined in the first update of the hth round.

[0040] In one possible implementation, for any i-th functional unit included in the first large model after h-1 rounds of pruning, the i-th functional unit can be pruned from the first large model after h-1 rounds of pruning to obtain the i-th test model; then, based on the i-th test model, the importance score of the i-th functional unit in the first update of the h-th round can be determined.

[0041] For example, the number of functional units allowed for lossy compression in the first large model is N, and the number of functional units pruned in each round of pruning of the first large model is T. The number of functional units included in the first large model after the (h-1)th round of pruning is NT*(h-1). During the first update of the hth round of the second large model, the importance scores of the NT*(h-1) functional units included in the first large model after the (h-1)th round of pruning in the first update of the hth round can be determined.

[0042] The number of execution rounds H for the first update, and the number of functional units pruned per round of pruning for the first or second largest model / the number of PEFT modules T, can all be flexibly set based on the required compression ratio. For example, for a first largest model containing 64 functional units, if a lossy compression ratio of 50% is required, the number of execution rounds for the first update can be set to 8, and the number of functional units pruned per round of pruning for the first largest model can be set to 4. As another example, for a first largest model containing 12 functional units, if a lossy compression ratio of 50% is required, the number of execution rounds for the first update can be set to 6, and the number of functional units pruned per round of pruning for the first largest model can be set to 1.

[0043] For example, referring to Figure 3, it is assumed that the first large model without trimming includes 12 functional units, such as functional units 1 to 12. After 1 to 5 rounds of trimming performed on the first large model, functional units 9, 8, 3, 11 and 5 were trimmed from the first large model in sequence. Therefore, in the first round of the first update performed on the second largest model, the 12 importance scores of functional units 1-12 in the first round of the first update need to be calculated; in the second round of the first update performed on the second largest model, the first largest model after the first round of pruning still includes functional units 1-8 and functional units 10-12, and the 11 importance scores of functional units 1-8 and functional units 10-12 in the first round of the first update need to be calculated; in the third round of the first update performed on the second largest model, the first largest model after the second round of pruning still includes functional units 1-7 and functional units 10-12, and the 10 importance scores of functional units 1-7 and functional units 10-12 in the third round of the first update need to be calculated; in the fourth round of the first update performed on the second largest model, the first largest model after the third round of pruning... The model also includes functional units 1, 2, 4-7, and 10-12. Nine importance scores for functional units 1, 2, 4-7, and 10-12 need to be calculated in the first update of round 4. In the first update of round 5 performed on the second largest model, the first largest model after round 4 pruning still includes functional units 1, 2, 4, 6, 7, and 10-12. Eight importance scores for functional units 1, 2, 4, 6, 7, and 10-12 need to be calculated in the first update of round 5. In the first update of round 6 performed on the second largest model, the first largest model after round 5 pruning still includes functional units 1, 2, 4, 6, 7, 10, and 12. Seven importance scores for functional units 1, 2, 4, 6, 7, 10, and 12 need to be calculated in the first update of round 6.

[0044] In a more concrete example, the i-th test model can be used to process the sample word sequence to obtain the predicted probability sequence corresponding to the word sequence. Then, based on the predicted probability sequence, the perplexity of the i-th test model is calculated. Finally, based on the perplexity of the i-th test model, the importance score of the i-th functional unit in the first update of the h-th round is determined. It's easy to understand that perplexity and importance score are positively correlated; the lower the perplexity of the i-th test model, the better its performance, and consequently, the lower the importance score of the i-th functional unit in the first update of the h-th round.

[0045] Alternatively, the importance score of the i-th functional unit in the first update of round h can be determined based on the i-th test model. For example, the first loss of the i-th test model on a certain validation dataset can be calculated, as well as the second loss of the largest model after round h-1 pruning on the same validation dataset. The rate of change of loss calculated based on the first and second losses is positively correlated with the importance score of the i-th test model in the first update of round h.

[0046] Alternatively, other implementation methods can be used to determine the importance scores of each functional unit in the first update of the h-th round of the first large model after h-1 rounds of pruning. For example, N importance scores of N functional units in the first large model without pruning can be pre-calculated; for any functional unit among these N functional units, the importance score of that functional unit in the first update of the h-th round remains unchanged, that is, if the functional unit exists in the first large model after h-1 rounds of pruning, the importance score of that functional unit in the first update of the h-th round is still the same as the importance score of that functional unit in the first large model without pruning. This means that the aforementioned step S201 is actually optional rather than mandatory.

[0047] When the execution round h is less than H, the following step S203 can also be performed: based on the importance scores of each functional unit in the first major model after the (h-1)th round of pruning in the first update of the hth round, several functional units are pruned from the first major model after the (h-1)th round of pruning to obtain the first major model after the hth round of pruning.

[0048] And, in step S205, based on the importance scores of each functional unit in the first large model after the h-1th round of pruning in the first update of the h-th round, several functional units and their corresponding parameter fine-tuning PEFT modules are pruned from the second large model after the h-1th round of the first update to obtain the second large model after the h-th round of pruning. The functional units included in the second large model after the h-1th round of the first update are the same as the functional units included in the first large model after the h-1th round of pruning.

[0049] Continuing with the previous example, after obtaining the importance scores of each of the NT*(h-1) functional units included in the first large model after the h-1th round of pruning in the first update of the h-th round, T functional units can be pruned from the first large model after the h-1th round of pruning in order of increasing importance scores to obtain the first large model after the h-th round of pruning; and, from the second large model after the h-1th round of the first update, the T functional units and their corresponding T PEFT modules are pruned.

[0050] For example, please refer to Figure 3. In the first update of the first round performed on the second large model, after determining the 12 importance scores of functional units 1 to 12 in the first update of the first round, functional unit 9 has the smallest importance score among the 12 importance scores, so functional unit 9 can be cut from the first large model after the 0th round of pruning; and functional unit 9 and its corresponding PEFT module 9 are cut from the second large model after the 0th round of the first update. Similarly, in the first update of the second large model, after determining the 11 importance scores of functional units 1 to 7 and 10 to 12 in the first update of the second round, functional unit 8 has the smallest importance score among the 11 importance scores, so functional unit 8 can be cut from the first large model after the 1st round of pruning; and functional unit 8 and its corresponding PEFT module 8 are cut from the second large model after the 1st round of the first update. Similarly, in the third round of the first update performed on the second large model, functional unit 3 may be cut from the first large model after the second round of trimming, and functional unit 3 and its corresponding PEFT module 3 may be cut from the second large model after the second round of the first update; in the fourth round of the first update performed on the second large model, functional unit 11 may be cut from the first large model after the third round of trimming, and functional unit 11 and its corresponding PEFT module 11 may be cut from the second large model after the third round of the first update; in the fifth round of the first update performed on the second large model, functional unit 5 may be cut from the first large model after the fourth round of trimming, and functional unit 5 and its corresponding PEFT module 5 may be cut from the second large model after the fourth round of the first update; in the sixth round of the first update performed on the second large model, functional unit 6 and its corresponding PEFT module 6 may be cut from the second large model after the fifth round of the first update.

[0051] After obtaining the second largest model after the h-th round of pruning, it can be used as the student model, and the first largest model after the (h-1)-th round of pruning can be used as the teacher model. On the same input samples, knowledge distillation is performed on the second largest model after the h-th round of pruning, that is, through the following steps S207 and S209, knowledge distillation is achieved on the second largest model after the h-th round of pruning.

[0052] Step S207: Process the first input sample using the second largest model after the h-th round of cropping to obtain the first processing result.

[0053] Step S209: Based on the first processing result and the second processing result, update the parameters in each PEFT module of the second large model after the h-th round of pruning, wherein the second processing result is obtained by processing the first input sample using the first large model after the h-1-th round of pruning.

[0054] When updating the parameters of each PEFT module in the second large model after the h-th round of pruning, the goal is to minimize the difference between the first and second processing results. It is understood that in the h-th round of the first update performed on the second large model, steps S207 and S209 can be executed multiple times. The h-th round of the first update on the second large model is completed when the difference between the first and second processing results meets a certain preset condition or the number of iterations reaches a threshold.

[0055] After completing the H-round first update of the second major model through the aforementioned steps S201 to S209, the second generation stage can be entered. As mentioned earlier, in the second generation stage, a simulator corresponding to the first major model can be generated based on the second major model after the H-round first update. For example, for the second major model after the H-round first update, the H-1-round second update is then performed, and the simulator corresponding to the first major model is determined based on the second major model after the H-1-round second update.

[0056] The following example describes the process of performing an arbitrary k-th round of second update on a large model that has undergone the first update of round H in the second generation phase.

[0057] Figure 4 is a schematic diagram illustrating the process of performing the kth round of the second update on the second large model, as exemplarily provided in the embodiments of this specification.

[0058] Step S401: Process the second input sample using the second largest model that has undergone the (k-1)th round of second updates to obtain the third processing result. It can be understood that when k is 1, the second largest model that has undergone the (k-1)th round of second updates mentioned in step S401 refers to the second largest model that has undergone the Hth round of first updates.

[0059] Step S403: Based on the third processing result and the fourth processing result, update the parameters in each PEFT module of the second large model that has undergone the second update in the (k-1)th round. The fourth processing result is obtained by processing the second input sample using the first large model that has been pruned in the (Hk-1)th round.

[0060] When updating the parameters in the PEFT module of the second largest model after the (k-1)th round of the second update, the goal is to minimize the difference between the third and fourth processing results. It can be understood that in the kth round of the second update performed on the second largest model, steps S401 and S403 can be executed multiple times. When the difference between the third and fourth processing results meets a certain preset condition or the number of iterations reaches a threshold, the kth round of the second update for the second largest model is completed.

[0061] After performing the H-1 round of the second update on the second major model that underwent the first update in round H, the simulator corresponding to the first major model can be determined based on the second major model that underwent the second update in round H-1. Taking the PEFT module as an example of a LORA module, the structure of the second major model that underwent the second update in round H-1 is the same as the structure of the second major model that underwent the trimming in round H. For example, referring to Figure 3, in the second major model that underwent the second update in round H-1, functional units 3, 5, 6, 8, 9, and 11, as well as their corresponding LORA modules 3, 5, 6, 8, 9, and 11, have been trimmed. That is, the second major model that underwent the second update in round H-1 also includes functional units 1, 2, 4, 7, 10, and 12, as well as their corresponding LORA modules 1, 2, 4, 7, 10, and 12. Based on the parameter matrices of LORA modules 1, 2, 4, 7, 10, and 12, the parameters of functional units 1, 2, 4, 7, 10, and 12 in the second large model are updated accordingly. At the same time, LORA modules 1, 2, 4, 7, 10, and 12 are deleted from the second large model after the second update in round H-1. The resulting second large model, which only includes functional units 1, 2, 4, 7, 10, and 12 and does not include LORA modules 1, 2, 4, 7, 10, and 12, can be used as the generated simulator for subsequent cross-domain fine-tuning of the first large model.

[0062] Based on the same concept as the aforementioned method embodiments, this specification also provides an apparatus 500 for generating a simulator for a pre-trained large model. The simulator is used to perform cross-domain fine-tuning on a first pre-trained large model, which includes N functional units. The apparatus includes an update processing unit 501, used to perform H rounds of first updates. Any h-th round of first updates includes: based on the importance scores of each functional unit in the first large model after h-1 rounds of pruning in the h-th round of first updates, pruning several functional units from the second large model after h-1 rounds of first updates and their corresponding parameter-efficiently fine-tuning PEFT modules, obtaining the h-th round of pruning... The functional units included in the second major model after the first update in round h-1 are the same as those included in the first major model after the trimming in round h-1. The first input sample is processed using the second major model after the trimming in round h to obtain a first processing result. Based on the first processing result and the second processing result, the parameters in each PEFT module included in the second major model after the trimming in round h are updated, wherein the second processing result is obtained by processing the first input sample using the first major model after the trimming in round h-1. The simulator generation unit 503 is used to generate a simulator corresponding to the first major model based on the second major model after the first update in round H.

[0063] This specification also provides a computer-readable storage medium storing a computer program / instruction that, when executed in a computer, causes the computer to perform a method for generating a simulator for a pre-trained large model, as provided in the foregoing embodiments.

[0064] This specification also provides a computing device in the embodiments, including a memory and a processor. The memory stores computer programs / instructions, and when the processor executes the computer programs / instructions, it implements a method for generating a simulator for a pre-trained large model provided in the foregoing embodiments.

[0065] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0066] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0067] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a server system. Of course, this application does not exclude the possibility that, with the future development of computer technology, the computer implementing the functions of the above embodiments can be, for example, a personal computer, a laptop computer, an in-vehicle human-machine interaction device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0068] While one or more embodiments of this specification provide the operational steps of the methods described in the embodiments or flowcharts, more or fewer operational steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is merely one possible order of execution among many steps and does not represent the only possible order. In actual device or end product execution, the methods shown in the embodiments or drawings may be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment). The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, the presence of other identical or equivalent elements in the process, method, product, or apparatus that includes the elements is not excluded. For example, the use of terms such as "first," "second," etc., is to denote names and does not indicate any particular order.

[0069] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, when implementing one or more of these specifications, the functions of each module can be implemented in one or more software and / or hardware components, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0070] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0071] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0072] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0073] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0074] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0075] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage, graphene storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0076] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0077] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0078] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0079] The above description is merely an embodiment of one or more embodiments of this specification and is not intended to limit the scope of these embodiments. Various modifications and variations can be made to these embodiments by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims.

Claims

1. A method for generating a simulator for a pre-trained large model, the simulator being used for cross-domain fine-tuning of a first pre-trained large model, the first large model comprising N functional units, the method comprising: Execute the first update of round H. Any first update of round h includes: Based on the importance scores of each functional unit in the first major model after h-1 rounds of pruning in the first update of h rounds, several functional units and their corresponding parameter fine-tuning PEFT modules are pruned from the second major model after h-1 rounds of the first update to obtain the second major model after h rounds of pruning. The functional units included in the second major model after h-1 rounds of the first update are the same as those included in the first major model after h-1 rounds of pruning. The first input sample is processed using the second largest model after the h-th round of pruning to obtain the first processing result; Based on the first and second processing results, update the parameters in each PEFT module of the second large model after the h-th round of pruning, wherein the second processing result utilizes the parameters after the h-1 round of pruning. The first large model of the round clipping is obtained by processing the first input sample; Based on the second largest model after the first update in round H, generate a simulator corresponding to the first largest model.

2. The method according to claim 1, wherein the PEFT module is a low-rank adaptive LORA module.

3. The method according to claim 1, wherein when h is less than H, the first update in the h-th round further includes: The functional units are trimmed from the first large model after the h-1th round of trimming to obtain the first large model after the h-th round of trimming.

4. The method according to claim 1, wherein generating a simulator corresponding to the first large model based on the second large model after the first update of H rounds comprises: For the second largest model that has undergone the first update in round H, perform the second update in round H-1. Any second update in round k includes: processing the second input sample using the second largest model that has undergone the second update in round k-1 to obtain a third processing result; updating the parameters in each PEFT module of the second largest model that has undergone the second update in round k-1 based on the third processing result and the fourth processing result, wherein the fourth processing result is obtained by processing the second input sample using the first largest model that has undergone the pruning in round Hk-1; Based on the second major model after the second update of round H-1, the simulator corresponding to the first major model is determined.

5. The method according to claim 1, wherein the first large model comprises a plurality of Transformer modules stacked sequentially, and the N functional units are N Transformer modules belonging to the plurality of Transformer modules.

6. The method according to claim 1, wherein the first large model comprises a plurality of Transformer modules stacked sequentially, and the N functional units include N / 2 multi-head attention (MHA) modules and N / 2 multilayer perceptron (MLP) modules belonging to the N / 2 Transformer modules of the plurality of Transformer modules.

7. The method according to claim 1, wherein the first update in the h-th round further comprises: Determine the importance score of each functional unit in the first major model after the h-1th round of pruning in the first update of the hth round.

8. The method according to claim 7, wherein determining the importance score of each functional unit in the first large model after the (h-1)th round of pruning in the first update of the hth round comprises: For any i-th functional unit included in the first large model after h-1 rounds of pruning, prune the i-th functional unit from the first large model after h-1 rounds of pruning to obtain the i-th test model; Based on the i-th test model, determine the importance score of the i-th functional unit in the first update of the h-th round.

9. The method according to claim 8, wherein determining the importance score of the i-th functional unit in the first update of the h-th round based on the i-th test model comprises: The i-th test model is used to process the sample word sequence to obtain the predicted probability sequence corresponding to the word sequence; Calculate the perplexity of the i-th test model based on the predicted probability sequence; Based on the confusion level, the importance score of the i-th functional unit in the first update of the h-th round is determined.

10. An apparatus for generating a simulator for a pre-trained large model, the simulator being used for cross-domain fine-tuning of a first pre-trained large model, the first large model comprising N functional units, the apparatus comprising: The update processing unit is used to execute the first update of round H. The first update of any round h includes: Based on the importance scores of each functional unit in the first major model after h-1 rounds of pruning in the first update of h rounds, several functional units and their corresponding parameter fine-tuning PEFT modules are pruned from the second major model after h-1 rounds of the first update to obtain the second major model after h rounds of pruning. The functional units included in the second major model after h-1 rounds of the first update are the same as those included in the first major model after h-1 rounds of pruning. The first input sample is processed using the second largest model after the h-th round of pruning to obtain the first processing result; Based on the first and second processing results, update the parameters in each PEFT module of the second large model after the h-th round of pruning, wherein the second processing result utilizes the parameters after the h-1 round of pruning. The first large model of the round clipping is obtained by processing the first input sample; The simulator generation unit is used to generate a simulator corresponding to the first large model based on the second large model after the first update of the H round.

11. A computing device comprising a memory and a processor, wherein the memory stores a computer program, and the processor, when executing the computer program, implements the method of any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computing device, the computing device performs the method of any one of claims 1-9.

Images