Method for determining configuration for artificial intelligence (AI) model and related apparatus

Abstract

A method for determining a configuration for an artificial intelligence (AI) model, which is used to determine a parallel configuration in an AI model training scenario. By pre-acquiring a basic operator context and a differential operator context corresponding to a model running on the basis of multiple parallel modes, the influence of configuration parameters of each parallel mode on operators in a model running process can be learned, and then an operator context corresponding to each of M groups of parallel configurations can be predicted. Finally, an optimal parallel configuration is selected for the model on the basis of the running performance of the operator context corresponding to each group of parallel configurations, so that the efficiency of determining a configuration for the model is effectively improved while consuming fewer resources, helping to meet the requirements of constantly changing scenarios.

Description

A method and related apparatus for determining the configuration of an artificial intelligence (AI) model.

[0001] This application claims priority to Chinese Patent Application No. 202411993246.3, filed with the State Intellectual Property Office of China on December 30, 2024, entitled "A Method and Apparatus for Determining the Configuration of an Artificial Intelligence (AI) Model", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method and apparatus for determining model configuration. Background Technology

[0003] With the continuous development of AI technology, the parameter size of AI models and the scale of training data are constantly increasing, significantly impacting the training efficiency of AI models. To improve the training efficiency of AI models, distributed parallel training technology has emerged. Distributed parallel training technology is a computational strategy that distributes the training task of the AI ​​model across multiple AI accelerators for parallel execution by partitioning the AI ​​model or training data along its dimensions. Current distributed parallel training technologies typically include data parallelism (DP), tensor parallelism (TP), pipeline parallelism (PP), context parallelism (CP), and expert parallelism (EP).

[0004] Because different distributed parallel training techniques can often be combined to achieve better training results, they are frequently used simultaneously in AI model training. However, the same distributed parallel training technique often has different configurations, and different configurations have different impacts on AI model training. Therefore, when training an AI model, it is often necessary to determine the configuration parameters of each distributed parallel training technique used to achieve optimal model training results. Currently, the main method for determining the parallel configuration of multiple distributed parallel training techniques is to train the AI ​​model with different combinations of parallel configurations in actual training scenarios, thereby testing the training effect of the AI ​​model under various combinations of parallel configurations, and then determining the optimal combination of parallel configurations.

[0005] However, testing the training effects of various parallel configuration combinations by running AI models has the problems of high resource costs and long optimization time, making it difficult to meet the ever-changing needs of scenarios. Summary of the Invention

[0006] This application provides a method for determining AI model configuration, which can effectively improve the efficiency of determining model configuration while consuming less resources, and is conducive to meeting the ever-changing needs of scenarios.

[0007] Firstly, a method for determining AI model configuration is provided, applied to determining the parallel configuration in an AI model training scenario. This method includes: an execution device acquiring a sequence of basic operators for N parallel modes of a first AI model, and a sequence of differential operators for each of the N parallel modes. The sequence of basic operators is the set of operators executed by all N parallel modes when running the first AI model, and the sequence of differential operators for each parallel mode consists of operators other than those in the sequence of basic operators when running the first AI model with the minimum amount of parallelism required for each parallel mode.

[0008] Then, the execution device determines the operator sequence corresponding to each of the M parallel configurations based on the basic operator sequence and the difference operator sequence for each of the N parallel modes. The M parallel configurations can be obtained by the execution device iterating through all combinations of configuration parameters for the N parallel modes.

[0009] Secondly, the execution device determines the performance of each operator in the operator sequence corresponding to each group of parallel configurations, and obtains the performance of each group of parallel configurations based on the performance of each operator, thus obtaining M performance metrics corresponding to M groups of parallel configurations. That is, the execution device obtains the performance of a group of parallel configurations by superimposing the performance of each operator in the operator sequence.

[0010] Finally, the execution device selects at least one parallel configuration from the M groups of parallel configurations based on the M performance metrics. For example, the execution device selects the parallel configuration with the highest performance from the M groups of parallel configurations as the recommended parallel configuration.

[0011] In this scheme, by pre-obtaining the basic operator sequences and differential operator sequences corresponding to the model running under multiple parallel modes, it is possible to know the impact of the configuration parameters of each parallel mode on the operators during the model's operation. This allows for the prediction of the operator sequences corresponding to each of the M parallel configurations. Finally, based on the running performance of the operator sequences corresponding to each parallel configuration, the optimal parallel configuration is selected for the model. This effectively improves the efficiency of determining the model configuration while consuming fewer resources, which is beneficial for meeting the ever-changing needs of the scenario.

[0012] In one possible implementation, when obtaining the basic operator sequences of N parallel modes of the first AI model, and the difference operator sequences of each of the N parallel modes, the execution device runs the first AI model using K parallel configurations respectively, and obtains the K first operator sequences executed by any computing device in the AI ​​cluster running the first AI model. Typically, N parallel modes can be pre-set, where each of the K parallel configurations does not include at least one of the N parallel modes of the first AI model, and the at least one parallel mode not included in each of the K parallel configurations is different, where K is less than or equal to N. This is equivalent to disabling one or more of the N parallel modes each time. There are multiple parallel methods, and the parallel method is different each time it is closed; then, the execution device obtains the same second operator sequence from K first operator sequences; it also runs the first AI model using one of the N parallel methods to obtain the third operator sequence corresponding to each of the N parallel methods; then, the execution device calculates the difference operator between the third operator sequence and the second operator sequence corresponding to each of the N parallel methods, that is, the additional communication-related operator introduced under each parallel method due to the introduction of parallelism; then, based on the number of N parallel methods in the target parallel configuration and the difference operator and the second operator sequence of each parallel method, the operator sequence in the computing device corresponding to the target parallel configuration is generated.

[0013] In this scheme, a common first operator sequence, namely the basic operator sequence, is determined in various parallel modes of the first AI model, and a difference operator sequence that is different from other parallel modes in each parallel mode. In this way, for any parallel configuration of the first AI model, the operator sequence corresponding to the parallel configuration can be obtained based on the basic subsequence and the difference operator sequence of various parallel modes in the parallel configuration. This not only improves the accuracy of operator sequence prediction, but also improves efficiency.

[0014] In one possible implementation, K equals N, and the number of at least one parallel method is 1. That is, in one possible scenario, one type of parallelism can be turned off sequentially, resulting in N parallel configurations.

[0015] In one possible implementation, obtaining the identical second operator sequence among the K first operator sequences includes: taking the intersection of the K first operator sequences to obtain the second operator sequence. In this scheme, the basic operator sequence contained in each first operator sequence can be obtained by taking the intersection of each first operator sequence.

[0016] In one possible implementation, the aforementioned calculation of the difference operator between the third operator sequence and the second operator sequence corresponding to each of the N parallel methods includes: subtracting the third operator sequence and the second operator sequence corresponding to each of the N parallel methods to obtain the difference operator corresponding to each parallel method, wherein each of the N parallel methods uses the minimum parallelism of each parallel method.

[0017] In this scheme, the communication-related operators introduced by parallelism under each parallel mode can be obtained by subtraction operation, so that the additional operator overhead under each parallel mode can be quickly known.

[0018] In one possible implementation, the execution device constructs a first AI model based on a second AI model, wherein the type of neural network layers included in the second AI model is the same as the type of neural network layers included in the first AI model, and the number of neural network layers included in the second AI model is greater than the number of neural network layers included in the first AI model.

[0019] Furthermore, when the execution device determines the operator sequence corresponding to each of the K parallel configurations based on the basic operator sequence and the difference operator sequence of each of the N parallel modes, the execution device specifically determines the operator sequence corresponding to each of the K parallel configurations based on the difference between the second AI model and the first AI model, the basic operator sequence, and the difference operator sequence of each of the N parallel modes. All K parallel configurations are parallel configurations of the second AI model.

[0020] That is, the second AI model can be the AI ​​model submitted by the user, and the first AI model is a scaled-down version of the second AI model. The execution device obtains the basic operator sequence and the difference operator sequence by running the first AI model. Since each type of neural network layer included in the second AI model appears in the first AI model, the operator sequence corresponding to each of the K parallel configurations corresponding to the second AI model can be determined based on the differences between the second and first AI models, as well as the basic and difference operator sequences. In other words, after determining the operator sequence corresponding to the first AI model based on a certain parallel configuration among the K parallel configurations, the basic operator sequence, and the difference operator sequence, the operators corresponding to the repeated neural network layers are added to the operator sequence corresponding to the first AI model based on the differences between the first and second AI models (i.e., the repeated neural network layers in the second AI model), thus obtaining the operator sequence corresponding to any set of parallel configurations of the second AI model.

[0021] In other words, although the execution device needs to determine the optimal parallel configuration for the second AI model, it does not obtain the basic operator sequence and the difference operator sequence by running the complex second AI model, but by running the simple first AI model.

[0022] In this approach, by reducing the size of the actual model, the operator sequence of each parallel configuration during model training is tested. This ensures that the accurate correspondence between the parallel method and the operator usage can be determined later, while minimizing resource consumption during operator sequence testing and improving the efficiency of operator sequence testing.

[0023] In one possible implementation, the execution device is to select a portion of the computing devices in the AI ​​cluster to run the first AI model, in order to obtain the basic operator sequence of N parallel modes of the first AI model, and the difference operator sequence of each of the N parallel modes.

[0024] In this solution, model training is performed using a computing cluster with limited computing resources. The basic operator sequence and the difference operator sequence are obtained through testing. Based on the operator sequence obtained from the test, the operator sequence corresponding to various parallel configurations in real application scenarios is predicted. This ensures that the optimal parallel configuration is selected under limited computing resources, meets the actual scenario requirements, and guarantees the feasibility of this solution.

[0025] In one possible implementation, the execution device determines the performance of each operator in the operator sequence corresponding to each group of parallel configurations, and obtains the performance of each group of parallel configurations based on the performance of each operator. This includes: the execution device first obtains an operator performance library, which records the performance of the input data of each operator in multiple operators under different shapes; then, based on the parallelism of the parallel mode in each group of parallel configurations, it determines the shape of the input data of each operator in the operator sequence corresponding to each group of parallel configurations; secondly, the execution device queries the operator performance library to obtain the performance of each operator in the operator sequence corresponding to each group of parallel configurations based on the type of operator and the shape of the input data of the operator; finally, the execution device obtains the performance of each group of parallel configurations by adding the performance of each operator in the operator sequence corresponding to each group of parallel configurations.

[0026] That is, the performance of each operator is obtained by querying the operator performance library based on the operator type and the shape of the operator's input data. The operator performance library is, for example, pre-built by the execution device or obtained from an external source, and stores the performance of various operators running on a specific AI accelerator.

[0027] In one possible implementation, the performance of an operator includes its runtime.

[0028] In one possible implementation, the N parallelism methods include at least two of data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism.

[0029] Secondly, an apparatus for determining AI model configuration is provided, comprising: an acquisition module, configured to acquire a basic operator sequence of N parallel modes of a first AI model, and a difference operator sequence for each of the N parallel modes, wherein the basic operator sequence is a set of operators that are executed by all N parallel modes when running the first AI model, and the difference operator sequence for each parallel mode is the operators other than those in the basic operator sequence when running the first AI model with the minimum amount of parallelism of each parallel mode; a processing module, configured to determine the operator sequence corresponding to each of the M parallel configurations based on the basic operator sequence and the difference operator sequence for each of the N parallel modes; the processing module is further configured to determine the running performance of each operator in the operator sequence corresponding to each parallel configuration, and obtain the running performance of each parallel configuration based on the running performance of each operator, so as to obtain M running performances corresponding to the M parallel configurations; the processing module is further configured to select at least one parallel configuration from the M parallel configurations based on the M running performances.

[0030] In one possible implementation, the acquisition module is specifically used for: running a first AI model using K parallel configurations respectively, acquiring K sequences of first operators executed by any computing device in the AI ​​cluster running the first AI model, wherein each of the K parallel configurations does not include at least one of the N parallel methods of the first AI model, and the at least one parallel method not included in each of the K parallel configurations is different, where K is less than or equal to N; acquiring the same second operator sequence among the K sequences of first operators; running the first AI model using one of the N parallel methods respectively, obtaining a third operator sequence corresponding to each of the N parallel methods; and calculating the difference operator between the third operator sequence and the second operator sequence corresponding to each of the N parallel methods.

[0031] In one possible implementation, K equals N, and the number of at least one parallel method is 1.

[0032] In one possible implementation, when the acquisition module acquires the same second operator sequence among the K first operator sequences, it specifically performs the following: taking the intersection of the K first operator sequences to obtain the second operator sequence.

[0033] In one possible implementation, when the acquisition module calculates the difference operator between the third operator sequence and the second operator sequence corresponding to each of the N parallel methods, it specifically performs the following: subtracts the third operator sequence and the second operator sequence corresponding to each of the N parallel methods to obtain the difference operator corresponding to each parallel method, wherein each of the N parallel methods uses the minimum amount of parallelism for each parallel method.

[0034] In one possible implementation, the processing module is further configured to: construct a first AI model based on the second AI model, wherein the type of neural network layers included in the second AI model is the same as the type of neural network layers included in the first AI model, and the number of neural network layers included in the second AI model is greater than the number of neural network layers included in the first AI model; when determining the operator sequence corresponding to each group of parallel configurations in the M groups of parallel configurations based on the basic operator sequence and the difference operator sequence of each of the N parallel methods, the processing module is specifically configured to: determine the operator sequence corresponding to each group of parallel configurations in the M groups of parallel configurations based on the difference between the second AI model and the first AI model, the basic operator sequence, and the difference operator sequence of each of the N parallel methods, wherein the M groups of parallel configurations are all parallel configurations of the second AI model.

[0035] In one possible implementation, when the acquisition module acquires the basic operator sequence of N parallel modes of the first AI model and the difference operator sequence of each of the N parallel modes, it specifically includes: selecting some computing devices in the AI ​​cluster to run the first AI model in order to acquire the basic operator sequence of N parallel modes of the first AI model and the difference operator sequence of each of the N parallel modes.

[0036] In one possible implementation, when the processing module determines the performance of each operator in the operator sequence corresponding to each group of parallel configurations and obtains the performance of each group of parallel configurations based on the performance of each operator, it specifically performs the following: It acquires an operator performance library, which records the performance of the input data of each operator in multiple operators under different shapes; it determines the shape of the input data of each operator in the operator sequence corresponding to each group of parallel configurations based on the parallelism of the parallel methods in each group of parallel configurations; it queries the operator performance library to obtain the performance of each operator in the operator sequence corresponding to each group of parallel configurations based on the operator type and the shape of the operator's input data; and it obtains the performance of each group of parallel configurations by summing the performance of each operator in the operator sequence corresponding to each group of parallel configurations.

[0037] In one possible implementation, the performance of an operator includes its runtime.

[0038] In one possible implementation, the N parallelism methods include at least two of data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism.

[0039] Thirdly, an apparatus for determining an AI model configuration is provided, comprising: a processor and a memory; the memory is used to store computer instructions, which, when executed by the processor, cause the apparatus for determining the AI ​​model configuration to perform the method described above.

[0040] Fourthly, a computer-readable storage medium is provided that stores instructions which, when executed on a computer, cause the computer to perform the methods of any of the above aspects.

[0041] Fifthly, a computer program product containing instructions is provided, which, when executed on a computer, enable the computer to perform the methods described above.

[0042] In a sixth aspect, a chip system is provided, the chip system including a processor and a communication interface for communicating with a module other than the chip shown, the processor for running computer programs or instructions such that an apparatus on which the chip system is mounted can perform the methods of any of the above aspects.

[0043] In a seventh aspect, a computing device is provided, comprising an AI model configuration determination device of the third aspect or a chip system of the sixth aspect, wherein the AI ​​model configuration determination device or the chip system in the computing device is used to implement the operational steps of the method of any of the above aspects.

[0044] Eighthly, a computing device cluster is provided, comprising at least one computing device, wherein any one computing device is used to run a computer program or instructions, such that the computing device cluster can perform the methods of any of the above aspects. Alternatively, some or all of the computing devices are used together to run a computer program or instructions, such that the computing device cluster can perform the methods of any of the above aspects.

[0045] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0046] Figure 1 is a schematic diagram of a system architecture provided in this application;

[0047] Figure 2 is a flowchart illustrating a method for determining AI model configuration provided in this application;

[0048] Figure 3 is a schematic diagram of a method for determining the operator sequence based on K sets of parallel configurations provided in this application;

[0049] Figure 4 is a schematic diagram of a prediction operator sequence provided in this application;

[0050] Figure 5 is a schematic diagram of a target parallel configuration provided in this application;

[0051] Figure 6 is a flowchart illustrating a method for determining the parallel configuration for model training provided in this application;

[0052] Figure 7 is a flowchart illustrating another method for determining the parallel configuration of model training provided in this application;

[0053] Figure 8 is a schematic diagram of the structure of an AI model configuration determination device provided in this application;

[0054] Figure 9 is a schematic diagram of the structure of a computing device provided in this application;

[0055] Figure 10 is a schematic diagram of the structure of a computing device cluster provided in this application;

[0056] Figure 11 is a schematic diagram of another computing device cluster provided in this application;

[0057] Figure 12 is a schematic diagram of the structure of a computer-readable storage medium provided in this application. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some, and not all, of the embodiments of this application. Those skilled in the art will recognize that, with the emergence of new application scenarios, the technical solutions provided by this application are also applicable to similar technical problems.

[0059] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such descriptions can be used interchangeably where appropriate to allow embodiments to be implemented in a sequence other than that illustrated or described in this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved. The division of units in this application is a logical division. In practical applications, there may be other division methods. For example, multiple units may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interface, and the indirect coupling or communication connection between units may be electrical or other similar forms, none of which are limited in this application. Furthermore, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed among multiple circuit units. Some or all of the units can be selected to achieve the purpose of the solution in this application according to actual needs.

[0060] To facilitate understanding, some technical terms used in this application will be introduced below.

[0061] (1) Data Parallelism

[0062] Data parallelism refers to dividing the training dataset into several blocks and distributing them to different training devices, with each training device running the same model to process the assigned training data.

[0063] In simple terms, in data parallelism, the training dataset is divided into several blocks, and each block is assigned to a training device. Each training device holds a complete model and performs model training based on its assigned block. Furthermore, during model training, after backpropagation, the gradients obtained on different training devices are aggregated to keep the model parameters synchronized across different devices.

[0064] (2) Tensor Parallelism

[0065] Tensor parallelism refers to splitting a single tensor computation across multiple training devices and merging the results from these devices into a target tensor. In essence, tensor parallelism involves dividing a single input data (i.e., a tensor) along a specific dimension into N parts, with each training device performing computation on only one part, and finally obtaining the output data by merging the results from each training device.

[0066] (3) Parallel production lines

[0067] Pipeline parallelism refers to distributing different parts of a model (such as different neural network layers) to different training devices for execution. Since the neural network layers in a model often have data dependencies, different training devices often have a specific order of execution.

[0068] Both pipelined parallelism and tensor parallelism can be considered types of model parallelism, differing only in the dimensions of model partitioning. Pipeline parallelism can be viewed as inter-layer parallelism of the model, while tensor parallelism can be viewed as intra-layer parallelism of the model.

[0069] (4) Sequence parallelism

[0070] Sequence parallelism is a parallel technique used to process long sequences of data. It essentially involves dividing the input sequence into multiple blocks and distributing these blocks across different training devices for processing, thereby reducing the memory requirements of a single training device.

[0071] (5) Experts in parallel

[0072] Expert parallelism is a parallel training technique for Mixture of Experts (MoE) models. Since MoE models typically include multiple expert models, expert parallelism actually refers to distributing multiple expert models across multiple different training devices.

[0073] (6) Operators

[0074] In the field of AI technology, an operator is a basic unit that performs specific mathematical or logical operations.

[0075] Please refer to Figure 1, which is a schematic diagram of a system architecture provided in this application. As shown in Figure 1, in this system architecture, the execution device 10 for performing the method for determining the model configuration provided in this application can be implemented by a single physical host (computing device) or multiple physical hosts (computing device cluster).

[0076] In addition, the system architecture includes a data storage system 11, which stores training data, such as multimedia data like text, audio, image, or video data. Specifically, the data stored in the data storage system 11 can be applied to model training scenarios.

[0077] Optionally, for persistent data storage, the data storage system 11 can be located external to the execution device 10, exchanging data with the execution device 10 via a network. Alternatively, if the execution device 10 is a physical host, the data storage system 11 can also be located internally, such as exchanging data with the processor via a bus. In this case, the data storage system 11 functions as a hard disk. With the data storage system 11, the execution device 10 can determine the model configuration based on the training data stored in the data storage system 11, thereby determining the parallel configuration corresponding to the model.

[0078] Optionally, the execution device 10 may be connected to a model training device 12. The model training device 12 is used to obtain the parallel configuration corresponding to the model from the execution device 10 or the data storage system 11, and to execute the model training task using the obtained parallel configuration. The model training device 12 may be implemented by a single physical host (computing device) or multiple physical hosts (computing device cluster).

[0079] Furthermore, in some possible implementations, the model training device 12 and the execution device 10 can be different devices. That is, the execution device 10 is only responsible for determining the parallel configuration corresponding to the model, while the model training device 12 is only responsible for executing the model training. In other possible implementations, the execution device 10 is, for example, a subset of the model training device 12. That is, after determining the parallel configuration corresponding to the model, the execution device also performs model training as part of the model training device 12.

[0080] Please refer to Figure 2, which is a flowchart illustrating a method for determining AI model configuration provided in this application. As shown in Figure 2, the method for determining AI model configuration provided in this application includes the following steps 201-204.

[0081] Step 201: Obtain the basic operator sequence of N parallel modes of the first AI model, and the difference operator sequence of each of the N parallel modes. The basic operator sequence is the set of operators that will be executed when running the first AI model in the N parallel modes. The difference operator sequence of each parallel mode is the operators other than the operators in the basic operator sequence when running the first AI model with the minimum amount of parallelism in each parallel mode.

[0082] In this application, to determine the one or more parallel configurations that yield the best model training results, the execution device can obtain the basic operator sequences for N parallel modes of the first AI model, as well as the difference operator sequences corresponding to each parallel mode. The basic operator sequence refers to the set of operators that will inevitably be executed when running the first AI model using any of the N parallel modes, where N is an integer greater than 1. The difference operator sequence for each parallel mode is the set of operators associated with that parallel mode, indicating which additional operators are added compared to the basic operator sequence when running the first AI model using that parallel mode.

[0083] For example, the execution device can obtain multiple parallel configurations by disabling different parallel modes, and run the AI ​​model using different parallel configurations, thereby obtaining the sequence of basic operators that the execution device needs to compute during the process of running the AI ​​model with parallel modes disabled in different dimensions.

[0084] For example, considering N parallelism methods, when running the first AI model using any of these methods, operators 1 through 100 will be executed, so operators 1 through 100 can be considered the basic operator sequence. When running the first AI model using the first parallelism method among the N methods, operators 1 through 105 will be executed, so operators 101 through 105 can be considered the difference operator sequence corresponding to the first parallelism method. When running the first AI model using the second parallelism method among the N methods, operators 1 through 100 and operator 106 will be executed, so operator 106 can be considered the difference operator sequence corresponding to the first parallelism method.

[0085] Step 202: Determine the operator sequence corresponding to each group of parallel configurations in the M groups of parallel configurations based on the basic operator sequence and the difference operator sequence of each of the N parallel modes.

[0086] In simple terms, the basic operator sequences for N parallelism modes acquired by the execution device, and the difference operator sequences for each of the N parallelism modes, can be understood as the correspondence between the configuration parameters of each parallelism mode and the operator usage. That is, how the configuration parameters of each parallelism mode affect the use of the operators. In this way, based on the correspondence between the configuration parameters of the parallelism mode and the operator usage, the execution device can determine the operator sequence corresponding to the configuration parameters of any parallelism mode.

[0087] Step 203: Determine the running performance of each operator in the operator sequence corresponding to each group of parallel configurations, and obtain the running performance of each group of parallel configurations based on the running performance of each operator, so as to obtain M running performances corresponding to M groups of parallel configurations.

[0088] Specifically, for each set of parallel configurations corresponding to the operator sequence, the execution device determines the performance of each operator in each operator sequence, and obtains the performance of each set of parallel configurations by summing the performance of each operator in the operator sequence. In this way, by calculating the performance of the operator sequence corresponding to each of the M sets of parallel configurations, the M performance metrics corresponding to the M sets of parallel configurations can be obtained.

[0089] Step 204: Select the parallel configuration of the first AI model from the M groups of parallel configurations based on the M running performances.

[0090] For example, the execution device compares the performance of M running configurations and selects the best-performing parallel configuration from the M groups of parallel configurations as the parallel configuration for the first AI model.

[0091] In this scheme, by pre-obtaining the basic operator sequences and differential operator sequences corresponding to the model running under multiple parallel modes, it is possible to know the impact of the configuration parameters of each parallel mode on the operators during the model's operation. This allows for the prediction of the operator sequences corresponding to each of the M parallel configurations. Finally, based on the running performance of the operator sequences corresponding to each parallel configuration, the optimal parallel configuration is selected for the model. This effectively improves the efficiency of determining the model configuration while consuming fewer resources, which is beneficial for meeting the ever-changing needs of the scenario.

[0092] Optionally, in step 201 above, when obtaining the basic operator sequence of N parallel modes of the first AI model and the difference operator sequence of each of the N parallel modes, the execution device runs the first AI model using K parallel configurations respectively, and obtains the K first operator sequences executed by any computing device in the AI ​​cluster running the first AI model. Typically, N parallel modes can be preset, and each of the K parallel configurations does not include at least one of the N parallel modes of the first AI model, and the at least one parallel mode not included in each of the K parallel configurations is different, where K is less than or equal to N, which is equivalent to turning off one or more of the N parallel modes each time, and the parallel mode turned off each time is different;

[0093] Then, the execution device obtains the same second operator sequence from K first operator sequences; it also runs the first AI model using one of N parallel methods to obtain the third operator sequence corresponding to each of the N parallel methods.

[0094] Secondly, the execution device calculates the difference operator between the third operator sequence and the second operator sequence for each of the N parallel modes, which is the additional communication-related operator introduced in each parallel mode due to the introduction of parallelism.

[0095] Finally, the execution device generates the operator sequence in the computing device corresponding to the target parallel configuration based on the number of N parallel modes in the target parallel configuration and the difference operator and second operator sequence for each parallel mode.

[0096] In this scheme, a common first operator sequence, namely the basic operator sequence, is determined in various parallel modes of the first AI model, and a difference operator sequence that is different from other parallel modes in each parallel mode. In this way, for any parallel configuration of the first AI model, the operator sequence corresponding to the parallel configuration can be obtained based on the basic subsequence and the difference operator sequence of various parallel modes in the parallel configuration. This not only improves the accuracy of operator sequence prediction, but also improves efficiency.

[0097] Optionally, K equals N, and the number of at least one parallel mode is 1. That is, in one possible scenario, one of the parallel modes can be turned off in turn to obtain N parallel configurations.

[0098] Optionally, the execution device obtains the same second operator sequence among the K first operator sequences, including: taking the intersection of the K first operator sequences to obtain the second operator sequence. In this scheme, the basic operator sequence contained in each first operator sequence can be obtained by taking the intersection of each first operator sequence.

[0099] Optionally, the execution device calculates the difference operator between the third operator sequence and the second operator sequence corresponding to each of the N parallel methods. Specifically, this includes: subtracting the third operator sequence and the second operator sequence corresponding to each of the N parallel methods to obtain the difference operator corresponding to each parallel method. Each of the N parallel methods uses the minimum parallelism of each parallel method.

[0100] In this scheme, the communication-related operators introduced by parallelism under each parallel mode can be obtained by subtraction operation, so that the additional operator overhead under each parallel mode can be quickly known.

[0101] Optionally, the above-mentioned parallelism methods include at least two of data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism.

[0102] It should be noted that the execution device can use multiple pre-configured AI accelerators to perform model training, thereby ensuring that the model training process under various parallel configurations can be completed based on multiple AI accelerators. These AI accelerators can be hardware such as Graphics Processing Units (GPUs), Tensor Processing Units (TPUs), or Neural Processing Units (NPUs).

[0103] For example, please refer to Figure 3, which is a schematic diagram of determining the basic operator sequence based on N sets of parallel configurations provided in this application. As shown in Figure 3, for the first AI model, the execution device can set N sets of parallel configurations, namely the first set of parallel configurations to the Nth set of parallel configurations. In this way, by setting the first set of parallel configurations as the parallel configuration during the training of the first AI model, the execution device starts the training of the first AI model and detects the operators used during the training of the first AI model, thereby obtaining the first set of operator sequences corresponding to the first AI model during training. Similarly, by setting the second set of parallel configurations as the parallel configuration during the training of the first AI model, the execution device starts the training of the first AI model and detects the operators used during the training of the first AI model, thereby obtaining the second set of operator sequences corresponding to the first AI model during training. And so on, the execution device independently executes the training process of the first AI model N times with each of the N sets of parallel configurations as the parallel configuration during model training, thereby obtaining the first set of operator sequences to the Kth set of operator sequences, which correspond one-to-one with the first set of parallel configurations to the Nth set of parallel configurations. Furthermore, the basic operator sequence can be obtained by taking the intersection of N parallel configurations.

[0104] In simple terms, to obtain the basic operator sequences and difference operator sequences corresponding to N parallelism methods based on the aforementioned K sets of parallelism configurations, the execution device can first disable different parallelism methods and enable others, thus obtaining N sets of parallelism configurations. That is, for the N parallelism methods, the execution device can set one parallelism method to non-parallelism and the others to parallelism in each set of parallelism configurations, thereby obtaining N sets of parallelism configurations. For example, the execution device can set data parallelism to non-parallelism and sequence parallelism, tensor parallelism, pipeline parallelism, and expert parallelism to parallelism, obtaining one set of parallelism configurations; the execution device can set sequence parallelism to non-parallelism and data parallelism, tensor parallelism, pipeline parallelism, and expert parallelism to parallelism, obtaining another set of parallelism configurations, and so on.

[0105] Then, the execution device runs the N parallel configurations to obtain N sets of operator sequences corresponding to the N parallel configurations. Furthermore, the execution device determines the longest common operator sequence among the N sets of operator sequences as the basic operator sequence. Simply put, by taking the intersection of the N sets of operator sequences, the basic operator sequence corresponding to each of the N parallelization methods can be obtained.

[0106] Secondly, for each of the N parallelism methods, the execution device separately activates one of the N parallelism methods and obtains the operator sequence for each parallelism method individually, thus obtaining N sets of operator sequences. For example, the execution device can set data parallelism to be parallel, while sequence parallelism, tensor parallelism, pipeline parallelism, and expert parallelism are set to non-parallelism, resulting in one set of parallel configurations; the execution device can also set sequence parallelism to be parallel, while data parallelism, tensor parallelism, pipeline parallelism, and expert parallelism are set to non-parallelism, resulting in another parallel configuration, and so on.

[0107] Finally, by comparing a set of operator sequences enabled for each parallel mode with the basic operator sequence, the execution device can obtain the difference operator sequence corresponding to each parallel mode.

[0108] In summary, for N parallelism modes, the execution device first obtains N sets of parallel configurations and N sets of operator sequences by disabling one of the parallelism modes, and then obtains the basic operator sequence by comparing these N sets of operator sequences. Next, the execution device obtains another N sets of parallel configurations and N sets of operator sequences by enabling one of the parallelism modes, and then obtains the difference operator sequence by comparing these N sets of operator sequences with the basic operator sequence.

[0109] Furthermore, after obtaining the basic operator sequences and the difference operator sequences, the execution device can predict the operator sequences corresponding to any set of parallel configurations based on these sequences. That is, for a given set of M parallel configurations, the execution device can predict the operator sequences for model training based on the basic operator sequences of N parallel methods and the difference operator sequences corresponding to each parallel method, thus obtaining M sets of operator sequences. Each of the M sets of operator sequences indicates the operator usage under the corresponding set of parallel configurations, i.e., what type of operator is needed, the number of each type of operator, and the shape of the input data for each operator.

[0110] Please refer to Figure 4, which is a schematic diagram of a prediction operator sequence provided in this application. As shown in Figure 4, based on the basic operator sequence and the difference operator sequence, and combined with any one of the M parallel configurations, a corresponding set of operator sequences can be predicted. Specifically, for the first parallel configuration in the M parallel configurations, the corresponding first set of operator sequences can be predicted; for the second parallel configuration in the M parallel configurations, the corresponding second set of operator sequences can be predicted, and so on, for the Mth parallel configuration in the M parallel configurations, the corresponding Mth set of operator sequences can be predicted. That is, there is a one-to-one correspondence between the M parallel configurations and the M set of operator sequences.

[0111] In simple terms, since the configuration parameters (parallelism) of each parallel mode may affect the operation of the operator, when the execution device determines the operator sequence corresponding to a set of parallel configurations, it can actually comprehensively consider the impact of different parallel modes in the same set of parallel configurations on the operation of the operator, and determine the operation of the operator when multiple parallel modes are combined, thereby obtaining the operator sequence corresponding to the set of parallel configurations.

[0112] It should be noted that, in this application, the number of M parallel configurations actually predicted by the execution device is greater than the N parallel configurations used to determine the correspondence in the above steps. That is, the execution device determines the correspondence between each parallel mode and the operator usage based on a small number of N parallel configurations, and then predicts the operator sequence under a large number of parallel configurations based on the obtained correspondence.

[0113] In practical applications, in order to obtain the optimal set of parallel configurations, the aforementioned M sets of parallel configurations are, for example, obtained by the execution device traversing all combinations of configuration parameters for various parallel modes. That is, the M sets of parallel configurations include various combinations of configuration parameters for multiple parallel modes.

[0114] After obtaining M sets of operator sequences, since each set of operator sequences indicates a specific operator usage, the execution device can determine the corresponding operating performance of each set of operator sequences based on the operating performance of each operator indicated in each set. That is, a set of operator sequences can be understood as a list of operators recording all the required operators. Therefore, by querying the operating performance corresponding to all operators included in the operator list and summing these operating performances, the operating performance corresponding to a set of operator sequences can be obtained. This results in M ​​parallel configurations with operating performance corresponding to the M sets of parallel configurations, each representing a different operation of the first AI model.

[0115] Therefore, based on the performance of each parallel configuration, the execution device can determine at least one parallel configuration from the M parallel configurations. For example, the execution device determines the parallel configuration with the optimal performance from the M parallel configurations.

[0116] The performance of the aforementioned operators includes, for example, their runtime. Therefore, the performance of a set of parallel configurations includes, for example, the total runtime of the operators. Generally, since different operators are usually executed sequentially, the performance of an operator is primarily measured by its runtime. Therefore, when operator performance is measured by its runtime, the execution device can determine the set of parallel configurations with the shortest total runtime among M sets of parallel configurations, so that the subsequent model training process can achieve the highest efficiency.

[0117] Optionally, when determining the performance of a set of operators, the performance of a single operator can be obtained by querying the operator performance library based on the type of operator and the shape of the input data of the operator.

[0118] Generally, given that the type of operator, the shape of the operator's input data, and the model of the AI ​​accelerator running the operator are all determined, the operator's performance is often fixed. Since each operator sequence indicates the type of each operator and the shape of its input data, the execution device can directly query the operator's performance from an operator performance database based on the operator's type and the shape of its input data. This operator performance database, for example, is pre-built by the execution device or obtained from an external source, and it stores the performance of various operators running on a specific AI accelerator.

[0119] For example, the execution device determines the running performance of each operator in the operator sequence corresponding to each group of parallel configurations, and obtains the running performance of each group of parallel configurations based on the running performance of each operator. This includes: the execution device first obtains an operator performance library, which records the running performance of the input data of each operator in multiple operators under different shapes; then, based on the parallelism of the parallel mode in each group of parallel configurations, it determines the shape of the input data of each operator in the operator sequence corresponding to each group of parallel configurations; secondly, the execution device queries the operator performance library to obtain the running performance of each operator in the operator sequence corresponding to each group of parallel configurations based on the type of operator and the shape of the input data of the operator; finally, the execution device obtains the running performance of each group of parallel configurations by adding the running performance of each operator in the operator sequence corresponding to each group of parallel configurations.

[0120] It should be noted that, with a fixed operator type, the same type of operator often exhibits different performance characteristics depending on the shape of the input data. Therefore, in this application, the execution device needs to determine the operator's performance based on the operator type and the shape of the operator's input data to ensure accurate performance results.

[0121] For example, please refer to Figure 5, which is a schematic diagram of determining a target parallel configuration provided by this application. As shown in Figure 5, for the first set of parallel configurations to the Mth set of parallel configurations, the execution device can obtain the running performance corresponding to each set of operator sequences by querying the running performance of each operator in the operator performance library and superimposing the running performance of each operator. That is, the execution device can obtain the running performance 1 corresponding to the first set of operator sequences, the running performance 2 corresponding to the second set of operator sequences, ..., the running performance M corresponding to the Mth set of operator sequences. In this way, by comparing the running performance corresponding to each set of operator sequences, the set of parallel configurations with the best running performance can be selected as the target parallel configuration. For example, in Figure 5, the execution device determines the Mth set of parallel configurations with the best running performance as the parallel configuration of the first AI model.

[0122] Furthermore, for certain specific configuration parameters, such as pipelined parallelism which specifically employs a virtual pipelined parallelism scheme, since the impact of these specific configuration parameters on model training is relatively fixed, the execution device does not need to go through the above steps to obtain the impact of these specific configuration parameters on operator usage. Instead, when calculating the total running performance corresponding to a set of parallel configurations, it directly adjusts the total running performance based on the impact of the specific configuration parameter on operator usage.

[0123] Virtual pipelined parallelism is actually a type of pipelined parallel training method. The difference is that virtual pipelined parallelism distributes as few neural network layers as possible across devices, thereby reducing the latency between devices. For example, suppose a model includes 12 neural network layers, and these 12 neural network layers need to be processed on 4 devices. In a conventional pipelined parallelism scheme, every three adjacent neural network layers are assigned to the same device for processing. In this case, the latency of the later device is often the same as the latency of the earlier device processing three neural network layers. However, the virtual pipelined parallelism scheme distributes the four adjacent neural network layers to four separate devices (e.g., the first neural network layer is assigned to the first device, the second neural network layer to the second device, ..., the fifth neural network layer to the first device, ..., the twelfth neural network layer to the fourth device), thus making the latency of the later device the same as the latency of the earlier device processing one neural network layer.

[0124] The above describes the process of selecting the optimal parallel configuration by predicting the runtime performance of each set of parallel configurations. The following will discuss how to further reduce the resource consumption of the parallel configuration determination process and improve its efficiency.

[0125] Optionally, the execution device constructs a first AI model based on the second AI model. The types of neural network layers included in the second AI model are the same as those included in the first AI model, but the number of neural network layers included in the second AI model is greater than the number of neural network layers included in the first AI model. That is, the first AI model includes all the types of neural network layers included in the second AI model. For example, if the second AI model includes three different types (structures) of neural network layers, then the first AI model will also include at least these three different types of neural network layers. However, the second AI model includes a large number of neural network layers with the same structure (i.e., repetitive neural network layers), while the first AI model does not include too many repetitive neural network layers. Therefore, the number of neural network layers included in the second AI model is greater than the number of neural network layers included in the first AI model. Simply put, the first AI model can be understood as a smaller model obtained by reducing the repetitive neural network layers in the second AI model.

[0126] Furthermore, when the execution device determines the operator sequence corresponding to each group of parallel configurations in the M groups of parallel configurations based on the basic operator sequence and the difference operator sequence of each of the N parallel modes, the execution device specifically determines the operator sequence corresponding to each group of parallel configurations in the M groups of parallel configurations based on the difference between the second AI model and the first AI model, the basic operator sequence, and the difference operator sequence of each of the N parallel modes. All M groups of parallel configurations are parallel configurations of the second AI model.

[0127] That is, the second AI model can be the AI ​​model submitted by the user, and the first AI model is a scaled-down version of the second AI model. The execution device obtains the basic operator sequence and the difference operator sequence by running the first AI model. Since each type of neural network layer included in the second AI model appears in the first AI model, the operator sequence corresponding to each of the M parallel configurations corresponding to the second AI model can be determined based on the differences between the second and first AI models, as well as the basic and difference operator sequences. In other words, after determining the operator sequence corresponding to the first AI model based on a certain parallel configuration in the M parallel configurations, the basic operator sequence, and the difference operator sequence, the operators corresponding to the repeated neural network layers are added to the operator sequence corresponding to the first AI model based on the differences between the first and second AI models (i.e., the repeated neural network layers in the second AI model), thus obtaining the operator sequence corresponding to any set of parallel configurations of the second AI model.

[0128] Since the second AI model in practical applications is usually composed of a large number of repetitive neural network layers, the execution device constructs the first AI model based on the types of neural network layers included in the second AI model. This reduces the number of repetitive neural network layers as much as possible, resulting in a simpler first AI model. Furthermore, because the first AI model includes various types of neural network layers found in the second AI model, the operator sequences determined based on the first AI model are often similar to those determined based on the second AI model. The only difference lies in the number of times the operators are executed in the two sequences (i.e., because the second AI model includes more repetitive neural network layers, some operators in the operator sequences determined based on the first AI model are executed more frequently).

[0129] In this approach, by reducing the size of the actual model, the operator sequence of each parallel configuration during model training is tested. This ensures that the accurate correspondence between the parallel method and the operator usage can be determined later, while minimizing resource consumption during operator sequence testing and improving the efficiency of operator sequence testing.

[0130] It should be noted that when determining the basic operator sequence and the difference operator sequence by testing the operator sequence, the execution device can determine the operator sequence based on the first AI model. However, in step 204 above, when the execution device predicts the operator sequence corresponding to the M sets of parallel configurations, it can predict the operator sequence when training the second AI model based on these M sets of parallel configurations, thereby obtaining the operator sequence in the actual application scenario (i.e., the scenario of training the second AI model).

[0131] Optionally, when determining the basic operator sequence and the difference operator, the execution device may specifically select some computing devices in the AI ​​cluster to run the first AI model, so as to obtain the basic operator sequence of N parallel modes of the first AI model, and the difference operator sequence of each of the N parallel modes.

[0132] For example, the execution device determines the basic operator sequence and difference operators based on the first computing cluster. However, when predicting the operator sequence when performing model training based on M sets of parallel configurations, the execution device can predict the operator sequence when performing model training on the second computing cluster based on the M sets of parallel configurations. That is, the M sets of parallel configurations are actually applied to the scenario of performing model training on the second computing cluster.

[0133] For example, the first computing cluster includes 50 computing devices, while the second computing cluster includes 2,000, 3,000, or 5,000 computing devices.

[0134] In this solution, model training is performed using a computing cluster with limited computing resources. The basic operator sequence and the difference operator sequence are obtained through testing. Based on the operator sequence obtained from the test, the operator sequence corresponding to various parallel configurations in real application scenarios is predicted. This ensures that the optimal parallel configuration is selected under limited computing resources, meets the actual scenario requirements, and guarantees the feasibility of this solution.

[0135] To facilitate understanding, the following will provide a detailed explanation of the execution process of the model configuration determination method provided in this application in practical applications, using specific examples.

[0136] It's important to note that in real-world applications, the process of determining model configuration is often constrained by resources. Therefore, the computational resources used to determine the model configuration are often less than those actually used for model training. For example, the model configuration might be determined using a first computing cluster with a small number of AI accelerators, while the actual model training might be performed using a second computing cluster with a large number of AI accelerators. The first computing cluster can be understood as the one used for recommending the model configuration, while the second computing cluster can be understood as the one used for the actual model training.

[0137] Please refer to Figures 6 and 7. Figure 6 is a flowchart illustrating one method for determining the parallel configuration of model training according to this application; Figure 7 is a flowchart illustrating another method for determining the parallel configuration of model training according to this application. As shown in Figure 6, the process for determining the parallel configuration of model training includes the following steps 601-608.

[0138] Step 601: Based on the first computing cluster, determine 2N groups of parallel configurations.

[0139] Specifically, for the first computing cluster used to perform the model configuration determination process, the execution device first disables different parallel modes and enables other parallel modes, thereby obtaining N sets of parallel configurations for determining the basic operator sequence. The execution device then enables one of the N parallel modes individually and obtains the operator sequence when each parallel mode is enabled individually, thereby obtaining N sets of operator sequences for determining the difference operator sequence.

[0140] It should be noted that the execution device in this application may be, for example, a first computing cluster (i.e., the execution device is a computing device in the first computing cluster), or it may be another computing device independent of the first computing cluster.

[0141] Step 602: Reduce the size of the second AI model to obtain the first AI model.

[0142] To improve the efficiency of determining model configuration, the execution device can reduce the size of the second AI model that actually needs to determine parallel configuration, thereby obtaining a smaller first AI model. Specifically, the execution device can reduce the number of repeated neural network layers in the second AI model, resulting in a first AI model with fewer neural network layers. Furthermore, when reducing the size of the second AI model, the execution device needs to ensure that at least one neural network layer of each type is retained, thereby ensuring that the number of types of neural network layers in the first AI model is the same as the number of types of neural network layers in the second AI model.

[0143] In addition to pruning the structure of the second AI model to obtain the first AI model, the length of the training data corresponding to the first AI model can also be pruned to speed up the training process on the first AI model.

[0144] Step 603: On the first computing cluster, the training of the first AI model is performed based on 2N sets of parallel configurations, and the 2N sets of operator sequences are obtained by profiling.

[0145] Specifically, as shown in Figure 7, the first computing cluster can sequentially execute the training of the first AI model 2N times, with each training iteration based on one of the 2N parallel configurations. This allows for the separate execution of the first AI model training across the 2N parallel configurations. Furthermore, each time the first computing cluster executes the training of the first AI model, profiling can be performed to obtain performance test results. These profiling results include the various operators used by the first computing cluster, i.e., a complete sequence of operators. Therefore, based on the performance test results of each execution of the first AI model training by the first computing cluster, 2N operator sequences can be obtained.

[0146] Step 604: Analyze the 2N sets of operator sequences to obtain the basic operator sequence and the difference operator sequence.

[0147] Specifically, by taking the intersection of the first N sets of operator sequences, the basic operator sequences corresponding to each of the N parallelization methods can be obtained. Then, the difference operator sequences are obtained by comparing the last N sets of operator sequences with the basic operator sequences. Therefore, the N sets of operator sequences used by the execution device to obtain the basic operator sequences and the N sets of operator sequences used to obtain the difference operator sequences are equivalent to the K sets of operator sequences mentioned above.

[0148] Step 605: Based on the second computing cluster, determine the parallel configuration of M groups.

[0149] Since the first and second computing clusters contain different numbers of AI accelerators, the execution device needs to determine M parallel configurations based on the number of AI accelerators in the second computing cluster. The M parallel configurations, for example, represent all parallel configurations that the execution device can run on the second computing cluster; that is, the M parallel configurations include all combinations of parallelism methods adapted to the second computing cluster. Each parallel configuration in the M parallel configurations includes configuration parameters for multiple parallelism methods, and the product of the configuration parameters for DP, TP, PP, and CP among these methods is the number of AI accelerators in the second computing cluster. For example, if the second computing cluster contains 5000 AI accelerators, and each parallel configuration includes 4 parallelism methods, then the product of the configuration parameters for the 4 parallelism methods in each parallel configuration is 5000.

[0150] Step 606: Based on the basic operator sequence and the difference operator sequence, predict the operator sequence when training the second AI model on the second computing cluster based on M sets of parallel configurations, and obtain M sets of operator sequences.

[0151] Step 607: Obtain the running performance of each operator in the M sets of operator sequences from the operator performance library.

[0152] Given the M sets of operator sequences, the execution device can obtain the running performance of each operator in the M sets of operator sequences from the operator performance library based on the type of operator included in each set of operator sequences and the shape of the input data of the operator.

[0153] Furthermore, if the performance of certain operators cannot be found in the operator performance library, the execution device can analyze the performance test results collected through profiling to obtain the performance of these operators and add their performance to the operator performance library.

[0154] Step 608: Based on the M sets of operator sequences and the operating performance of the operators, determine the target parallel configuration with the best performance among the M sets of parallel configurations.

[0155] Specifically, based on the M sets of operator sequences and their performance, the execution device can search for the optimal parallel configuration among the M sets of parallel configurations to obtain the target parallel configuration with the best performance. For example, the execution device determines the performance of the operators indicated in each set of operator sequences and sums these performances to obtain the total performance corresponding to each set of parallel configurations. Furthermore, when the total performance is the total training time of the second computing cluster, the execution device can also determine the latency (bubble) of each AI accelerator based on the pipeline parallel configuration parameters when calculating the total performance corresponding to each set of parallel configurations, thereby predicting the total performance. In this way, the execution device can select the set of parallel configurations with the best total performance from the M sets of parallel configurations as the target parallel configuration.

[0156] The method provided in this application has been described in detail above. Next, the device provided in this application for performing the above method will be described.

[0157] Please refer to Figure 8, which is a schematic diagram of the structure of an AI model configuration determination device provided in this application. As shown in Figure 8, the AI ​​model configuration determination device includes: an acquisition module 801, used to acquire the basic operator sequence of N parallel modes of a first AI model, and the difference operator sequence of each of the N parallel modes, wherein the basic operator sequence is the set of operators that will be executed when running the first AI model in the N parallel modes, and the difference operator sequence of each parallel mode is the operators other than the operators in the basic operator sequence when running the first AI model with the minimum parallelism of each parallel mode; a processing module 802, used to determine the operator sequence corresponding to each of the M parallel configurations based on the basic operator sequence and the difference operator sequence of each of the N parallel modes; the processing module 802 is also used to determine the running performance of each operator in the operator sequence corresponding to each parallel configuration, and obtain the running performance of each parallel configuration based on the running performance of each operator, so as to obtain M running performances corresponding to the M parallel configurations; the processing module 802 is also used to select at least one parallel configuration from the M parallel configurations based on the M running performances.

[0158] In one possible implementation, the acquisition module 801 is specifically used for: running the first AI model using K parallel configurations respectively, acquiring K first operator sequences executed by any computing device in the AI ​​cluster running the first AI model, wherein each of the K parallel configurations does not include at least one of the N parallel methods of the first AI model, and the at least one parallel method not included in each of the K parallel configurations is different, where K is less than or equal to N; acquiring the same second operator sequence among the K first operator sequences; running the first AI model using one of the N parallel methods respectively, obtaining the third operator sequence corresponding to each of the N parallel methods; and calculating the difference operator between the third operator sequence corresponding to each of the N parallel methods and the second operator sequence.

[0159] In one possible implementation, K equals N, and the number of at least one parallel method is 1.

[0160] In one possible implementation, when the acquisition module 801 acquires the same second operator sequence among the K first operator sequences, it is specifically used to: take the intersection of the K first operator sequences to obtain the second operator sequence.

[0161] In one possible implementation, when the acquisition module 801 calculates the difference operator between the third operator sequence and the second operator sequence corresponding to each of the N parallel methods, it is specifically used to: subtract the third operator sequence and the second operator sequence corresponding to each of the N parallel methods to obtain the difference operator corresponding to each parallel method, wherein each of the N parallel methods uses the minimum amount of parallelism for each parallel method.

[0162] In one possible implementation, the processing module 802 is further configured to: construct a first AI model based on the second AI model, wherein the type of neural network layers included in the second AI model is the same as the type of neural network layers included in the first AI model, and the number of neural network layers included in the second AI model is greater than the number of neural network layers included in the first AI model; when determining the operator sequence corresponding to each group of parallel configurations in the M groups of parallel configurations based on the basic operator sequence and the difference operator sequence of each of the N parallel methods, the processing module 802 is specifically configured to: determine the operator sequence corresponding to each group of parallel configurations in the M groups of parallel configurations based on the difference between the second AI model and the first AI model, the basic operator sequence, and the difference operator sequence of each of the N parallel methods, wherein the M groups of parallel configurations are all parallel configurations of the second AI model.

[0163] In one possible implementation, when the acquisition module 801 acquires the basic operator sequence of N parallel modes of the first AI model and the difference operator sequence of each of the N parallel modes, it specifically includes: selecting some computing devices in the AI ​​cluster to run the first AI model in order to acquire the basic operator sequence of N parallel modes of the first AI model and the difference operator sequence of each of the N parallel modes.

[0164] In one possible implementation, when the processing module 802 determines the running performance of each operator in the operator sequence corresponding to each group of parallel configurations and obtains the running performance of each group of parallel configurations based on the running performance of each operator, it specifically performs the following: acquiring an operator performance library, which records the running performance of the input data of each operator in multiple operators under different shapes; determining the shape of the input data of each operator in the operator sequence corresponding to each group of parallel configurations based on the parallelism of the parallel mode in each group of parallel configurations; querying the operator performance of each operator in the operator sequence corresponding to each group of parallel configurations in the operator performance library based on the operator type and the shape of the operator's input data; and obtaining the running performance of each group of parallel configurations by adding the running performance of each operator in the operator sequence corresponding to each group of parallel configurations.

[0165] In one possible implementation, the performance of an operator includes its runtime.

[0166] In one possible implementation, the N parallelism methods include at least two of data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism.

[0167] Both the acquisition module 801 and the processing module 802 can be implemented in software or in hardware. For example, the implementation of the processing module 802 will be described below. Similarly, the implementation of the acquisition module 801 can be referenced from the implementation of the processing module 802.

[0168] As an example of a software functional unit, processing module 802 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, processing module 802 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0169] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0170] As an example of a hardware functional unit, the processing module 802 may include at least one computing device, such as a server. Alternatively, the processing module 802 may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0171] The processing module 802 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the processing module 802 can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the processing module 802 can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0172] Please refer to Figure 9, which is a schematic diagram of the structure of a computing device provided in this application. The computing device 900 shown in Figure 9 can be used to execute the data cleaning method provided in this embodiment. As shown in Figure 9, the computing device 900 includes: a bus 902, a processor 904, a memory 906, and a communication interface 908. The processor 904, the memory 906, and the communication interface 908 communicate with each other via the bus 902. The computing device 900 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 900.

[0173] Bus 902 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 9, but this does not imply that there is only one bus or one type of bus. Bus 902 can include pathways for transmitting information between various components of computing device 900 (e.g., memory 906, processor 904, communication interface 908).

[0174] Processor 904 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0175] The memory 906 may include volatile memory, such as random access memory (RAM). The processor 904 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0176] The memory 906 stores executable program code, and the processor 904 executes this executable program code to implement the functions of the aforementioned acquisition module and processing module, thereby realizing the data cleaning method described above. That is, the memory 906 stores instructions for executing the data cleaning method.

[0177] The communication interface 908 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 900 and other devices or communication networks.

[0178] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0179] Please refer to Figure 10, which is a schematic diagram of a computing device cluster provided in this application. As shown in Figure 10, the computing device cluster includes at least one computing device 900. The memory 906 of one or more computing devices 900 in the computing device cluster may store the same instructions for executing data cleaning methods.

[0180] In some possible implementations, the memory 906 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing the data cleaning method. In other words, a combination of one or more computing devices 900 can jointly execute the instructions for executing the data cleaning method.

[0181] It should be noted that the memories 906 in different computing devices 900 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the data cleaning method apparatus. That is, the instructions stored in the memories 906 of different computing devices 900 can implement the functions of one or more of the aforementioned acquisition and processing modules.

[0182] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 11 illustrates one possible implementation. Figure 11 is also a schematic diagram of another computing device cluster structure provided in this application. As shown in Figure 11, in computing device cluster 1100, two computing devices 900A and 900B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 906 in computing device 900A stores instructions for executing the functions of the acquisition module. Simultaneously, the memory 906 in computing device 900B stores instructions for executing the functions of the processing module.

[0183] It should be understood that the functions of computing device 900A shown in Figure 11 can also be performed by multiple computing devices 900. Similarly, the functions of computing device 900B can also be performed by multiple computing devices 900.

[0184] This application also provides a chip comprising a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in a storage unit to cause the chip within the electronic device to perform the methods described in the above embodiments. Optionally, the storage unit may be an in-chip storage unit, such as a register or cache. Alternatively, the storage unit may be an external storage unit located within a wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0185] Referring to Figure 12, which is a schematic diagram of the structure of a computer-readable storage medium provided in this application. This application also provides a computer-readable storage medium in which, in some embodiments, the method disclosed in Figure 2 can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0186] Figure 12 schematically illustrates a conceptual partial view of an example computer-readable storage medium arranged according to at least some of the embodiments shown herein, the example computer-readable storage medium including a computer program for executing computer processes on a computing device.

[0187] In one embodiment, the computer-readable storage medium 1200 is provided using a signal bearer medium 1201. The signal bearer medium 1201 may include one or more program instructions 1202 that, when executed by one or more processors, can provide the functions or parts thereof described above with reference to FIG2.

[0188] In some examples, signal carrying medium 1201 may include computer-readable medium 1203, such as, but not limited to, hard disk drive, compact disc (CD), digital video disc (DVD), digital magnetic tape, memory, ROM or RAM, etc.

[0189] In some embodiments, the signal-bearing medium 1201 may comprise a computer-recordable medium 1204, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, etc. In some embodiments, the signal-bearing medium 1201 may comprise a communication medium 1205, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, etc.). Therefore, for example, the signal-bearing medium 1201 may be transmitted by a wireless communication medium 1205 (e.g., a wireless communication medium conforming to the IEEE 1102.X standard or other transmission protocols).

[0190] One or more program instructions 1202 may be, for example, computer-executable instructions or logical implementation instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to one or more program instructions 1202 conveyed to the computing device via a computer-readable medium 1203, a computer-recordable medium 1204, and / or a communication medium 1205.

[0191] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0192] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0193] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0194] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, computer instructions can be transferred from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for determining the configuration of an artificial intelligence (AI) model, characterized in that, include: Obtain the basic operator sequence of N parallel methods of the first AI model, and the difference operator sequence of each of the N parallel methods. The basic operator sequence is the set of operators that will be executed when the first AI model is run by the N parallel methods. The difference operator sequence of each parallel method is the operators other than the operators in the basic operator sequence when the first AI model is run with the minimum amount of parallelism of each parallel method. Based on the basic operator sequence and the difference operator sequence for each of the N parallel methods, determine the operator sequence corresponding to each of the M parallel configurations; Determine the running performance of each operator in the operator sequence corresponding to each group of parallel configurations, and obtain the running performance of each group of parallel configurations based on the running performance of each operator, so as to obtain M running performances corresponding to the M groups of parallel configurations; Based on the M performance metrics, at least one parallel configuration is selected from the M groups of parallel configurations.

2. The method according to claim 1, characterized in that, The acquisition of the basic operator sequence for N parallel modes of the first AI model, and the difference operator sequence for each of the N parallel modes, includes: The first AI model is run using K parallel configurations. M sequences of first operators are obtained from any computing device in the AI ​​cluster running the first AI model. Each of the K parallel configurations does not include at least one of the N parallel methods of the first AI model, and the at least one parallel method not included in each of the K parallel configurations is different, where K is less than or equal to N. Obtain the identical second operator sequences among the K first operator sequences; The first AI model is run using one of the N parallel methods to obtain the third operator sequence corresponding to each of the N parallel methods. Calculate the difference operator between the third operator sequence and the second operator sequence for each of the N parallel methods.

3. The method according to claim 2, characterized in that, K is equal to N, and the number of the at least one parallel method is 1.

4. The method according to claim 1 or 2, characterized in that, Obtaining the identical second operator sequence among the K first operator sequences includes: The second operator sequence is obtained by taking the intersection of the K first operator sequences.

5. The method according to any one of claims 1-3, characterized in that, The operator for calculating the difference between the third operator sequence and the second operator sequence for each of the N parallelization methods includes: The difference operator corresponding to each of the N parallel methods is obtained by subtracting the third operator sequence from the second operator sequence. Each of the N parallel methods uses the minimum amount of parallelism required for each method.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: Based on the second AI model, the first AI model is constructed. The types of neural network layers included in the second AI model are the same as those included in the first AI model, and the number of neural network layers included in the second AI model is greater than the number of neural network layers included in the first AI model. The step of determining the operator sequence corresponding to each of the M parallel configurations based on the basic operator sequence and the difference operator sequence for each of the N parallel modes includes: Based on the differences between the second AI model and the first AI model, the basic operator sequence, and the difference operator sequence for each of the N parallel methods, the operator sequence corresponding to each of the M parallel configurations is determined, where the M parallel configurations are all parallel configurations of the second AI model.

7. The method according to any one of claims 2-4, characterized in that, The acquisition of the basic operator sequence for N parallel modes of the first AI model, and the difference operator sequence for each of the N parallel modes, includes: Select some computing devices in the AI ​​cluster to run the first AI model, so as to obtain the basic operator sequence of N parallel modes of the first AI model, and the difference operator sequence of each of the N parallel modes.

8. The method according to any one of claims 1-7, characterized in that, The step of determining the running performance of each operator in the operator sequence corresponding to each group of parallel configurations, and obtaining the running performance of each group of parallel configurations based on the running performance of each operator, includes: Obtain an operator performance library, which records the performance of each operator under different shapes based on the input data of multiple operators; Based on the amount of parallelism in each parallel configuration, determine the shape of the input data for each operator in the operator sequence corresponding to each parallel configuration; Based on the operator type and the shape of the operator's input data, the runtime performance of each operator in the operator sequence corresponding to each set of parallel configurations is obtained by querying the operator performance library. The running performance of each set of parallel configurations is obtained by summing the running performance of each operator in the operator sequence corresponding to each set of parallel configurations.

9. The method according to any one of claims 1-8, characterized in that, The performance of the operator includes the operator's execution time.

10. The method according to any one of claims 1-9, characterized in that, The N parallelism methods include at least two of the following: data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism.

11. A device for determining the configuration of an artificial intelligence (AI) model, characterized in that, include: The acquisition module is used to acquire the basic operator sequence of N parallel modes of the first AI model, and the difference operator sequence of each of the N parallel modes. The basic operator sequence is the set of operators that will be executed when the first AI model is run by the N parallel modes. The difference operator sequence of each parallel mode is the operators other than the operators in the basic operator sequence when the first AI model is run with the minimum amount of parallelism of each parallel mode. The processing module is used to determine the operator sequence corresponding to each group of parallel configurations in the M groups of parallel configurations based on the basic operator sequence and the difference operator sequence of each of the N parallel modes; The processing module is further configured to determine the running performance of each operator in the operator sequence corresponding to each group of parallel configurations, and obtain the running performance of each group of parallel configurations based on the running performance of each operator, so as to obtain M running performances corresponding to the M groups of parallel configurations. The processing module is further configured to select at least one set of parallel configurations from the M sets of parallel configurations based on the M running performances.

12. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the operational steps of the method as described in any one of claims 1 to 10.

13. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed by the computer, cause the computer to perform the method according to any one of claims 1 to 10.

14. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 10.

Images