Data processing method and apparatus, and computer device, computer-readable storage medium and computer program product

Abstract

Disclosed in the present application are a data processing method and apparatus, and a computer device, a computer-readable storage medium and a computer program product. The method comprises: acquiring M initial chip groups; on the basis of the number of learnable parameters that corresponds to an individual network layer in an initial model, performing grouping on chips in each initial chip group, so as to obtain P pipeline parallel groups; and on the basis of a load balancing condition, an average computing power coefficient of the P pipeline parallel groups, the number of network layers in the initial model, and group computing power coefficients respectively corresponding to the P pipeline parallel groups, performing grouping and combination on the P pipeline parallel groups, so as to obtain D data parallel groups and sample input quantities respectively corresponding to the D data parallel groups, wherein input samples indicated by the D sample input quantities are used for performing model training in parallel on initial models configured on corresponding data parallel groups.

Description

Data processing methods, apparatus, computer equipment, computer-readable storage media and computer program products

[0001] Cross-reference to related applications

[0002] This application is based on and claims priority to Chinese Patent Application No. 202411598030.7, filed on November 11, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0004] With the rapid development of artificial intelligence technology, the parameter scale of large-scale language models is growing daily, ranging from millions to trillions. In existing model training, due to the massive number of parameters, the demands on computing resources and time are enormous, typically requiring chip clusters for training. Current technologies generally employ homogeneous chips for data parallelism or pipelined parallelism for distributed training. Data parallelism involves configuring the same initial model across multiple data parallel groups, each processing its own input samples independently, and then aggregating all model parameters. Pipeline parallelism involves distributing the model's network layers across multiple pipelined parallel groups, each responsible for processing a portion of the model's network layers.

[0005] Building a large-scale cluster of 10,000 chips using a single type of chip presents significant challenges due to the difficulty in obtaining a sufficient number of homogeneous chips. When using a hybrid deployment of different types of chips, employing the same parallel strategy as with homogeneous chips, the significant differences in computing power (comprehensive computing power, storage resources, and communication resources) among the different chips lead to uneven chip load. Chips with stronger computing power are not effectively utilized, while chips with weaker computing power severely hinder the training performance of the chip cluster, resulting in low computing power utilization and low training efficiency. Summary of the Invention

[0006] This application provides a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product, which can improve the utilization rate of computing resources in hybrid-deployed chip clusters and improve the training efficiency of models.

[0007] This application provides a data processing method applied to a computer device, including:

[0008] Obtain M initial chipsets; M is a positive integer, all chips in an initial chipset are the same model, and the chip models of each initial chipset are different from each other.

[0009] Based on the number of learning parameters corresponding to a single network layer in the initial model and the chip models of the M initial chip groups, the chips in each initial chip group are grouped to obtain P pipeline parallel groups; P is a positive integer, the chip models in a pipeline parallel group are all the same, and the computing memory corresponding to each pipeline parallel group is greater than or equal to the number of learning parameters.

[0010] Based on load balancing conditions, the average computing power coefficient of P pipeline parallel groups, the number of network layers of the initial model, and the group computing power coefficients corresponding to the P pipeline parallel groups, the P pipeline parallel groups are grouped and combined to obtain D data parallel groups and the sample input amount corresponding to each of the D data parallel groups; D is a positive integer less than P, and the computing memory corresponding to each data parallel group is greater than or equal to the total number of learning parameters of the initial model; the input samples indicated by the sample input amounts corresponding to the D data parallel groups are used to train the initial model configured on the corresponding data parallel groups in parallel.

[0011] This application provides a data processing apparatus, including:

[0012] The chip acquisition module is used to acquire M initial chipsets; M is a positive integer, all chips in an initial chipset are the same model, and the chip models of each initial chipset are different from each other.

[0013] The pipeline parallel grouping module is used to group the chips in each initial chipset according to the number of learning parameters corresponding to a single network layer in the initial model and the chip models of M initial chipsets, to obtain P pipeline parallel groups; P is a positive integer, the chip models in a pipeline parallel group are all the same, and the computing memory corresponding to each pipeline parallel group is greater than or equal to the number of learning parameters.

[0014] The data parallel grouping module is used to group and combine the P pipeline parallel groups based on load balancing conditions, the average computing power coefficient of the P pipeline parallel groups, the number of network layers of the initial model, and the group computing power coefficients corresponding to the P pipeline parallel groups, to obtain D data parallel groups and the sample input amount corresponding to each of the D data parallel groups; D is a positive integer less than P, and the computing memory corresponding to each data parallel group is greater than or equal to the total number of learning parameters of the initial model; the input samples indicated by the sample input amounts corresponding to the D data parallel groups are used to train the initial model configured on the corresponding data parallel groups in parallel.

[0015] This application provides a computer device, including: a processor, a memory, and a network interface;

[0016] The processor is connected to a memory and a network interface. The network interface is used to provide data communication functions, and the memory is used to store computer programs. When the computer program is executed by the processor, the computer device performs the method provided in the embodiments of this application.

[0017] This application provides a computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor, so that a computer device having the processor performs the method provided in this application.

[0018] This application provides a computer program product including a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the method provided in this application.

[0019] In this embodiment, all chips are grouped by chip model to obtain initial chip groups consisting entirely of the same chip model. Each initial chip group has a different chip model. Based on the number of learning parameters corresponding to a single network layer in the initial model and the chip models of the M initial chip groups, the chips within each initial chip group are further grouped to obtain P pipelined parallel groups. All chips within a pipelined parallel group are of the same model, and the computational memory corresponding to each pipelined parallel group is greater than or equal to the number of learning parameters. This avoids computational waste caused by "performance mismatch" when different chip models are mixed, and memory constraints ensure that each pipelined parallel group can independently handle its corresponding training task, preventing model deployment failure or training crashes due to insufficient hardware resources. Tensor parallel training of the initial model's network layers is performed using chips of the same model. Based on load balancing conditions, the average computational power coefficient of the P pipelined parallel groups, the number of network layers in the initial model, and the group computational power coefficients corresponding to the P pipelined parallel groups, the P pipelined parallel groups are grouped and combined to obtain D data parallel groups and the sample input amounts corresponding to each of the D data parallel groups. The D data parallel groups each correspond to a specific input sample, which is used to train the initial model configured on the corresponding data parallel group in parallel. The D data parallel groups satisfy a load balancing condition. Data parallelism refers to configuring the same initial model on multiple data parallel groups, with each group independently processing its own input samples, and finally aggregating all model parameters. This application embodiment uses pipelined parallel groups composed of chips of the same model to determine the average computing power coefficient and group computing power coefficient of each pipelined parallel group. Based on the idea of ​​computing power equivalence, the pipelined parallel groups are combined to obtain data parallel groups, and a computing power equivalent task load is allocated to each data parallel group. While ensuring that the computing memory corresponding to each data parallel group is greater than or equal to the total number of learning parameters of the initial model, the task load of the initial model is split by the data parallel groups in the dimension of sample input size to determine the sample input amount corresponding to each data parallel group, so that the load of each data parallel group is balanced, that is, global load balancing is achieved. This ensures the stability of distributed training, avoids training interruption or efficiency loss caused by uneven load, thereby improving the utilization rate of computing resources of hybrid deployed chips and improving the training efficiency of the model. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 is a schematic diagram of a network architecture provided in an embodiment of this application;

[0022] Figure 2 is a schematic diagram of a data processing scenario provided in an embodiment of this application;

[0023] Figure 3 is a schematic diagram of a data processing scenario provided in an embodiment of this application;

[0024] Figure 4 is a schematic flowchart of a data processing method provided in an embodiment of this application;

[0025] Figure 5 is a schematic flowchart of a data processing method provided in an embodiment of this application;

[0026] Figure 6 is a schematic diagram of a data processing scenario provided in an embodiment of this application;

[0027] Figure 7 is a schematic diagram of a data processing scenario provided in an embodiment of this application;

[0028] Figure 8 is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0029] Figure 9 is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0031] Please refer to Figure 1, which is a schematic diagram of a network architecture provided in an embodiment of this application. As shown in Figure 1, the network architecture may include a service server 100 and N training nodes. The N training nodes may include training node 10a, training node 10b, ..., training node 10n. Each training node may have a communication connection with the service server 100. For example, training node 10a may have a communication connection with the service server 100, and training node 10b may have a communication connection with the service server 100. The above communication connection is not limited to a specific connection method. It may be directly or indirectly connected via wired communication, or directly or indirectly connected via wireless communication, or through other methods. This application does not impose any restrictions on this method.

[0032] The training nodes can include: smartphones, tablets, laptops, desktop computers, smart voice interaction devices, smart home appliances (e.g., smart TVs), wearable devices, in-vehicle terminals, aircraft, and other smart terminals with data processing capabilities. In-vehicle terminals can be devices used in smart transportation scenarios and assisted driving scenarios. It should be understood that the training nodes shown in Figure 1 can be equipped with application clients that have data processing capabilities. When the application client runs on each training node, it can interact with the business server 100 shown in Figure 1.

[0033] The application clients may include: in-vehicle clients, smart home clients, entertainment clients (e.g., game clients), multimedia clients (e.g., video clients), social clients, and information clients (e.g., news clients). In this embodiment, the application client may be integrated into a client (e.g., a social client) or may be a standalone client (e.g., a news client). This embodiment does not limit the type of application client.

[0034] As shown in Figure 1, the business server 100 and N training nodes (which may include training node 10a, training node 10b, ..., training node 10n) can collaborate to perform distributed learning of the initial model. The business server 100 can be a server for aggregating model parameters, and the training nodes can be servers for model training. Each training node can be deployed with one or more chips of the same model (also referred to as graphics card chips, GPUs (Graphics Processing Units), or graphics processors). The chip models of different training nodes can be the same or different. The chip model refers to the chip's model number, used to distinguish the chip's product type and performance level. Different training nodes can have different machine models, and different machine models also mean that the training nodes are equipped with different chip models.

[0035] The business server 100 can define a training node as a pipeline parallel group. A pipeline parallel group can be used to train several network layers of the initial model. Taking pipeline parallel group 1 corresponding to a training node as an example, which includes chip 1 and chip 2 of the same chip model and is used to train two network layers of the initial model, chip 1 and chip 2 can each be used to calculate half of the learning parameters in the two network layers. That is, the tensor parallelism of pipeline parallel group 1 is 2. Tensor parallelism, also known as intra-layer parallelism, refers to dividing the parameters within each network layer of the model and distributing the training tasks of different parameters of the same network layer to multiple chips for completion. Pipeline parallelism, also known as inter-layer parallelism, refers to distributing one or more network layers (usually consecutive layers) of the model to multiple pipeline parallel groups, with each pipeline parallel group responsible for processing a portion of the network layers of the model.

[0036] The business server 100 can group and combine pipeline parallel groups based on the average computing power coefficient and the group computing power coefficient, resulting in D data parallel groups for model training. The data parallelism (Data Parallel Group Size) is D. Data parallelism refers to configuring the same initial model across multiple data parallel groups, with each group training its model using different batches of data. Each data parallel group independently calculates its loss function and gradients, then aggregates the gradients through some form of communication to update the model parameters of all data parallel groups. Each data parallel group can be configured with an initial model, and the initial model deployed in each group can be identical. The group computing power coefficient is determined comprehensively based on the chip's business computing power parameters, storage parameters, bandwidth parameters, and communication parameters. The average computing power coefficient is the average of the group computing power coefficients of each pipeline parallel group.

[0037] The business server 100 can allocate different sample input amounts to each data parallel group. Each data parallel group can train its locally configured initial model based on the allocated input samples. After completing forward and backward propagation calculations, the D data parallel groups can return the calculated sub-model parameters to the business server 100. The learned parameters can include the total number of model parameters, activation parameters, and gradient parameters. The business server 100 aggregates the sub-model parameters to obtain aggregated model parameters, and then updates the initial model configured in each data parallel group based on the aggregated model parameters for the next batch of training. When the initial model in the business server meets the model convergence condition, the initial model that meets the model convergence condition is determined as the target model.

[0038] This application embodiment utilizes pipelined parallel groups composed of chips of the same model. It determines the average computing power coefficient and group computing power coefficient of each pipelined parallel group. Based on the idea of ​​computing power equivalence, the pipelined parallel groups are grouped and combined to obtain data parallel groups. Each data parallel group is allocated a task load with equivalent computing power. While ensuring that the computing memory corresponding to each data parallel group is greater than or equal to the total number of learning parameters of the initial model, the task load of the initial model is split according to the sample input size of the data parallel groups. This determines the sample input volume corresponding to each data parallel group, ensuring load balancing for each data parallel group. This improves the utilization rate of computing resources of the hybrid deployed chips and enhances model training efficiency. Each data parallel group can independently process its own input samples, performing model training in parallel with the configured initial model. Finally, the target model is generated based on the parameters of multiple sub-models obtained from the training, further improving model training efficiency.

[0039] Please refer to Figures 2 and 3 together. Figure 2 is a schematic diagram of a data processing scenario provided by an embodiment of this application, and Figure 3 is a schematic diagram of a data processing scenario provided by an embodiment of this application. As shown in Figure 2, the computer device can be the business server 100 of the embodiment corresponding to Figure 1 above. The computer device can acquire M initial chipsets. Taking the M initial chipsets as initial chipset 1, initial chipset 2, initial chipset 3, and initial chipset 4 as an example, initial chipset 1 can include chips 11, 12, 13, and 14, all with chip model A. Initial chipset 2 can include chips 22 and 23, all with chip model B. Initial chipset 3 can include chips 31, 32, 33, and 34, all with chip model C. Initial chipset 4 can include chips 41, 42, 43, 44, 45, 46, 47, and 48, all with chip model D.

[0040] The computer device can obtain the number of learning parameters corresponding to a single network layer in the initial model. The number of learning parameters can refer to the total number of model parameters, activation parameters, and gradient parameters calculated during the training of a single network layer. Based on the number of learning parameters and the chip models of the M initial chipsets, the computer device can group the chips within each initial chipset to obtain P pipeline parallel groups. Taking P pipeline parallel groups including pipeline parallel group 1a, pipeline parallel group 1b, pipeline parallel group 2a, pipeline parallel group 3a, pipeline parallel group 3b, pipeline parallel group 3c, pipeline parallel group 3d, pipeline parallel group 4a, and pipeline parallel group 4b as an example. Pipeline parallel group 1a may include chips 11 and 12; pipeline parallel group 1b may include chips 13 and 14; pipeline parallel group 2a may include chips 22 and 23; pipeline parallel group 3a may include only chip 31; pipeline parallel group 3b may include only chip 32; pipeline parallel group 3c may include only chip 33; pipeline parallel group 3d may include only chip 34; pipeline parallel group 4a may include chips 41, 42, 43, and 44; and pipeline parallel group 4b may include chips 45, 46, 47, and 48. The computational memory of pipeline parallel groups 1a, 1b, and 2a is greater than or equal to four times the number of learning parameters, meaning that pipeline parallel groups 1a, 1b, and 2a can all be used to train at least four network layers in the initial model in parallel. The computational memory of pipelined parallel groups 3a, 3b, 3c, 3d, 4a, and 4b is all greater than or equal to four times the number of learning parameters. That is, pipelined parallel groups 3a, 3b, 3c, 3d, 4a, and 4b can all be used to train at least four network layers in the initial model in parallel.

[0041] Computer equipment can obtain the average computing power coefficient of the pipelined parallel groups and the group computing power coefficient corresponding to each pipelined parallel group. Based on load balancing conditions, the average computing power coefficient of the pipelined parallel groups, the number of network layers of the initial model, and the group computing power coefficient corresponding to each pipelined parallel group, the pipelined parallel groups are grouped and combined to obtain D data parallel groups and the sample input volume corresponding to each D data parallel group. Taking D data parallel groups including data parallel group 1, data parallel group 2, and data parallel group 3 as an example, the computing memory corresponding to each data parallel group is greater than or equal to the total number of learning parameters of the initial model, that is, each data parallel group can train the initial model independently, and the sample input volume corresponding to each data parallel group can be the same or different. Grouping and combining based on load balancing conditions means that the difference in training time between different data parallel groups is less than a preset threshold, and the sample input volume corresponding to the data parallel group matches the overall performance of the chip within the group, so as to effectively utilize the chip's computing resources.

[0042] Taking data parallel group 1 as an example, the computer device can partition the training sample set based on the sample input volume corresponding to data parallel group 1, data parallel group 2, and data parallel group 3, resulting in data partition 1 (data parallel group 1), data partition 2 (data parallel group 2), and data partition 3 (data parallel group 3). Taking data parallel group 1 as an example, data parallel group 1 can train the locally configured initial model based on the allocated data partition 1. The initial model can have 8 network layers. Pipeline parallel group 1a can be used to train 4 of these network layers, pipeline parallel group 3a can be used to train 2 of these network layers, and pipeline parallel group 4a can be used to train 2 of these network layers.

[0043] It is understandable that the splitting strategy corresponding to Figure 3 can be arbitrarily combined while satisfying the load balancing condition. In some embodiments, corresponding to the distributed learning of the initial model, the data parallelism in Figure 3 is 3, and the sample input amount and batch size corresponding to each data parallel group can be different. Here, the batch size refers to the amount of sample data input in each training batch. In data parallel group 1, the sample input amount of data parallel group 1 can be 1 / 4 of the training sample set, the batch size can be 2, the pipeline parallelism of data parallel group 1 is 3, the tensor parallelism of pipeline parallel group 1a is 2, the tensor parallelism of pipeline parallel group 3a is 1, and the tensor parallelism of pipeline parallel group 4a is 4. In data parallelism group 2, the sample input size can be 1 / 4 of the training sample set, the batch size can be 2, and the pipeline parallelism of data parallelism group 2 is 3. The tensor parallelism of pipeline parallelism group 1b is 2, the tensor parallelism of pipeline parallelism group 3b is 1, and the tensor parallelism of pipeline parallelism group 3d is 1. In data parallelism group 3, the sample input size can be 1 / 2 of the training sample set, the batch size can be 4, the pipeline parallelism of data parallelism group 2 is 3, the tensor parallelism of pipeline parallelism group 3c is 1, the tensor parallelism of pipeline parallelism group 2a is 2, and the tensor parallelism of pipeline parallelism group 4b is 4.

[0044] After each data parallel group completes its forward and backward propagation computations, it can return the calculated sub-model parameters to the computing device. The computing device then aggregates these sub-model parameters to obtain aggregated model parameters. Based on these aggregated model parameters, the initial model configured in each data parallel group is updated, and the next batch of training is performed. When the initial model in the computing device meets the model convergence condition, it can be determined as the target model.

[0045] This application supports flexible multi-dimensional model parallelism strategies. The training platform can dynamically decompose the training task into multiple sub-tasks based on the characteristics of the training task and the availability of resources, and allocate them to different machine types. Different tensor parallelism degrees can be configured within each pipeline parallel group. Based on the average and group computational power coefficients of the pipeline parallel groups, each pipeline parallel group can be assigned a task load matching its own group computational power coefficient. Different pipeline parallel groups can be assigned to different types of data parallel groups. Data parallel groups can be machines or devices equipped with chips within the pipeline parallel groups. Thus, on different types of machines, the number of model layers and the tensor parallelism of the chips within the pipeline parallel groups can be configured according to the machine's hardware characteristics. Based on the sample input volume of the data parallel groups and the group computational power coefficients of the pipeline parallel groups, different pipeline parallelism degrees can be configured for each data parallel group. This flexible workload task splitting mechanism provides a larger solution space for parallel strategy search, optimizes the allocation of training tasks, and improves overall training performance.

[0046] Please refer to Figure 4, which is a schematic flowchart of a data processing method provided in an embodiment of this application. This data processing method can be executed by a computer device, which can be the business server 100 shown in Figure 1. The following description will use the execution of this data processing method by a computer device as an example. The data processing method may include at least the following steps S101-S103:

[0047] Step S101: Obtain M initial chipsets; M is a positive integer. All chips in an initial chipset are the same model, and the chip models in each initial chipset are different.

[0048] In some embodiments, the computer device can group all chips (also referred to as graphics card chips, GPU chips, or graphics processing units) according to their chip models. Each initial chip group contains chips of the same model, and tensor parallelism is used within each group. For example, if there are M chip models, the computer device can group them into M initial chip groups. Within an initial chip group, all chips have the same model, but the chip models within each initial chip group are different. Tensor parallelism, also known as intra-layer parallelism, refers to dividing the parameters within each network layer of the model and distributing the training tasks for different parameters of the same network layer to multiple chips for completion. The chip model refers to the chip's designation, used to distinguish the chip's product type and performance level. In some embodiments, the chip can also be a computing chip containing processing units, such as a CPU (Central Processing Unit) chip.

[0049] In some embodiments, the computer device determines the group computing power coefficient of each initial chipset. The group computing power coefficient is a comprehensive indicator used to quantify the overall computing performance of the initial chipset. The group computing power coefficient is a parameter determined based on the business computing power parameters, storage parameters, bandwidth parameters and communication parameters of each chip in the initial chipset. In practical applications, computer equipment acquires the computing power, storage, bandwidth, and communication parameters of each chip. These parameters are then normalized to a uniform range (e.g., 0 to 1). Weights are defined for each parameter based on the business scenario, with each weight being a real number between 0 and 1, and the sum of all weights equals 1. For example, in an AI model training scenario, the weights for computing power parameters could be 0.5, storage parameters 0.3, bandwidth parameters 0.15, and communication parameters 0.05. In a high-performance computing scenario, the weights could be 0.3, 0.25, 0.05, and 0.4. Finally, based on each parameter and its corresponding weight, a group computing power coefficient is determined.

[0050] Computer devices can sort initial chipsets in ascending order using their group computing power coefficients, placing chipsets with larger coefficients first, resulting in a chipset sequence. This sequence includes initial chipset 1, initial chipset 2, ..., initial chipset M. Initial chipset 1 has a larger group computing power coefficient than initial chipset 2, and initial chipset M has the smallest coefficient. A larger coefficient indicates lower overall chip performance, while a smaller coefficient indicates higher overall chip performance.

[0051] In the M initial chipsets of the chipset sequence, the number of chips of each chip model can be {N1, N2, N3, ..., N} in sequence. M}, where N1 represents the number of chips in the initial chipset 1, N1, and N M This means that the initial chipset M has N chips. M .

[0052] It is understood that the aforementioned chips, or chipsets composed of chips, as well as the parallel groups mentioned below, can all be machines or devices equipped with chips, which can be used to train the initial model. Since the scale of machines or devices equipped with chips is usually 8 cards per machine, the number of chips in each initial chipset can be a multiple of 8.

[0053] Step S102: Based on the number of learning parameters corresponding to a single network layer in the initial model and the chip models of the M initial chip groups, the chips in each initial chip group are grouped to obtain P pipeline parallel groups; P is a positive integer, the chip models in a pipeline parallel group are all the same, and the computing memory corresponding to each pipeline parallel group is greater than or equal to the number of learning parameters.

[0054] In some embodiments, the computer device can obtain the number of learning parameters corresponding to a single network layer in the initial model. The number of learning parameters can refer to the total number of model parameters, activation parameters, and gradient parameters calculated during training of a single network layer. Based on the number of learning parameters and the chip models of the M initial chipsets, the computer device can group the chips within each initial chipset to obtain P pipeline parallel groups.

[0055] For example, an initial chipset 1 may include chip model A, an initial chipset 2 may include chip model B, and an initial chipset 3 may include chip model C. Each initial chipset contains 16 chips. The computer device can divide the chips in initial chipset 1 into pipelined parallel groups 1a (8 chips), 1b (4 chips), and 1c (4 chips); divide the chips in initial chipset 2 into pipelined parallel groups 2a (12 chips) and 2b (4 chips); and divide the chips in initial chipset 3 into pipelined parallel groups 3a and 3b (8 chips). Within each pipelined parallel group, the chip models are identical, and the computational memory corresponding to each pipelined parallel group is greater than or equal to the number of learning parameters. Pipeline parallelism refers to distributing one or more network layers (usually consecutive layers) of a model to multiple pipelined parallel groups for processing, with each pipelined parallel group responsible for processing a portion of the model's network layers.

[0056] Step S103: Based on the load balancing condition, the average computing power coefficient of the P pipeline parallel groups, the number of network layers of the initial model, and the group computing power coefficients corresponding to the P pipeline parallel groups, the P pipeline parallel groups are grouped and combined to obtain D data parallel groups and the sample input amount corresponding to the D data parallel groups respectively; D is a positive integer less than P, and the computing memory corresponding to each data parallel group is greater than or equal to the total number of learning parameters of the initial model; the input samples indicated by the sample input amounts corresponding to the D data parallel groups are used to train the initial model configured on the corresponding data parallel groups in parallel.

[0057] In some embodiments, the computer device can obtain the average computing power coefficient of P pipelined parallel groups. The average computing power coefficient of the P pipelined parallel groups is the average of the group computing power coefficients corresponding to the P pipelined parallel groups. The group computing power coefficient is determined comprehensively based on the chip's business computing power parameters, storage parameters, bandwidth parameters, and communication parameters. Based on the average computing power coefficient of the P pipelined parallel groups, the number of network layers in the initial model, and the group computing power coefficients corresponding to the P pipelined parallel groups, the computer device can group and combine the P pipelined parallel groups to obtain D data parallel groups and the sample input quantities corresponding to the D data parallel groups.

[0058] Data parallelism refers to configuring the same initial model across multiple data parallel groups, with each group trained using different batches of data. Each data parallel group independently computes the loss function and gradients, then aggregates the gradients through some form of communication to update the model parameters of all data parallel groups. Each data parallel group can include allocated pipeline parallel groups, and the computational memory corresponding to each data parallel group is the sum of the computational memory of the allocated pipeline parallel groups. The computational memory corresponding to each data parallel group is greater than or equal to the total number of learning parameters of the initial model, meaning each data parallel group can train the initial model independently. The sample input size for each data parallel group can be the same or different. Grouping based on load balancing conditions means grouping pipeline parallel groups with the aim of ensuring the difference in training time between different data parallel groups is less than a preset threshold, and that the sample input size of each data parallel group matches the overall performance of the chips within the group, thus effectively utilizing the chip's computational resources.

[0059] It is understandable that computer equipment can allocate task loads to each pipeline parallel group based on the average computing power coefficient of the P pipeline parallel groups and the corresponding group computing power coefficients of the P pipeline parallel groups. This allows the task loads allocated to pipeline parallel groups with larger group computing power coefficients (i.e., weaker overall computing power) to meet their own overall computing power, thus preventing excessive task loads from hindering the training performance of the data parallel groups. Conversely, it allows the task loads allocated to pipeline parallel groups with smaller group computing power coefficients (i.e., stronger overall computing power) to meet their own overall computing power, improving the utilization rate of chip computing resources within pipeline parallel groups with stronger overall computing power.

[0060] The computer device can partition the training sample set based on the sample input volume corresponding to each data parallel group, resulting in a training sample group for each pipeline parallel group. Based on the training sample groups, the initial model configured for each data parallel group is trained in parallel. After each data parallel group completes forward and backward propagation computations, it can return the calculated sub-model parameters to the computer device. The computer device aggregates the sub-model parameters to obtain aggregated model parameters, and then updates the initial model configured in each data parallel group based on the aggregated model parameters for the next batch of training. When the initial model in the computer device meets the model convergence condition, the initial model that meets the model convergence condition can be determined as the target model.

[0061] In this embodiment, all chips are grouped by chip model to obtain initial chip groups containing only the same chip model. Each initial chip group has a different chip model. Based on the number of learning parameters corresponding to a single network layer in the initial model and the chip models of the M initial chip groups, the chips within each initial chip group are further grouped to obtain P pipelined parallel groups. All chips within a pipelined parallel group are of the same model, and the computational memory corresponding to each pipelined parallel group is greater than or equal to the number of learning parameters. Tensor parallel training of the initial model's network layers is performed using chips of the same model. Based on load balancing conditions, the average computational power coefficient of the P pipelined parallel groups, the number of network layers in the initial model, and the group computational power coefficients corresponding to the P pipelined parallel groups, the P pipelined parallel groups are grouped and combined to obtain D data parallel groups and their corresponding sample input quantities. The input samples indicated by the sample input quantities corresponding to the D data parallel groups are used to train the initial model configured on the corresponding data parallel group in parallel. The D data parallel groups satisfy the load balancing condition. Data parallelism refers to configuring the same initial model across multiple data parallel groups, with each group independently processing its own input samples, and then aggregating them to integrate all model parameters. This application's embodiment uses pipelined parallel groups composed of chips of the same model to determine the average computing power coefficient and group computing power coefficient of each pipelined parallel group. Based on the idea of ​​computing power equivalence, the pipelined parallel groups are grouped and combined to obtain data parallel groups. Each data parallel group is allocated a task load with equivalent computing power. While ensuring that the computing memory corresponding to each data parallel group is greater than or equal to the total number of learning parameters of the initial model, the task load of the initial model is split along the dimension of sample input size by the data parallel groups, determining the sample input amount corresponding to each data parallel group. This ensures load balancing for each data parallel group, thereby improving the utilization rate of computing resources of the hybrid deployed chips and improving model training efficiency.

[0062] Please refer to Figure 5, which is a schematic flowchart of a data processing method provided in an embodiment of this application. This data processing method can be executed by a computer device, which can be the business server 100 shown in Figure 1. The following description will use the execution of this data processing method by a computer device as an example. The data processing method may include at least the following steps S201-S207:

[0063] Step S201: Obtain M initial chipsets; M is a positive integer, all chips in an initial chipset are the same model, and the chip models of each initial chipset are different.

[0064] In some embodiments, the computer device can group all chips according to their chip models. Each initial chip group contains chips of the same model, and tensor parallelism is used within each group. For example, if there are M chip models, the computer device can group them into M initial chip groups. Within an initial chip group, all chips have the same model, and the chip models within each initial chip group are different from each other. The chip model refers to the chip's designation, used to distinguish the chip's product type and performance level.

[0065] The computer equipment can sort the initial chipsets in ascending order using their group computing power coefficients, placing chipsets with larger coefficients first. The M initial chipsets are named Initial Chipset 1, Initial Chipset 2, ..., Initial Chipset M. Initial Chipset 1 has a larger group computing power coefficient than Initial Chipset 2, and Initial Chipset M has the smallest coefficient. The group computing power coefficient is a parameter determined based on the comprehensive computing power, storage, bandwidth, and communication parameters of the chips within the parallel group. A larger coefficient indicates poorer overall chip performance, while a smaller coefficient indicates better overall chip performance.

[0066] In the M initial chipsets, the number of each chip model can be {N1, N2, N3, ..., N} in sequence. M}, where N1 represents the number of chips in the initial chipset 1, N1, and N M This means that the initial chipset M has N chips. M .

[0067] It is understandable that the aforementioned chips, or chipsets or parallel groups composed of chips, can be machines or devices equipped with chips, which can be used to train initial models. Since the scale of machines or devices equipped with chips is usually 8 cards per machine, the number of chips in each initial chipset can be a multiple of 8.

[0068] Step S202: Obtain the initial grouping parameters corresponding to the M initial chipsets respectively; all M initial grouping parameters are greater than or equal to 2.

[0069] In some embodiments, the computer device can acquire M pre-set initial grouping parameters. These M initial grouping parameters are hyperparameters determined based on the chip model, the number of chips, and the number of learning parameters corresponding to a single network layer in the initial model. The M initial grouping parameters can be {g1, g2, g3, ..., g...} M The initial grouping parameters are positive integers.

[0070] Among them, g i The value needs to satisfy a storage constraint, which can be g. i The computational memory of a certain chip model is greater than or equal to the number of learning parameters for X network layers of the initial model, i.e., g. i A certain type of chip can hold at least the storage required for one network layer computation, where X can be a positive integer greater than 2.

[0071] There are M initial chipsets, including initial chipset 1 (chip type A), initial chipset 2 (chip type B), and initial chipset 3 (chip type C). Each initial chipset contains 16 chips. Assume that one chip of chip type A is sufficient to meet the storage limit, two chips of chip type B are required to meet the storage limit, and four chips of chip type C are required to meet the storage limit. Then g i The solution space for the initial chipset 1 can take values ​​of (1~16), g i The solution space for the initial chipset 2 can take values ​​of (2~16), g i The solution space for the initial chipset 3 is (4~16).

[0072] For g i The number of chips in the resulting pipeline parallel groups after grouping must be a positive integer, i.e., N. i It can be g i Divisible. Computer devices can reduce g using pruning and greedy search strategies. i The solution space is searched to obtain the M initial grouping parameters corresponding to each of the M initial chipsets. Assuming that each initial chipset contains 16 chips, the M initial grouping parameters can be {2, 4, 8} in sequence.

[0073] Step S203: Based on M initial grouping parameters, the chips in each initial chipset are grouped separately to obtain N initial parallel groups;

[0074] In some embodiments, the computer device can group the chips within each initial chipset based on M initial grouping parameters, resulting in N initial parallel groups. Each initial chipset can be evenly divided by the corresponding initial grouping parameters. M initial parallel groups, i.e., M initial grouping parameters, can divide each chip model into groups of the following numbers: The initial parallel group. M initial chipsets including initial chipset K. i Initial chipset K i Includes S target chip models, with an initial chipset K. i The corresponding grouping parameter is G, where G, S, and i are all positive integers. For example, the computer device uses M initial grouping parameters to initialize the initial chipset K. i The process of grouping the chips within the chip group to obtain the initial parallel group can be as follows: Based on the initial chip group K i The grouping parameter G, for the initial chipset K i The chips within are grouped to obtain the initial chipset K. i The corresponding T target parallel groups; initial chipset K i Each target parallel group includes G target chip models, and the product of T and G is S. When the target parallel groups corresponding to M initial chip groups are obtained, the target parallel groups corresponding to the M initial chip groups are determined as N initial parallel groups. The N initial parallel groups include T target parallel groups.

[0075] In some embodiments, G can be 2, S can be 16, i.e., the initial chipset K i Including 16 target chip models, computer equipment can target the initial chipset K. i The chips within are grouped to obtain the initial chipset K. i There are T corresponding parallel target groups, and the product of T and G is S. When G is 2 and S is 16, the computer device can obtain the initial chipset K. i There are 8 corresponding target parallel groups, and each of the 8 target parallel groups includes 2 target chip models.

[0076] When M initial chipsets are obtained, the target parallel groups corresponding to each of the M initial chipsets are defined as N initial parallel groups, which include T target parallel groups. For example, the M initial chipsets may include initial chipset 1 (chip type A), initial chipset 2 (chip type B), and initial chipset 3 (chip type C). Each initial chipset contains 16 chips, and the M initial grouping parameters can be {2, 4, 8}. That is, the computer device can divide the chips in initial chipset 1 into 8 target parallel groups, each of which includes 2 chips of chip type A; the chips in initial chipset 2 can be divided into 4 target parallel groups, each of which includes 4 chips of chip type B; and the chips in initial chipset 3 can be divided into 2 target parallel groups, each of which includes 8 chips of chip type C. Finally, the 8 target parallel groups divided into initial chipset 1, the 4 target parallel groups divided into initial chipset 2, and the 2 target parallel groups divided into initial chipset 3 are determined as initial parallel groups, resulting in a total of 14 initial parallel groups.

[0077] Step S204: Based on the average computing power coefficient of N initial parallel groups and the group computing power coefficients corresponding to the N initial parallel groups, the chips in each initial parallel group are grouped to obtain P pipeline parallel groups.

[0078] In some embodiments, the computer device can obtain the average computing power coefficient of N initial parallel groups and the group computing power coefficients corresponding to the N initial parallel groups. Taking the calculation of the group computing power coefficient of initial parallel group 1 among the N initial parallel groups as an example, initial parallel group 1 may include S target model chips. The computer device can obtain the service computing power parameters, storage parameters, bandwidth parameters, and communication parameters of the target model chips. Among them, the service computing power parameters can be parameters obtained by normalizing the computing power performance of M types of chips, the storage parameters can be parameters obtained by normalizing the storage performance of M types of chips, the bandwidth parameters can be parameters obtained by normalizing the bandwidth performance (e.g., h2d (host to device) bandwidth, d2h (device to host) bandwidth) of M types of chips, and the communication parameters can be parameters obtained by normalizing the communication performance (e.g., communication performance between chips, communication performance between machines) of M types of chips.

[0079] The computer device can determine the intra-group communication coefficient δ based on the number of cards S and communication parameters of the initial parallel group 1. SThe intra-group communication coefficient is a coefficient used to quantify the impact of inter-chip communication overhead on overall computing power during multi-card (S>1) parallel computing. The more cards S there are, the more frequent the inter-chip communication, the greater the communication overhead, and the greater the negative impact on computing power. However, the communication parameters themselves characterize the hardware's communication performance; the better the hardware's communication performance, the smaller the communication overhead. Therefore, the intra-group communication coefficient is jointly determined by the number of cards S and the communication parameters. In some embodiments, the number of cards and the communication parameters can be used as inputs to a preset function to determine the intra-group communication coefficient. This preset function can be a linear function, a logarithmic function, a quadratic function, etc. Based on the target chip's service computing power parameters, storage parameters, bandwidth parameters, and the initial number of cards S in parallel group 1, the service computing power coefficient is determined. Based on the intra-group communication coefficient and the service computing power coefficient, the initial parallel group 1's group computing power coefficient is determined.

[0080] When S equals 1, since no intra-group communication is required, the intra-group communication coefficient is 0, and the group computing power coefficient is... The calculation process can be shown in formula (1):

[0081] Where, α P α is a hyperparameter representing the linear relationship between business computing power parameters and group computing power coefficients. m To represent the hyperparameter α that indicates a linear relationship between storage parameters and group computing power coefficients B α is a hyperparameter representing the linear relationship between bandwidth and group computing power coefficient, and β is a hyperparameter characterizing the intercept term in the linear relationship. P′ is the service computing power parameter, M′ is the storage parameter, B′ is the bandwidth parameter, and α P ·P′+α m ·M′+α B •B′ represents the computing power parameters of the business.

[0082] When S is greater than 1, the group computing power coefficient The calculation process can be shown in formula (2):

[0083] Where C′ is the communication parameter, δ S The relationship between the number of cards S and the function mapping is not limited in the embodiments of this application.

[0084] Because chips with poor overall computing power will delay the overall computing latency, computer equipment can optimize the computing power coefficients of all groups. The initial parallel groups with a computational power coefficient greater than the average computational power coefficient are further split to reduce the group computational power coefficient, and the most suitable grouping strategy is found through traversal. The process of further splitting the initial parallel groups by the computer device can be as follows: obtain the group computational power coefficient and the average computational power coefficient of the N initial parallel groups respectively, and determine the initial parallel groups with a group computational power coefficient greater than the average computational power coefficient of the N initial parallel groups as the groups to be split, the number of groups to be split is Q, where Q is a positive integer; based on the number of learning parameters, the chips in each group to be split are split separately to obtain the set of split groups corresponding to each group to be split; Q is a positive integer, and the computing memory corresponding to each split group in the set of split groups is greater than or equal to the number of learning parameters; based on the set of split groups and the NQ initial parallel groups other than the Q groups to be split, generate A sets of undetermined parallel groups, where A is a positive integer; obtain the communication evaluation score corresponding to each of the A sets of undetermined parallel groups, and determine the set of undetermined parallel groups with the largest communication evaluation score as P pipeline parallel groups; A is a positive integer.

[0085] In some embodiments, the computer device can obtain the average computing power coefficient of N initial parallel groups, where the average computing power coefficient can be the average of the group computing power coefficients of the N initial parallel groups. The computer device can determine Q initial parallel groups with group computing power coefficients greater than the average computing power coefficient of the N initial parallel groups as groups to be split, and based on the number of learning parameters, split the Q groups to be split, obtaining a set of split groups corresponding to each group to be split. The computational memory corresponding to each split group in the set of split groups is greater than or equal to the number of learning parameters.

[0086] The computer equipment can generate A sets of undetermined parallel groups based on the already split set and NQ initial parallel groups excluding the Q groups to be split. Taking an average computing power coefficient of 1 as an example, the N initial parallel groups can include initial parallel group 1 with a computing power coefficient of 1.4, initial parallel group 2 with a computing power coefficient of 1.2, and initial parallel group 3 with a computing power coefficient of 0.8. Initial parallel group 1 can include 8 chips, and initial parallel group 2 and initial parallel group 3 each include 4 chips. The computer device can split the initial parallel group 1 (with a computing power coefficient of 1.4) and the initial parallel group 2 (with a computing power coefficient of 1.2) into a set of split groups corresponding to each initial parallel group. For example, the computer device can further split the initial parallel group 1 to reduce the impact of communication parameters on the overall computing power. If at least two chips are needed in the initial parallel group 1 to accommodate the storage required for one network layer computation (i.e., the computational memory of the two chips in the initial parallel group 1 is greater than or equal to the number of learning parameters), the computer device can split the chips in the initial parallel group 1 into a set of split groups 1 and a set of split groups 2. Set of split groups 1 refers to splitting the chips in the initial parallel group 1 into a set of split groups containing two chips and a set of split groups containing six chips. Set of split groups 2 refers to splitting the chips in the initial parallel group 1 into a set of split groups containing four chips and a set of split groups containing four chips. Since the splitting is symmetrical, there are only two possible splitting scenarios, and the computer device only needs to traverse and compute these two locations. The computational memory corresponding to the split groups in both split group set 1 and split group set 2 is greater than or equal to the number of learning parameters.

[0087] The computer device can further split the initial parallel group 2 to reduce the impact of communication parameters on the overall computing power. If at least two chips are needed in the initial parallel group 2 to accommodate the storage required for one network layer computation (i.e., the computational memory of the two chips in the initial parallel group 2 is greater than or equal to the number of learning parameters), the computer device can split the initial parallel group 2 into a split group set 3. The split group set 3 refers to splitting the chips in the initial parallel group 2 into a split group containing two chips and a split group containing two chips. The computational memory corresponding to each split group in the split group set 3 is greater than or equal to the number of learning parameters.

[0088] A computer device can generate A sets of undetermined parallel groups based on the already split set and NQ initial parallel groups excluding Q groups to be split, in the example above. The computer device can arrange and combine the splitting states of initial parallel group 1, initial parallel group 2, and initial parallel group 3 to obtain A sets of undetermined parallel groups. These A sets of undetermined parallel groups can include undetermined parallel group set 1 and undetermined parallel group set 2. Undetermined parallel group set 1 includes the already split set 1, the already split set 3, and the initial parallel group 3. Undetermined parallel group set 2 includes the already split set 2, the already split set 3, and the initial parallel group 3.

[0089] The computer device can obtain the communication evaluation scores corresponding to A sets of undetermined parallel groups, and determine the set of undetermined parallel groups with the highest communication evaluation score as P pipelined parallel groups, with A sets of undetermined parallel groups including set A of undetermined parallel groups. i The set of undetermined parallel groups A i The set includes L undetermined parallel groups, consisting of the already split groups and the initial parallel groups, where i and L are positive integers. Obtain the set A of undetermined parallel groups. i The corresponding communication evaluation score process can be: obtaining the set A of undetermined parallel groups. i The L undetermined parallel groups are defined with their respective communication parameters and group computing power coefficients. Chips in the L undetermined parallel groups whose communication parameters are greater than the average communication parameter are identified as the first chip, and chips in the L undetermined parallel groups whose communication parameters are less than or equal to the average communication parameter are identified as the second chip. The average communication parameter is determined based on the communication parameters of the chip models in the M initial chip sets. First and second evaluation parameters are obtained. Based on the first evaluation parameters and the group computing power coefficients of the undetermined parallel groups including the first chip, a pipeline parallel evaluation score is determined. Based on the second evaluation parameters and the group computing power coefficients of the undetermined parallel groups including the second chip, a data parallel evaluation score is determined. The sum of the pipeline parallel evaluation score and the data parallel evaluation score is used to determine the set A of undetermined parallel groups. i The communication evaluation score.

[0090] In some embodiments, the computer device may acquire a set A of pending parallel groups. i The communication parameters C′ of the chips in the L undetermined parallel groups and the group computing power coefficients of the L undetermined parallel groups are respectively. In the set of L undetermined parallel groups, the chip whose communication parameters are greater than the average communication parameter is identified as the first chip. For example, in the set of undetermined parallel groups A iIn a set of L parallel chips, the communication parameters of chips 1, 2, ..., n are greater than the average communication parameter value. A computer device can identify a chip in a set of L pending parallel groups whose communication parameters are less than or equal to the average communication parameter value as the second chip. For example, in the set of pending parallel groups A... i Chip L1 to chip L C If the communication parameters are less than the average communication parameters, the set of undetermined parallel groups A can be determined. i Chip L1 to chip L C All are identified as the first chip. The set A of undetermined parallel groups. i The communication parameters of chips n+1, ..., L are greater than the average communication parameter. The set A of undetermined parallel groups can be... i Chips n+1, ..., L are all designated as the second chip. Here, n is a positive integer, and the average communication parameters are determined based on the communication parameters of the chip models of the M initial chipsets.

[0091] The computer device can acquire a first evaluation parameter and a second evaluation parameter. Based on the first evaluation parameter and the group computing power coefficient of the undetermined parallel group including the first chip, it determines the pipeline parallel evaluation score, S. PP The calculation method can be shown in formula (3):

[0092] Where, λ i γ is the first evaluation parameter used to evaluate the communication score of chips within a parallel group during pipeline parallelism. n The hyperparameters are determined based on the maximum value of the communication parameters in the first chip.

[0093] The computer device can determine the data parallelism evaluation score S based on the second evaluation parameter and the group computing power coefficient of the undetermined parallel group, including the second chip. DP The calculation method can be shown in formula (4):

[0094] Where, λ j The second evaluation parameter, γ, is used to evaluate the communication score of chips within the parallel group in data parallelism. L-n The hyperparameters are determined based on the maximum value of the communication parameters in the second chip.

[0095] The computer equipment can determine the sum of the pipeline parallel evaluation score and the data parallel evaluation score as the set A of undetermined parallel groups. i The communication evaluation score. The communication evaluation score Score can be calculated as shown in formula (5): Score = S PP +S DP +θ Formula (5)

[0096] Where θ is the calibration term.

[0097] Step S205: Obtain the initial micro-batch values ​​of the initial model; generate the target task duration based on the initial micro-batch values ​​for the average computing power coefficient of P pipeline parallel groups; the initial micro-batch values ​​are determined based on the task domain of the initial model; determine the data parallel grouping parameters based on the initial micro-batch values ​​and the total number of samples; generate D selection coefficient lists and D initial sample input quantities corresponding to the allocation parallel groups based on the target task duration and the group computing power coefficients corresponding to the P pipeline parallel groups; each selection coefficient list includes the selection coefficients corresponding to the P pipeline parallel groups, and the selection coefficients of the pipeline parallel groups are used to indicate the selection mapping relationship between the pipeline parallel groups and the allocation parallel groups; D is the data parallel grouping parameter.

[0098] In some embodiments, the computer device can obtain the initial micro-batch values ​​of the initial model. These initial micro-batch values ​​are fixed values ​​determined based on the task domain of the initial model. For example, the initial micro-batch value could be 1 in the task domain of text-to-text, and other values ​​in the task domain of text-to-image. This embodiment of the application does not impose such limitations. The initial micro-batch value represents the amount of sample data input by the parallel group in each training batch within the task domain of the initial model. The computer device can generate a target task duration τ(b) based on the initial micro-batch values ​​for the average computing power coefficient of P pipeline parallel groups.

[0099] In some embodiments, the initial micro-batch number reflects the amount of sample data input to the parallel groups in each training batch, while the average computing power coefficient represents the overall computing power of the P pipeline parallel groups. When generating the target task duration for the average computing power coefficient of the P pipeline parallel groups based on the initial micro-batch number, the single-batch data size (denoted as K, i.e., the number of samples processed per batch by each parallel group) is first determined based on the initial micro-batch number. For the P pipeline parallel groups, assuming balanced load across groups, the overall batch processing size is K×P. If the average computing power coefficient is F (representing the total number of samples that the P parallel groups can process per unit time), the theoretical time for processing a single batch of data is (K×P) / F. The final target task duration needs to be adjusted based on the characteristics of pipeline parallelism: considering the dependencies between pipeline stages (e.g., the output of the previous stage serves as the input of the next stage), pipeline communication overhead and stage synchronization time need to be added to the theoretical time to obtain the target task duration.

[0100] The computer equipment can determine the data parallel grouping parameter D based on the initial micro-batch size and the total number of samples. This parameter D determines the data parallelism of the distributed training for the initial model. In some embodiments, the data parallelism D is calculated by the ratio of the total number of samples to the initial micro-batch size, combined with the resource allocation for distributed training. The value of D must ensure that the total sample size is evenly distributed among D parallel groups according to the initial micro-batch size, thereby achieving efficient data parallel training. In other words, D determines the number of data segments that are split and processed in parallel. The ratio of the total sample size to the initial micro-batch size is rounded up to obtain the total training batches. The core of data parallelism is to split the samples into batches and train them synchronously in D parallel groups. Therefore, D must satisfy the following: the number of micro-batches processed by a single parallel group is K (consistent with the initial micro-batch size), and the D parallel groups can process D×K samples simultaneously. Considering the distributed resource allocation (e.g., the total number of available devices is G), the value of D must satisfy D≤G (resource limit), and D×K should be as close as possible to the optimal efficiency of hardware parallel processing (e.g., memory limitations, communication bandwidth, etc.). Ultimately, D is usually chosen as the largest integer that satisfies the above conditions, thus ensuring that the data splitting can make full use of resources without exceeding the device's load capacity.

[0101] Taking data parallelism D as an example, the computer equipment can group and combine pipeline parallel groups based on the data parallel grouping parameter D, the target task duration, and the group computing power coefficients corresponding to the P pipeline parallel groups, resulting in D allocated parallel groups. The computer equipment can then determine the selection coefficient list and the initial sample input quantities corresponding to the D allocated parallel groups based on the data parallel grouping parameter D, the target task duration, and the group computing power coefficients corresponding to the P pipeline parallel groups.

[0102] D parallel allocation groups include parallel allocation group FP i For example, i is a positive integer; the process of generating a selection coefficient list and the initial sample input for each of the D allocated parallel groups based on the data parallel grouping parameter D, the target task duration, and the group computing power coefficients corresponding to the P pipeline parallel groups can be as follows: obtain the allocated parallel group FP i The corresponding P undetermined selection parameters are multiplied by the corresponding group computing power coefficients to determine the P first load parameters. Second load parameters are then generated based on these P first load parameters. Each of the P undetermined selection parameters corresponds one-to-one with one of the P pipeline parallel groups. The allocation of the parallel group FP is then obtained. iFor the corresponding undetermined sample parameters, the ratio between the second load parameter and the undetermined sample parameters is determined as the undetermined task duration. P undetermined selection parameters and undetermined sample parameters are adjusted to obtain H sets of first parameters; H is a positive integer, and each first parameter set includes the parameter values ​​of P undetermined selection parameters and the parameter values ​​of the undetermined sample parameters. One or more first parameter sets that satisfy the condition of maximizing the ratio are determined as second parameter sets. The second parameter set containing the smallest undetermined sample parameters is determined as the target allocation parameter set. The parameter values ​​of the undetermined sample parameters in the target allocation parameter set are determined as the allocation parallel group FP. i The corresponding initial sample input quantity determines the parameter values ​​of the P undetermined selection parameters in the target allocation parameter set as the allocation parallel group FP. i The corresponding list of selection coefficients; the maximum ratio condition refers to the condition where the ratio between the target task duration and the pending task duration reaches its maximum value.

[0103] In some embodiments, the computer device can generate and allocate parallel groups FP. i The corresponding P undetermined selection parameters ρ i,k , ρ i,k =1 indicates that the pipeline parallel group with sequence number k is assigned to the i-th parallel group, ρ i,k =0 indicates that the pipeline parallel group with sequence number k was not assigned to the i-th parallel group. Therefore, the computer device can select P undetermined parameters ρ. i,k Group computing power coefficients corresponding to P undetermined selection parameters The product of these parameters determines P first load parameters, and second load parameters are generated based on these P first load parameters. The P undetermined selection parameters each correspond one-to-one with P pipeline parallel groups. The allocation of parallel groups FP is then determined. i The corresponding undetermined sample parameter m i The ratio between the second load parameter and the undetermined sample parameter is determined as the undetermined task duration. P undetermined selection parameters and undetermined sample parameters are adjusted to obtain H sets of first parameters. Each first parameter set includes the parameter values ​​of P undetermined selection parameters and the parameter values ​​of the undetermined sample parameters. The computer device can determine one or more first parameter sets that satisfy the condition of maximizing the ratio as the second parameter set, determine the second parameter set containing the smallest undetermined sample parameter as the target allocation parameter set, and determine the parameter values ​​of the undetermined sample parameters in the target allocation parameter set as the allocation parallel group FP. i The corresponding initial sample input quantity determines the parameter values ​​of the P undetermined selection parameters in the target allocation parameter set as the allocation parallel group FP. iThe corresponding selection coefficient list. The maximum ratio condition refers to the condition where the ratio between the target task duration and the undetermined task duration reaches its maximum value. The above problem can be transformed into an integer linear programming problem, and the solution process is shown in formula (6):

[0104] in, This refers to the parameter m that makes the function reach its minimum value. i ,ρ i,k The value of max refers to The maximum value represents the condition for the maximum ratio. The first set of parameters that satisfies the condition for the maximum ratio can be... The solution set, which contains the second set of parameters with the smallest undetermined sample parameters, is determined as the target allocation parameter set. This can refer to the second set of parameters that satisfy... The solution set of m. i Let represent the parameters of the undetermined sample corresponding to the i-th parallel allocation group, τ(b) be the target task duration, and P be the number of pipeline parallel groups. This represents the group computing power coefficient corresponding to the kth pipeline parallel group, D is the number of parallel groups allocated, B is gbs (global batch size), b is mbs (micro batch size), and N0 refers to the set of natural numbers.

[0105] To allocate parallel groups FP i For example, a computer device can select P undetermined selection parameters ρ i,k Group computing power coefficients corresponding to P undetermined selection parameters The product of these two parameters is determined as P first load parameters, i.e., the first load parameters are... The product of the P first load parameters is used to determine the second load parameter, which is the sum of the P first load parameters. By solving the integer linear programming problem and partially enumerating, we can find the solution to m. i and ρ i,k This yields an optimal parallel strategy under the current mixed-distribution cluster resources.

[0106] Taking D as 2 and P as 10 as an example, P pipeline parallel groups can include 6 combinations of chip model A: pipeline parallel group A0 (1 card), pipeline parallel group A1 (1 card), pipeline parallel group A2 (2 cards), pipeline parallel group A3 (1 card), pipeline parallel group A4 (1 card), and pipeline parallel group A5 (2 cards); 3 combinations of chip model B: pipeline parallel group B0 (2 cards), pipeline parallel group B1 (2 cards), and pipeline parallel group B2 (4 cards); and 1 combination of chip model C: pipeline parallel group C0 (8 cards). For example, computer equipment can be obtained by solving formula (6). The undetermined sample parameters in the target allocation parameter set that meet the condition of maximizing the ratio are m1 = 16 and m2 = 4, respectively, and the selection coefficient list corresponding to the target allocation parameter set is ρ. 1,k =(1,1,1,1,0,0,1,1,0,0),ρ 2,k = (0, 0, 0, 0, 1, 1, 0, 0, 1, 1), the computer device can be based on the initial sample input m i The group computing power coefficient of each pipeline parallel group is used to determine the micro-batch value (mbs) within each pipeline parallel group. For example, a computer device can solve for two allocated parallel groups, including allocated parallel group FP0 and allocated parallel group FP1. Allocated parallel group FP0 includes pipeline parallel groups A0 (mbs = 2), A1 (mbs = 2), A2 (mbs = 4), A3 (mbs = 2), B0 (mbs = 2), and B1 (mbs = 4). Allocated parallel group FP1 includes pipeline parallel groups A4 (mbs = 1), A5 (mbs = 1), B2 (mbs = 2), and C0 (mbs = 1).

[0107] In some embodiments, the task load problem of each pipeline parallel group can be transformed into an integer linear programming problem by using the above formula (6) based on the data parallel grouping parameter D, the target task duration and the group computing power coefficients corresponding to the P pipeline parallel groups, thereby calculating the micro-batch value of each pipeline parallel group, and then grouping and combining each pipeline parallel group to obtain the allocated parallel group and the initial sample input corresponding to the allocated parallel group.

[0108] Step S206: Based on the selection coefficients in the selection coefficient list, P pipeline parallel groups are allocated to corresponding allocation parallel groups to obtain D transition parallel groups containing pipeline parallel groups. Based on the load balancing conditions, target task duration, D transition parallel groups, D initial sample inputs, and the group computing power coefficients corresponding to the P pipeline parallel groups, the layer number parameters corresponding to each pipeline parallel group in the D transition parallel groups and the transition sample inputs corresponding to each of the D transition parallel groups are determined. The D transition parallel groups containing the layer number parameters are determined as D data parallel groups, and the transition sample inputs corresponding to each of the D transition parallel groups are determined as the sample inputs corresponding to each of the D data parallel groups.

[0109] In some embodiments, the computer device may allocate P pipelined parallel groups to corresponding allocated parallel groups based on selection coefficients in a selection coefficient list, resulting in D transitional parallel groups containing the pipelined parallel groups. These D transitional parallel groups include transitional parallel group GP. i For example, transitional parallel group GP i This can include C pipelined parallel groups, where i and C are both positive integers. The process by which the computer equipment determines the layer number parameters for each pipelined parallel group in the D transitional parallel groups, and the transitional sample input quantities for each of the D transitional parallel groups, based on load balancing conditions, target task duration, D transitional parallel groups, D initial sample input quantities, and the group computing power coefficients corresponding to the P pipelined parallel groups, can be: generating GP values ​​for the transitional parallel groups based on the group computing power coefficients corresponding to the C pipelined parallel groups. i The C undetermined layer number parameters are used to determine the third load parameter by multiplying the undetermined layer number parameter corresponding to each pipeline parallel group with the corresponding group computing power coefficient; based on the transitional parallel group GP i The initial sample input quantity determines the target sample parameter, and the product of the third load parameter and the target sample parameter is determined as the fourth load parameter. Based on the load balancing condition, C undetermined layer number parameters and the target sample parameter are adjusted to obtain R sets of third parameters; R is a positive integer, and each set of third parameters includes the parameter values ​​of C undetermined layer number parameters and the parameter value of the target sample parameter. One or more sets of third parameters that satisfy the maximum product condition are determined as the fourth parameter set. The fourth parameter set containing the minimum target sample parameter is determined as the target transition parameter set. The parameter values ​​of the target sample parameter in the target transition parameter set are determined as the transition parallel group GP. i The corresponding transition sample input quantity determines the parameter values ​​of the C undetermined layer number parameters in the target transition parameter set as the transition parallel group GP. i The layer number parameter corresponds to each pipeline parallel group; the maximum product condition refers to the condition where the product of the fourth load parameter and the target task duration reaches its maximum value.

[0110] In some embodiments, in the case of pipelined parallel groups within fixed data parallelism, the problem of partitioning the model layers can also be transformed into an integer linear programming problem, with D transitional parallel groups including transitional parallel group GP. i For example, transitional parallel group GP i Including the C pipelined parallel groups obtained by solving the above formula (6), the computer equipment can generate GP for the transitional parallel group based on the group computing power coefficients corresponding to the C pipelined parallel groups respectively. i C undetermined layer number parameters l i,j The number of layers to be determined, parameter l i,j This represents the layer number parameter assigned to the j-th pipeline parallel group within the i-th transition parallel group. The undetermined layer number parameter l for each pipeline parallel group is then used. i,j With the corresponding group computing power coefficient The product of these parameters is determined as the third load parameter. Among them, there are C undetermined layer number parameters l. i,j This can refer to the condition that the computing memory of the chips within the pipelined parallel group is greater than or equal to the undetermined layer number parameter l. i,j The indicated parameter quantity. For example, it could be the maximum value within the computational memory of a pipelined parallel group, i.e., the undetermined number of layers parameter l. i,j The indicated parameter quantity is approximately equal to the computational memory of the chips within the pipelined parallel group.

[0111] Computer devices can be based on transitional parallel groups (GPs). i The initial sample input quantity determines the target sample parameter. The product of the third load parameter and the target sample parameter is determined as the fourth load parameter. Based on the load balancing condition, C undetermined layer number parameters and the target sample parameter are adjusted to obtain R sets of third parameters. Here, R is a positive integer, and each set of third parameters includes the parameter values ​​of the C undetermined layer number parameters and the parameter value of the target sample parameter. The computer device can determine one or more sets of third parameters that satisfy the maximum product condition as the fourth parameter set, determine the fourth parameter set containing the minimum target sample parameter as the target transition parameter set, and determine the parameter values ​​of the target sample parameter in the target transition parameter set as the transition parallel group (GP). i The corresponding transition sample input quantity determines the parameter values ​​of the C undetermined layer number parameters in the target transition parameter set as the transition parallel group GP. i The layer number parameter corresponds to each parallel group of the pipeline; the maximum product condition refers to the condition that the product of the fourth load parameter and the target task duration reaches its maximum value. The above problem can be transformed into an integer linear programming problem, and the solution process is shown in formula (7):

[0112] in, This refers to the parameter l that makes the function reach its minimum value.i,j r i The value of max refers to The maximum value represents the condition for the maximum product. The set of third parameters that satisfies the condition for the maximum product can refer to... The solution set, which determines the fourth parameter set containing the minimum target sample parameters, can be defined as the target transition parameter set. The solution set of r. i Let l represent the target sample parameters corresponding to the i-th transition parallel group. i,j This represents the layer number parameter assigned to the j-th pipeline parallel group within the i-th transition parallel group. For example, l 1,1 =2 indicates that the first pipeline parallel group in the first transition parallel group is assigned 2 model layers. M i,j Let l represent the storage size of the j-th pipeline parallel group within the i-th transitional parallel group. i,j ×α i,j (b)+β i,j (b) indicates that the storage required for the pipeline parallel group and the number of layers are linearly related. It can refer to the maximum number of undetermined layers parameter l i,j The largest undetermined layer number parameter l i,j The corresponding number of parameters can be equal to the maximum value in the computational memory of the pipeline parallel group.

[0113] To simplify the problem, the computer device can represent the memory constraint as linearly related to the number of model layers, that is, the memory requirement of the assigned tasks cannot exceed the total storage capacity that the current machine can provide. Since the worst-performing machine determines the latency of the overall system, by adjusting the division of the number of model layers, the number of model layers allocated to each pipeline parallel group can be matched with the group computing power coefficient of the pipeline parallel group, and the load balance between pipeline parallel groups can be achieved. By further adjusting the division of the transition sample input, the task load balance between each transition parallel group can be achieved. The computer device can solve the above formula (7) to obtain multiple solution sets that satisfy the maximum product condition, that is, the target sample parameter set. The largest target sample parameter in the target sample parameter set is determined as the transition parallel group GP. i The corresponding transitional sample input is used to determine the C undetermined layer number parameters corresponding to the largest target sample parameter in the undetermined sample parameter set as the transitional parallel group GP. i The layer number parameter corresponds to each pipeline parallel group. The computer device can determine the D transition parallel groups containing the layer number parameter as D data parallel groups, and determine the transition sample input amount corresponding to each of the D transition parallel groups as the sample input amount corresponding to each of the D data parallel groups.

[0114] It's understandable that the parallelism strategy among chips is determined based on the resources of the mixed-distribution cluster. Due to the constraint of integer reduction, multiple similar solutions may exist. By solving the integer linear programming problem and partially enumerating, the computer device can determine the solution with the highest overall throughput as an optimal parallelism strategy under the current mixed-distribution cluster resources. For example, some data parallel groups may consist of pipeline parallel groups with large storage. The parallelism of PP on these data parallel groups can be reduced, or the parallelism of TP of the pipeline parallel groups within the data parallel groups can be reduced, thereby reducing the performance overhead caused by communication, etc.

[0115] Task partitioning (model layer partitioning) under a fixed parallel strategy. Considering memory constraints. i,j ×α i,j (b)+β i,j (b) Furthermore, the model layer number, chip number, and sample input size in the variable parameters are subject to integer constraints. Therefore, the distributed training linear programming problem can be transformed into solving D+1 independent integer linear programming problems through methods such as branch and bound, cutting plane, and enumeration. This restricts the feasible region of the solution set to integer points for step-by-step solution. The parameter solution set of one integer linear programming problem is used to solve the layer partitioning problem (i.e., the layer parameter l). i,j Additionally, D integer linear programming problems are used to solve how many training data samples (i.e., target sample parameters r) need to be allocated to each data parallel group. i By solving the linear programming problem, the final task (level parameter l) can be determined. i,j and target sample parameters r i The division of ).

[0116] Step S207: Based on the sample input quantities corresponding to the D data parallel groups, the training sample set is divided to obtain the training sample group corresponding to each data parallel group; the initial model configured for each data parallel group is trained in parallel based on the training sample group to obtain the sub-model parameters corresponding to each data parallel group; and the target model is generated based on the sub-model parameters corresponding to each data parallel group.

[0117] In some embodiments, the computer device can partition the training sample set based on the sample input volume corresponding to each data parallel group, obtaining a training sample group corresponding to each pipeline parallel group. Based on the training sample group, the initial model configured for each data parallel group is trained in parallel. After each data parallel group completes forward and backward propagation computations, the data parallel group can return the calculated sub-model parameters to the computer device. The computer device aggregates the sub-model parameters to obtain aggregated model parameters, and then updates the initial model configured in each data parallel group based on the aggregated model parameters for the next batch of training. When the initial model in the computer device meets the model convergence condition, the initial model that meets the model convergence condition can be determined as the target model.

[0118] This application first constructs an evaluation model for different chip models in terms of computing power, bandwidth, and storage. Based on this model, parallel strategies are decomposed to achieve load balancing across heterogeneous machines, avoiding idle waiting scenarios. Different parallel dimensions can be deployed on different types of machines. This better leverages the strengths of each chip model. Based on the hybrid deployment of heterogeneous machines with equivalent computing power and flexible task load splitting methods, the formulation of each parallel group is achieved, ensuring that the allocated workload matches the comprehensive computing power of the corresponding chip. Ultimately, the completion time of the corresponding workload on each data parallel group is similar, achieving load balancing and maximizing hybrid training efficiency.

[0119] Please also refer to Figure 6, which is a schematic diagram of a data processing scenario provided by an embodiment of this application. As shown in Figure 6, the computer device can be configured with two data parallel groups (data parallel group 1 and data parallel group 2) for distributed learning. Data parallel group 1 independently trains the initial model using data 1, and data parallel group 2 independently trains the initial model using data 2. Data parallel group 1 includes pipelined parallel group 1 and pipelined parallel group 2, and data parallel group 2 may include pipelined parallel group 3 and pipelined parallel group 4. Among them, pipelined parallel group 1 may include chips 11, 12, 13, and 14; pipelined parallel group 2 may include chips 22 and 23; pipelined parallel group 3 may include chips 31 and 32; and pipelined parallel group 4 may include chips 41, 42, 43, and 44. Pipeline parallel group 1 can be used to train network layers 1 and 2 in the initial model, and pipelined parallel group 2 can be used to train network layers 3, 4, 5, and 6 in the initial model. Pipeline parallel group 3 can be used to train network layers 1 and 2 in the initial model, and pipeline parallel group 4 can be used to train network layers 3, 4, 6 and 6 in the initial model.

[0120] The tensor parallelism of the chips in pipeline parallel group 1 is 4, in pipeline parallel group 2 it is 2, in pipeline parallel group 3 it is 2, and in pipeline parallel group 4 it is 4. Therefore, this method, based on existing methods, further decomposes the configurable granularity of the task, supporting different tensor parallelism configurations within each pipeline parallel group. Different pipeline parallel groups can be assigned to different types of data parallel groups, which can be machines or devices equipped with the chips within the pipeline parallel groups. This allows for the configuration of the number of layers and different tensor parallelisms within the pipeline parallel groups based on the hardware characteristics of different types of machines. Computer devices can also achieve task balancing of chips with different capabilities by splitting the sample size within a global batch value and inputting data of different scales, avoiding the bottleneck effect. Simultaneously, different micro-batch values ​​are input based on the machine's storage capacity to achieve a balance between computing and storage resources within a single machine.

[0121] Please also refer to Figure 7, which is a schematic diagram of a data processing scenario provided by an embodiment of this application. As shown in Figure 7, during the training process of the data parallel group, the pipeline parallel group in the data parallel group can support different heterogeneous configurations of MBS. In Figure 7, the arrows indicate the direction of the computational data flow; solid arrows indicate forward propagation computation, and dashed arrows indicate backward propagation computation. Taking a data parallel group including pipeline parallel group 1 and pipeline parallel group 2 as an example, the MBS of both pipeline parallel group 1 and pipeline parallel group 2 can be 1. As shown in Figure 7, arrow F1 can be used to represent the forward propagation computation of pipeline parallel group 1 for network layers 1 and 2. Arrow F2 can be used to represent the forward propagation computation of pipeline parallel group 2 for network layers 3, 4, 5, and 6. Arrow B1 can be used to represent the backward propagation computation of pipeline parallel group 1 for network layers 1 and 2. Arrow B2 can be used to represent the backward propagation computation of pipeline parallel group 2 for network layers 3, 4, 5, and 6. The computer equipment can adjust the chips within the pipeline parallel group to obtain a new pipeline parallel group 1 and a new pipeline parallel group 2. The bit speed (mbs) of the new pipeline parallel group 1 can be 2, and the bit speed of the new pipeline parallel group 2 can be 1. That is, the new pipeline parallel group 1 performs one forward propagation calculation, and the new pipeline parallel group 2 can perform two forward propagation calculations. After the new pipeline parallel group 1 has completed two back propagation calculations, it can be directly passed back to the new pipeline parallel group 1 for one back propagation calculation.

[0122] It is understandable that the variables requiring storage in distributed training mainly include the total number of model parameters, activation parameters, and gradient parameters. The allocation between model parameters and storage can be achieved by adjusting the parallelism of PP (Programming Processing) and TP (Plain Text Processing) parallelism. The storage requirements for activation and gradient parameters are often positively correlated with the size of the input data sequence. In existing technologies, the micro-batch values ​​are identical across all parallel groups, leading to storage resource overhead on some machines in mixed-distribution clusters. This solution decouples these dependencies, allowing different DP, PP, and TP parallel groups to arbitrarily configure micro-batch values ​​while adhering to a multiple constraint. The multiple constraint means that if the megohmmeter (mbs) of the previous pipeline parallel group is 2, the next pipeline parallel group can perform two calculations with an mbs of 1, thus allowing for different mbs configurations. By reasonably configuring the size of the micro-batch values, the storage resources of different machines can be fully utilized, enabling the training of larger models on limited hardware resources. Ultimately, this achieves similar completion times for the corresponding workloads on each chip, realizing load balancing and maximizing the efficiency of hybrid training.

[0123] In this embodiment, all chips are grouped by chip model to obtain initial chip groups consisting entirely of the same chip model. Each initial chip group has a different chip model than the others. Based on the number of learning parameters corresponding to a single network layer in the initial model and the chip models of the M initial chip groups, the chips within each initial chip group are further grouped to obtain P pipelined parallel groups. All chips within a pipelined parallel group are of the same model, and the computational memory corresponding to each pipelined parallel group is greater than or equal to the number of learning parameters. Tensor parallel training of the initial model's network layers is performed using chips of the same model, ensuring that the computation time for tensor parallel training is exactly the same across multiple chips within a pipelined parallel group, thereby significantly reducing training latency. Tensor parallelism refers to dividing the parameters within each network layer of the model and distributing the training tasks of different parameters within the same network layer to multiple chips for completion. Based on load balancing conditions, the average computational power coefficient of the P pipelined parallel groups, the number of network layers in the initial model, and the group computational power coefficients corresponding to the P pipelined parallel groups, the P pipelined parallel groups are grouped and combined to obtain D data parallel groups and their corresponding sample input quantities. The input samples indicated by the sample input quantities corresponding to the D data parallel groups are used to train the initial model configured on the corresponding data parallel group in parallel. The D data parallel groups satisfy the load balancing condition. Pipeline parallelism refers to distributing one or more network layers of the model to multiple pipelined parallel groups for completion, with each pipelined parallel group responsible for processing a portion of the model's network layers. Data parallelism refers to configuring the same initial model on multiple data parallel groups, with each data parallel group independently processing its own input samples, and finally aggregating and integrating all model parameters. By employing the concept of computational power equivalence, and based on the average and group computational power coefficients of the pipeline parallel groups, a task load matching its own group computational power coefficient is allocated to each pipeline parallel group. This ensures effective utilization of the computing resources of the chips within the pipeline parallel group. Multiple pipeline parallel groups are combined into data parallel groups. While ensuring that the computing memory corresponding to each data parallel group is greater than or equal to the total number of learning parameters of the initial model, the sample input volume corresponding to the data parallel group is determined based on the task load allocated to each pipeline parallel group. The task load of the initial model can be split by the data parallel group in terms of sample input size, by the pipeline parallel group in terms of slicing between model layers, and by the tensor parallel training performed by the chips within the pipeline parallel group in terms of the partitioning within the model layers. This ensures that the load of each parallel group is balanced, thereby improving the utilization of computing resources and the training efficiency of the model.

[0124] This application supports flexible, multi-dimensional model parallelism strategies. The training platform can dynamically decompose the training task into multiple sub-tasks based on the characteristics of the training task and the availability of resources, and allocate them to different machine types. Different tensor parallelism degrees can be configured within each pipeline parallel group. Based on the average and group computing power coefficients of the pipeline parallel group, each pipeline parallel group can be assigned a task load matching its own group computing power coefficient. Different pipeline parallel groups can be assigned to different types of data parallel groups. Data parallel groups can be machines or devices equipped with chips within the pipeline parallel groups. Thus, on different types of machines, the number of model layers and the tensor parallelism of the chips within the pipeline parallel groups can be configured according to the machine's hardware characteristics. Alternatively, the number of samples within a global batch can be split, and task balancing of chips with different capabilities can be achieved by inputting data of different scales. Simultaneously, different micro-batch values ​​can be input based on the machine's storage capacity to achieve a balance of computing and storage resources within a single machine. This approach can adapt to different hybrid cluster configurations, effectively improving the efficiency of model training. Based on the core idea of ​​equivalent computational power coefficients, training tasks are allocated by simplifying the allocation problem into an integer programming problem, thus achieving a relatively balanced computational load for each parallel group and enabling efficient utilization of computing resources. This application's embodiments can be widely applied to distributed training platforms based on hybrid machine deployments. This flexible parallel strategy effectively optimizes computational resource utilization, reduces communication overhead, and offers high flexibility and scalability, effectively improving the efficiency and effectiveness of large model training while reducing computational costs.

[0125] Please refer to Figure 8, which is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. As shown in Figure 8, the data processing device 1 includes a chip acquisition module 810, a pipelined parallel grouping module 820, and a data parallel grouping module 830.

[0126] The chip acquisition module 810 is configured to acquire M initial chipsets; M is a positive integer, all chips in an initial chipset are the same model, and the chip models in each initial chipset are different from each other.

[0127] The pipeline parallel grouping module 820 is configured to group the chips in each initial chip group based on the number of learning parameters corresponding to a single network layer in the initial model and the chip models of M initial chip groups, resulting in P pipeline parallel groups; P is a positive integer, the chip models in a pipeline parallel group are all the same, and the computing memory corresponding to each pipeline parallel group is greater than or equal to the number of learning parameters.

[0128] The data parallel grouping module 830 is configured to group and combine the P pipeline parallel groups based on load balancing conditions, the average computing power coefficient of the P pipeline parallel groups, the number of network layers of the initial model, and the group computing power coefficients corresponding to the P pipeline parallel groups, to obtain D data parallel groups and the sample input amount corresponding to the D data parallel groups; D is a positive integer less than P, and the computing memory corresponding to each data parallel group is greater than or equal to the total number of learning parameters of the initial model; the input samples indicated by the sample input amounts corresponding to the D data parallel groups are used to train the initial model configured on the corresponding data parallel groups in parallel.

[0129] In some embodiments, the P pipeline parallel groups include pipeline parallel groups PP. i PP parallel assembly line i Includes S chips, where S is a positive integer; pipelined parallel group PP u Each chip in the system is used to train the network layers of the initial model using tensor parallelism, and the number of parameters allocated to the network layers of the S chips is equal.

[0130] In some embodiments, when the pipeline parallel grouping module 820 is configured to group the chips in each initial chip group based on the number of learning parameters corresponding to a single network layer in the initial model and the chip models of the M initial chip groups to obtain P pipeline parallel groups, it is configured to perform the following operations:

[0131] Obtain the initial grouping parameters corresponding to M initial chipsets respectively; the initial grouping parameters are hyperparameters pre-set based on the learning parameter quantity of a single network layer in the initial model and the chip model of the initial chipset, and all M initial grouping parameters are greater than or equal to 2;

[0132] Based on M initial grouping parameters, the chips in each initial chipset are grouped to obtain N initial parallel groups. Based on the average computing power coefficient of the N initial parallel groups and the group computing power coefficients corresponding to the N initial parallel groups, the chips in each initial parallel group are grouped to obtain P pipeline parallel groups. N is a positive integer, N is greater than M, and N is less than or equal to P.

[0133] In some embodiments, the M initial chipsets include initial chipset K. u Initial chipset K u Includes S target chip models, with an initial chipset K. i The corresponding grouping parameter is G, where G, S, and i are all positive integers; pipeline parallel grouping is used to group the chips in each initial chipset based on M initial grouping parameters, resulting in N initial parallel groups. When this is done, the following operations are configured:

[0134] Based on the initial chipset Ki The grouping parameter G, for the initial chipset K i The chips within are grouped to obtain the initial chipset K. i The corresponding T target parallel groups; initial chipset K i Each corresponding target parallel group includes G target model chips, and the product of T and G is S;

[0135] When the target parallel groups corresponding to the M initial chipsets are obtained, the target parallel groups corresponding to the M initial chipsets are determined as N initial parallel groups; the N initial parallel groups include T target parallel groups.

[0136] In some embodiments, when the pipeline parallel grouping module 820 is configured to group the chips within each initial parallel group based on the average computing power coefficient of N initial parallel groups and the group computing power coefficients corresponding to the N initial parallel groups respectively, resulting in P pipeline parallel groups, it is configured to perform the following operations:

[0137] Obtain the group computing power coefficients and the average computing power coefficients of the N initial parallel groups. Identify the initial parallel groups whose group computing power coefficients are greater than the average computing power coefficients of the N initial parallel groups as groups to be split, with Q being a positive integer. Based on the number of learning parameters, split the chips within each group to be split, obtaining a set of already split groups for each group; Q is a positive integer, and the computing memory corresponding to each split group in the set of already split groups is greater than or equal to the number of learning parameters. Based on the set of already split groups and the NQ initial parallel groups excluding the Q groups to be split, generate A sets of undetermined parallel groups, where A is a positive integer. Obtain the communication evaluation scores corresponding to the A sets of undetermined parallel groups, and identify the set of undetermined parallel groups with the highest communication evaluation score as P pipeline parallel groups; A is a positive integer.

[0138] In some embodiments, the A sets of undetermined parallel groups include the set A of undetermined parallel groups. i The set of undetermined parallel groups A i This includes L undetermined parallel groups, consisting of pre-split groups and initial parallel groups, where i and L are positive integers; the pipeline parallel grouping module 820 is configured to perform the following operations when obtaining the communication evaluation scores corresponding to the A sets of undetermined parallel groups:

[0139] Get the set A of undetermined parallel groups i The communication parameters of the chips in the L undetermined parallel groups and the group computing power coefficients of the L undetermined parallel groups;

[0140] In the L undetermined parallel groups, the chip whose communication parameters are greater than the average communication parameter is identified as the first chip, and the chip whose communication parameters are less than or equal to the average communication parameter is identified as the second chip; the average communication parameter is determined based on the communication parameters of the chip models of the M initial chip groups.

[0141] Obtain the first evaluation parameter and the second evaluation parameter. Based on the first evaluation parameter and the group computing power coefficient of the undetermined parallel group including the first chip, determine the pipeline parallel evaluation score. Based on the second evaluation parameter and the group computing power coefficient of the undetermined parallel group including the second chip, determine the data parallel evaluation score. The sum of the pipeline parallel evaluation score and the data parallel evaluation score is determined as the undetermined parallel group set A. i The communication evaluation score.

[0142] In some embodiments, the P pipeline parallel groups include pipeline parallel groups PP. i PP parallel assembly line i This includes S target chip models, where S is a positive integer; the pipelined parallel grouping module 820 is also configured to perform the following operations:

[0143] Obtain the computing power parameters, storage parameters, bandwidth parameters, and communication parameters of the target chip model;

[0144] Based on pipeline parallel group PP i Based on the number of cards S and communication parameters, determine the intra-group communication coefficient, and consider the service computing power parameters, storage parameters, bandwidth parameters, and pipeline parallel group PP of the target chip model. i The number of cards S is used to determine the business computing power coefficient. Based on the intra-group communication coefficient and the business computing power coefficient, the pipeline parallel group PP is determined. i The group's computing power coefficient.

[0145] In some embodiments, when the data parallel grouping module 830 is configured to group and combine the P pipeline parallel groups based on the average computing power coefficient of the P pipeline parallel groups, the number of network layers of the initial model, and the group computing power coefficients corresponding to the P pipeline parallel groups respectively, to obtain D data parallel groups and the sample input volume corresponding to the D data parallel groups respectively, it is configured to perform the following operations:

[0146] Obtain the initial micro-batch values ​​of the initial model, and generate the target task duration based on the average computing power coefficient for P pipeline parallel groups; the initial micro-batch values ​​are determined based on the task domain of the initial model.

[0147] Based on the initial micro-batch values ​​and total sample size, the data parallel grouping parameters are determined. Based on the target task duration and the group computing power coefficients corresponding to the P pipeline parallel groups, D selection coefficient lists and the initial sample input amounts corresponding to the D allocation parallel groups are determined. Each selection coefficient list includes selection coefficients corresponding to the P pipeline parallel groups. The selection coefficients of the pipeline parallel groups are used to indicate the selection mapping relationship between the pipeline parallel groups and the allocation parallel groups. D is the data parallel grouping parameter.

[0148] Based on the selection coefficients in the selection coefficient list, P pipeline parallel groups are assigned to the corresponding assigned parallel groups to obtain D transition parallel groups containing pipeline parallel groups.

[0149] Based on load balancing conditions, target task duration, D transitional parallel groups, D initial sample inputs, and P pipelined parallel groups, the layer number parameters corresponding to each pipelined parallel group in the D transitional parallel groups and the transitional sample inputs corresponding to each of the D transitional parallel groups are determined. The D transitional parallel groups containing the layer number parameters are determined as D data parallel groups, and the transitional sample inputs corresponding to each of the D transitional parallel groups are determined as the sample inputs corresponding to each of the D data parallel groups.

[0150] In some embodiments, the D allocation parallel groups include allocation parallel groups FP i Where i is a positive integer; when the data parallel grouping module 830 is configured to determine the selection coefficient list and the initial sample input amount corresponding to D allocation parallel groups based on the target task duration and the group computing power coefficients corresponding to P pipeline parallel groups, it is configured to perform the following operations:

[0151] Get the parallel group FP for allocation i The corresponding P undetermined selection parameters are multiplied by the group computing power coefficients corresponding to the P undetermined selection parameters to determine the P first load parameters, and the second load parameters are generated based on the P first load parameters; the P undetermined selection parameters correspond one-to-one with the P pipeline parallel groups;

[0152] Get the parallel group FP for allocation i The corresponding undetermined sample parameters are used to determine the undetermined task duration by the ratio between the second load parameter and the undetermined sample parameters. P undetermined selection parameters and undetermined sample parameters are adjusted to obtain H sets of first parameters; H is a positive integer, and each set of first parameters includes the parameter values ​​of P undetermined selection parameters and the parameter values ​​of undetermined sample parameters.

[0153] One or more sets of first parameters that satisfy the condition of maximizing the ratio are determined as the second parameter set. The second parameter set containing the smallest undetermined sample parameter is determined as the target allocation parameter set. The parameter values ​​of the undetermined sample parameters in the target allocation parameter set are determined as the allocation parallel group FP. i The corresponding initial sample input quantity determines the parameter values ​​of the P undetermined selection parameters in the target allocation parameter set as the allocation parallel group FP. i The corresponding list of selection coefficients; the maximum ratio condition refers to the condition where the ratio between the target task duration and the pending task duration reaches its maximum value.

[0154] In some embodiments, the D transition parallel groups include transition parallel group GP i Transitional parallel group GP i It includes C pipeline parallel groups, where i and C are both positive integers; the data parallel grouping module 830 is configured to determine the layer number parameters corresponding to each pipeline parallel group in the D transition parallel groups and the transition sample input volume corresponding to each of the D transition parallel groups based on load balancing conditions, target task duration, D transition parallel groups, D initial sample input volumes, and the group computing power coefficients corresponding to the P pipeline parallel groups, and to perform the following operations:

[0155] Generate GP for the transitional parallel group based on the group computing power coefficients corresponding to C pipeline parallel groups. i The C undetermined layer number parameters are used to determine the third load parameter by multiplying the undetermined layer number parameter corresponding to each pipeline parallel group with the corresponding group computing power coefficient.

[0156] Based on Transitional Parallel Group GP i The corresponding initial sample input quantity determines the target sample parameter, and the product of the third load parameter and the target sample parameter is determined as the fourth load parameter;

[0157] Based on load balancing conditions, C undetermined layer number parameters and target sample parameters are adjusted to obtain R sets of third parameters; R is a positive integer, and each set of third parameters includes the parameter values ​​of C undetermined layer number parameters and the parameter values ​​of the target sample parameters.

[0158] The set of third parameters that satisfies one or more of the maximum product condition is determined as the fourth parameter set. The fourth parameter set containing the minimum target sample parameter is determined as the target transition parameter set. The parameter values ​​of the target sample parameters in the target transition parameter set are determined as the transition parallel group GP. i The corresponding transition sample input quantity determines the parameter values ​​of the C undetermined layer number parameters in the target transition parameter set as the transition parallel group GP. iThe layer number parameter corresponds to each pipeline parallel group; the maximum product condition refers to the condition where the product of the fourth load parameter and the target task duration reaches its maximum value.

[0159] In some embodiments, the data parallel grouping module 830 is also configured to perform the following operations:

[0160] Based on the sample input volume corresponding to each of the D data parallel groups, the training sample set is divided to obtain the training sample group corresponding to each data parallel group.

[0161] The initial model configured for each data parallel group is trained in parallel based on the training sample group to obtain the sub-model parameters corresponding to each data parallel group. Based on the sub-model parameters corresponding to each data parallel group, the target model is generated.

[0162] In this embodiment, all chips are grouped by chip model to obtain initial chip groups containing only the same chip model. Each initial chip group has a different chip model. Based on the number of learning parameters corresponding to a single network layer in the initial model and the chip models of the M initial chip groups, the chips within each initial chip group are further grouped to obtain P pipelined parallel groups. All chips within a pipelined parallel group are of the same model, and the computational memory corresponding to each pipelined parallel group is greater than or equal to the number of learning parameters. Tensor parallel training of the initial model's network layers is performed using chips of the same model, ensuring that the computation time for tensor parallel training is exactly the same across multiple chips within a pipelined parallel group, thus significantly reducing training latency. Tensor parallelism refers to dividing the parameters within each network layer of the model and distributing the training tasks for different parameters of the same network layer to multiple chips for completion. Based on the average computational power coefficient of the P pipelined parallel groups, the number of network layers in the initial model, and the group computational power coefficients corresponding to the P pipelined parallel groups, the P pipelined parallel groups are further grouped and combined to obtain D data parallel groups and the corresponding sample input amounts for each of the D data parallel groups. In this context, the D data parallel groups each correspond to a specific input sample, used to train the initial model configured on that data parallel group in parallel. The D data parallel groups satisfy a load balancing condition. Pipeline parallelism refers to distributing one or more network layers of the model to multiple pipeline parallel groups for separate processing, with each pipeline parallel group handling a portion of the model's network layers. Data parallelism refers to configuring the same initial model on multiple data parallel groups, with each group independently processing its own input samples, and then finally aggregating and integrating all model parameters. By employing the concept of computational power equivalence, and based on the average and group computational power coefficients of the pipeline parallel groups, a task load matching its own group computational power coefficient is allocated to each pipeline parallel group. This ensures effective utilization of the computing resources of the chips within the pipeline parallel group. Multiple pipeline parallel groups are combined into data parallel groups. While ensuring that the computing memory corresponding to each data parallel group is greater than or equal to the total number of learning parameters of the initial model, the sample input volume corresponding to the data parallel group is determined based on the task load allocated to each pipeline parallel group. The task load of the initial model can be split by the data parallel group in terms of sample input size, by the pipeline parallel group in terms of slicing between model layers, and by the tensor parallel training performed by the chips within the pipeline parallel group in terms of the partitioning within the model layers. This ensures that the load of each parallel group is balanced, thereby improving the utilization of computing resources and the training efficiency of the model.

[0163] This application supports flexible, multi-dimensional model parallelism strategies. The training platform can dynamically decompose the training task into multiple sub-tasks based on the characteristics of the training task and the availability of resources, and allocate them to different machine types. Different tensor parallelism degrees can be configured within each pipeline parallel group. Based on the average and group computing power coefficients of the pipeline parallel group, each pipeline parallel group can be assigned a task load matching its own group computing power coefficient. Different pipeline parallel groups can be assigned to different types of data parallel groups. Data parallel groups can be machines or devices equipped with chips within the pipeline parallel groups. Thus, on different types of machines, the number of model layers and the tensor parallelism of the chips within the pipeline parallel groups can be configured according to the machine's hardware characteristics. Alternatively, the number of samples within a global batch can be split, and task balancing of chips with different capabilities can be achieved by inputting data of different scales. Simultaneously, different micro-batch values ​​can be input based on the machine's storage capacity to achieve a balance of computing and storage resources within a single machine. This approach can adapt to different hybrid cluster configurations, effectively improving the efficiency of model training. Based on the core idea of ​​equivalent computational power coefficients, training tasks are allocated by simplifying the allocation problem into an integer programming problem, thus achieving a relatively balanced computational load for each parallel group and enabling efficient utilization of computing resources. This application's embodiments can be widely applied to distributed training platforms based on hybrid machine deployments. This flexible parallel strategy effectively optimizes computational resource utilization, reduces communication overhead, and offers high flexibility and scalability, effectively improving the efficiency and effectiveness of large model training while reducing computational costs.

[0164] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0165] Please refer to Figure 9, which is a schematic diagram of the structure of a computer device provided in an embodiment of this application. As shown in Figure 9, the computer device 1000 may include: a processor 1001, a network interface 1004, and a memory 1005. Furthermore, the computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. The communication bus 1002 is used to realize communication between these components. The user interface 1003 may include a display screen and a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage device. Optionally, the memory 1005 may also be at least one storage device located remotely from the aforementioned processor 1001. As shown in Figure 9, the memory 1005, as a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application program.

[0166] In the computer device 1000 shown in Figure 9, the network interface 1004 provides network communication elements; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0167] Obtain M initial chipsets; M is a positive integer, all chips in an initial chipset are the same model, and the chip models of each initial chipset are different from each other.

[0168] Based on the number of learning parameters corresponding to a single network layer in the initial model and the chip models of the M initial chip groups, the chips in each initial chip group are grouped to obtain P pipeline parallel groups; P is a positive integer, the chip models in a pipeline parallel group are all the same, and the computing memory corresponding to each pipeline parallel group is greater than or equal to the number of learning parameters.

[0169] Based on load balancing conditions, the average computing power coefficient of P pipeline parallel groups, the number of network layers of the initial model, and the group computing power coefficients corresponding to the P pipeline parallel groups, the P pipeline parallel groups are grouped and combined to obtain D data parallel groups and the sample input amount corresponding to each of the D data parallel groups; D is a positive integer less than P, and the computing memory corresponding to each data parallel group is greater than or equal to the total number of learning parameters of the initial model; the input samples indicated by the sample input amounts corresponding to the D data parallel groups are used to train the initial model configured on the corresponding data parallel groups in parallel.

[0170] It should be understood that the computer device 1000 described in the embodiments of this application can execute the data processing method described in any of the embodiments corresponding to Figures 3 and 4 above. Furthermore, the beneficial effects of using the same method will not be repeated here.

[0171] Furthermore, it should be noted that this application also provides a computer-readable storage medium storing a computer program. When the processor executes the computer program, it can perform the data processing method described in any of the embodiments corresponding to Figures 3 and 4 above. Therefore, it will not be repeated here. Additionally, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the embodiments of the computer-readable storage medium involved in this application, please refer to the description of the method embodiments of this application.

[0172] The aforementioned computer-readable storage medium can be an internal storage unit of the data processing apparatus or computer device provided in any of the foregoing embodiments, such as a hard disk or memory of the computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been displayed or will be displayed.

[0173] Furthermore, it should be noted that this application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the method provided in any of the embodiments corresponding to Figures 3 and 4 above.

[0174] The terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0175] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the foregoing description as a network element. Whether these network elements are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can implement the described network elements using different methods for each specific application, but such implementation should not be considered beyond the scope of this application.

[0176] The methods and related apparatus provided in this application are described with reference to the method flowcharts and / or structural diagrams provided in this application. Specifically, each block of the method flowcharts and / or structural diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable device, create means for implementing the functions specified in one or more blocks of the flowcharts and / or one or more blocks of the structural diagrams. These computer program instructions can also be stored in a computer-readable storage medium capable of directing a computer or other programmable device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more blocks of the flowcharts and / or one or more blocks of the structural diagrams. These computer program instructions may also be loaded onto a computer or other programmable device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable device, provide steps for implementing the functions specified in one or more flowcharts and / or one or more blocks in the structural diagram.

[0177] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0178] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0179] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

A data processing method, applied to a computer device, includes: Obtain M initial chipsets; M is a positive integer. All chips in an initial chipset are the same model, and the chip models in each initial chipset are different from each other. Based on the number of learning parameters corresponding to a single network layer in the initial model and the chip models of the M initial chip groups, the chips in each initial chip group are grouped to obtain P pipeline parallel groups. P is a positive integer, all chips in a pipeline parallel group are of the same model, and the computing memory corresponding to each pipeline parallel group is greater than or equal to the amount of learning parameters. Based on load balancing conditions, the average computing power coefficient of the P pipeline parallel groups, the number of network layers of the initial model, and the group computing power coefficients corresponding to the P pipeline parallel groups, the P pipeline parallel groups are combined to obtain D data parallel groups and the sample input quantities corresponding to the D data parallel groups; D is a positive integer less than P, and the computing memory corresponding to each data parallel group is greater than or equal to the total number of learning parameters of the initial model; the input samples indicated by the sample input quantities corresponding to the D data parallel groups are used to train the initial model configured on the corresponding data parallel groups in parallel. According to the method of claim 1, wherein, The P parallel pipeline groups include pipeline parallel groups PP i The parallel assembly line PP i It includes S chips, where S is a positive integer; the pipeline parallel group PP i Each chip in the model is used to train the network layers of the initial model in tensor parallelism, and the number of parameters allocated to the network layers of the S chips is equal. The method according to claim 1 or 2, wherein, Based on the number of learning parameters corresponding to a single network layer in the initial model and the chip models of the M initial chip groups, the chips within each initial chip group are grouped to obtain P pipeline parallel groups, including: Obtain the initial grouping parameters corresponding to the M initial chipsets respectively; the initial grouping parameters are hyperparameters pre-set based on the learning parameter quantity of a single network layer in the initial model and the chip model of the initial chipset, and all M initial grouping parameters are greater than or equal to 2; Based on M initial grouping parameters, the chips in each initial chipset are grouped separately to obtain N initial parallel groups; Based on the average computing power coefficient of the N initial parallel groups and the group computing power coefficients corresponding to the N initial parallel groups, the chips in each initial parallel group are grouped to obtain P pipeline parallel groups; N is a positive integer, N is greater than M, and N is less than or equal to P. The method according to any one of claims 1 to 3, wherein, The M initial chipsets include initial chipset K. i The initial chipset K i The initial chipset includes S target models of chips. i The corresponding grouping parameter is G, where G and S are both positive integers; based on the M initial grouping parameters, the chips within each initial chipset are grouped to obtain N initial parallel groups, including: Based on the initial chipset K i The grouping parameter G, for the initial chipset K i The chips within are grouped to obtain the initial chipset K. i The corresponding T target parallel groups; the initial chipset K i Each corresponding target parallel group includes G target model chips, and the product of T and G is S; When the target parallel groups corresponding to the M initial chipsets are obtained, the target parallel groups corresponding to the M initial chipsets are determined as N initial parallel groups; the N initial parallel groups include the T target parallel groups. The method according to any one of claims 1 to 4, wherein, Based on the average computing power coefficient of the N initial parallel groups and the group computing power coefficients corresponding to the N initial parallel groups, the chips within each initial parallel group are grouped to obtain P pipeline parallel groups, including: Obtain the group computing power coefficients corresponding to the N initial parallel groups and the average computing power coefficients of the N initial parallel groups respectively; The initial parallel groups whose computing power coefficient is greater than the average computing power coefficient of the N initial parallel groups are determined as the groups to be split, and the number of the groups to be split is Q, where Q is a positive integer. Based on the amount of learning parameters, the chips in each group to be split are split separately to obtain a set of split groups corresponding to each group to be split; the computing memory corresponding to each split group in the set of split groups is greater than or equal to the amount of learning parameters. Based on the already split group set and NQ initial parallel groups excluding the Q groups to be split, generate A sets of undetermined parallel groups, where A is a positive integer; Obtain the communication evaluation scores corresponding to the A sets of undetermined parallel groups, and determine the set of undetermined parallel groups with the largest communication evaluation score as the P pipeline parallel groups. The method according to any one of claims 1 to 5, wherein, The set of A undetermined parallel groups includes the set A of undetermined parallel groups. i The set of undetermined parallel groups A i This includes L undetermined parallel groups, consisting of pre-split groups and initial parallel groups, where i and L are positive integers; obtaining the communication evaluation scores corresponding to the A undetermined parallel group sets respectively includes: Obtain the set A of the undetermined parallel groups i The communication parameters corresponding to the chips in the L undetermined parallel groups and the group computing power coefficients corresponding to the L undetermined parallel groups; Chips in the L undetermined parallel groups whose communication parameters are greater than the average communication parameter value are identified as first chips, and chips in the L undetermined parallel groups whose communication parameters are less than or equal to the average communication parameter value are identified as second chips; the average communication parameter value is determined based on the communication parameters of the chip models of the M initial chip groups; Obtain the first evaluation parameter and the second evaluation parameter, and determine the pipeline parallel evaluation score based on the first evaluation parameter and the group computing power coefficient of the undetermined parallel group including the first chip. Based on the second evaluation parameters and the group computing power coefficient of the undetermined parallel group including the second chip, the data parallel evaluation score is determined; The sum of the pipeline parallel evaluation score and the data parallel evaluation score is determined as the set A of undetermined parallel groups. i The communication evaluation score. The method according to any one of claims 1 to 6, wherein, The P parallel pipeline groups include pipeline parallel groups PP i The parallel assembly line PP i The method includes S target chip models, where S is a positive integer; the method further includes: Obtain the service computing power parameters, storage parameters, bandwidth parameters, and communication parameters of the target model chip; Based on the pipeline parallel group PP i The number of cards S and the communication parameters are used to determine the communication coefficient within the group; Based on the target model's chip's service computing power parameters, storage parameters, bandwidth parameters, and the pipeline parallel group (PP) i The number of cards S determines the business computing power coefficient; Based on the intra-group communication coefficient and the service computing power coefficient, the pipeline parallel group PP is determined. i The group's computing power coefficient. The method according to any one of claims 1 to 7, wherein, Based on load balancing conditions, the average computing power coefficient of the P pipelined parallel groups, the number of network layers in the initial model, and the group computing power coefficients corresponding to the P pipelined parallel groups, the P pipelined parallel groups are grouped and combined to obtain D data parallel groups and the sample input quantities corresponding to the D data parallel groups, including: Obtain the initial micro-batch values ​​of the initial model, and based on the initial micro-batch values, determine the target task duration for the average computing power coefficient of the P pipeline parallel groups; the initial micro-batch values ​​are determined based on the task domain of the initial model. Based on the initial micro-batch values ​​and the total sample size, the data parallel grouping parameter D is determined; Based on the target task duration and the group computing power coefficients corresponding to the P pipeline parallel groups, D selection coefficient lists and D initial sample input quantities corresponding to the allocation parallel groups are generated; each selection coefficient list includes selection coefficients corresponding to the P pipeline parallel groups, and the selection coefficients of the pipeline parallel groups are used to indicate the selection mapping relationship between the pipeline parallel groups and the allocation parallel groups; D is the data parallel grouping parameter; Based on the selection coefficients in the selection coefficient list, the P pipeline parallel groups are assigned to the corresponding assigned parallel groups to obtain D transition parallel groups containing the pipeline parallel groups. Based on the load balancing conditions, the target task duration, the D transitional parallel groups, the D initial sample inputs, and the group computing power coefficients corresponding to the P pipelined parallel groups, the layer number parameters corresponding to each pipelined parallel group in the D transitional parallel groups, and the transitional sample inputs corresponding to the D transitional parallel groups are determined. The D transition parallel groups containing the layer number parameter are determined as D data parallel groups, and the transition sample input amount corresponding to each of the D transition parallel groups is determined as the sample input amount corresponding to each of the D data parallel groups. The method according to any one of claims 1 to 8, wherein, The D parallel allocation groups include parallel allocation group FP i Where i is a positive integer; the step of generating a selection coefficient list and D initial sample input quantities corresponding to each of the P pipeline parallel groups based on the target task duration and the group computing power coefficients of each of the P pipeline parallel groups includes: Get the parallel group FP for allocation i The corresponding P undetermined selection parameters are multiplied by the group computing power coefficients corresponding to the P undetermined selection parameters to determine the P first load parameters; The second load parameters are generated based on the P first load parameters; the P undetermined selection parameters correspond one-to-one with the P pipeline parallel groups. Get the parallel group FP for allocation i The corresponding undetermined sample parameters are used to determine the undetermined task duration by the ratio between the second load parameter and the undetermined sample parameters. The P undetermined selection parameters and the undetermined sample parameters are adjusted to obtain H sets of first parameters; H is a positive integer, and each set of first parameters includes the parameter values ​​of the P undetermined selection parameters and the parameter values ​​of the undetermined sample parameters. One or more sets of first parameters that satisfy the maximum ratio condition are determined as the second parameter set, and the second parameter set containing the smallest undetermined sample parameter is determined as the target allocation parameter set; the maximum ratio condition refers to the condition that the ratio between the target task duration and the undetermined task duration reaches its maximum value. The parameter values ​​of the undetermined sample parameters in the target allocation parameter set are determined as the allocation parallel group FP. i The corresponding initial sample input quantity determines the parameter values ​​of the P undetermined selection parameters in the target allocation parameter set as the allocation parallel group FP. i The corresponding list of selection coefficients. The method according to any one of claims 1 to 9, wherein, The D transition parallel groups include transition parallel group GP i The transition parallel group GP i It includes C pipelined parallel groups, where i and C are both positive integers; the determination of the layer number parameter corresponding to each pipelined parallel group in the D transitional parallel groups, and the transitional sample input amount corresponding to each of the D transitional parallel groups, based on load balancing conditions, the target task duration, the D transitional parallel groups, the D initial sample input amounts, and the group computing power coefficients corresponding to the P pipelined parallel groups, includes: Based on the group computing power coefficients corresponding to the C pipeline parallel groups, a GP is generated for the transition parallel group. i The C undetermined layer number parameters are used to determine the third load parameter by multiplying the undetermined layer number parameter corresponding to each pipeline parallel group with the corresponding group computing power coefficient. Based on the transition parallel group GP i The corresponding initial sample input quantity determines the target sample parameter, and the product of the third load parameter and the target sample parameter is determined as the fourth load parameter; Based on load balancing conditions, the C undetermined layer number parameters and the target sample parameters are adjusted to obtain R sets of third parameters; R is a positive integer, and each set of third parameters includes the parameter values ​​of the C undetermined layer number parameters and the parameter values ​​of the target sample parameters. The set of third parameters that satisfies one or more of the maximum product condition is determined as the fourth parameter set, and the set of fourth parameters containing the minimum target sample parameter is determined as the target transition parameter set; the maximum product condition refers to the condition that the product of the fourth load parameter and the target task duration reaches the maximum value. The parameter values ​​of the target sample parameters in the target transition parameter set are determined as the transition parallel group GP. i The corresponding transition sample input quantity determines the parameter values ​​of the C undetermined layer number parameters in the target transition parameter set as the transition parallel group GP. i The layer number parameter corresponding to each pipeline parallel group. The method according to any one of claims 1 to 10, wherein, The method further includes: Based on the sample input volume corresponding to the D data parallel groups, the training sample set is divided to obtain the training sample group corresponding to each data parallel group. Based on the training sample group, the initial model configured for each data parallel group is trained in parallel to obtain the sub-model parameters corresponding to each data parallel group. The target model is generated based on the sub-model parameters corresponding to each data parallel group. A data processing apparatus, comprising: The chip acquisition module is configured to acquire M initial chipsets; M is a positive integer. All chips in an initial chipset are the same model, and the chip models in each initial chipset are different from each other. The pipeline parallel grouping module is configured to group the chips in each initial chip group based on the number of learning parameters corresponding to a single network layer in the initial model and the chip models of the M initial chip groups, to obtain P pipeline parallel groups. P is a positive integer, all chips in a pipeline parallel group are of the same model, and the computing memory corresponding to each pipeline parallel group is greater than or equal to the amount of learning parameters. The data parallel grouping module is configured to group and combine the P pipeline parallel groups based on load balancing conditions, the average computing power coefficient of the P pipeline parallel groups, the number of network layers of the initial model, and the group computing power coefficients corresponding to the P pipeline parallel groups, to obtain D data parallel groups and the sample input amount corresponding to the D data parallel groups; D is a positive integer less than P, and the computing memory corresponding to each data parallel group is greater than or equal to the total number of learning parameters of the initial model; the input samples indicated by the sample input amounts corresponding to the D data parallel groups are used to train the initial model configured on the corresponding data parallel groups in parallel. A computer device, comprising: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide data communication functions, the memory is used to store computer programs, and the processor is used to invoke the computer programs to cause the computer device to perform the method according to any one of claims 1 to 11. A computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1 to 11. A computer program product comprising a computer program stored in a computer-readable storage medium and adapted to be read and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1 to 11.

Images