Method and apparatus for parallel processing of data

By breaking down data processing tasks into several sub-tasks and having them processed in parallel by multiple computing devices, the problem of excessive communication overhead is solved, and the utilization rate of computing power and data processing efficiency are improved.

WO2026233319A1PCT designated stage Publication Date: 2026-11-12SHANGHAI XIYU TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2026/094942
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-09
Filing Date
2026-05-06
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

In existing technologies, communication overhead accounts for too much of the data processing process, resulting in a waste of computing resources and a reduction in the utilization rate of computing equipment.

Method used

The data processing task is broken down into several subtasks, and parallel processing is performed by at least two parallel computing devices. Subtasks are executed serially within the same stage, and subtasks in different stages are processed in parallel.

Benefits of technology

Reduce the idle time of computing devices, shorten the waiting time during the communication phase, improve the utilization rate of computing devices, and ensure the correct execution of data processing tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2026094942_12112026_PF_FP_ABST
    Figure CN2026094942_12112026_PF_FP_ABST
Patent Text Reader

Abstract

Provided in the present disclosure are a method and apparatus for the parallel processing of data, which are implemented by means of at least two mutually parallel computing devices, wherein a data processing task comprises tasks of at least two stages. The method comprises: splitting tasks of each of at least two stages into several sub-tasks; and on the basis of at least two mutually parallel computing devices, implementing parallel processing of data processing tasks, wherein the parallel processing of the data processing tasks comprises: within the same stage, sequentially performing the sub-tasks of the stage in a serial manner by means of the at least two computing devices; and performing parallel processing on sub-tasks of different stages by means of the at least two computing devices. In this way, the waiting time of a computing device in a communication stage can be reduced, and the total time consumed for data processing can be decreased, allowing more time for actual inference computation, and thereby improving the utilization rate of a computing power device.
Need to check novelty before this filing date? Find Prior Art

Description

A data parallel processing method and apparatus

[0001] Cross-references to related applications

[0002] This disclosure claims priority to Chinese Patent Application No. 202510597570.1, filed on May 9, 2025, entitled "A Data Parallel Processing Method and Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to the field of computer technology, and in particular to a data parallel processing method and apparatus. Background Technology

[0004] Currently, in related technologies, communication overhead accounts for too large a proportion of the data processing process. During a single processing run, most of the limited resources of the computing device are used for communication, with only a small portion dedicated to actual data inference. In today's world of limited computing resources, this data processing method significantly wastes computing power and reduces the utilization rate of computing devices. Summary of the Invention

[0005] In view of this, the purpose of this disclosure is to provide a data parallel processing method and apparatus to solve the problem that excessive communication overhead in the data processing process in the prior art leads to a waste of computing resources.

[0006] This disclosure provides a data parallel processing method implemented by at least two parallel computing devices. The data processing task includes at least two stages. The method includes:

[0007] The tasks of the at least two stages are each divided into several sub-tasks, and the data processing tasks are parallelized based on the at least two parallel computing devices.

[0008] Parallel processing of data processing tasks includes: within the same stage, at least two computing devices sequentially execute the sub-tasks of that stage in a serial manner; and between sub-tasks in different stages, the at least two computing devices process them in parallel.

[0009] Optionally, the method is based on a hybrid expert model, which includes at least two expert models, each expert model includes at least two computing devices, and the number of computing devices in each expert model is the same. The number of sub-tasks obtained by splitting the task at each stage corresponds to the total number of computing devices in all expert models.

[0010] All computing devices in the hybrid expert model process data processing tasks in parallel.

[0011] Optionally, the tasks of the at least two stages can each be divided into several sub-tasks, including:

[0012] Obtain the task splitting dimension, and based on the task splitting dimension, transform the weight matrix of the expert model into at least two sub-weight matrices;

[0013] The sub-weight matrices are distributed one by one to the computing devices in the corresponding expert model. The number of sub-weight matrices corresponds to the number of computing devices in the expert model. Each computing device performs parallel processing of data processing tasks based on its corresponding sub-weight matrix.

[0014] Optionally, parallel processing of data processing tasks may also include:

[0015] While executing multiple subtasks of any target stage sequentially, if the target stage is not the final stage, in response to the first computing device completing the first subtask in the target stage, the first computing device starts executing the first subtask in the next stage, and continues to execute multiple subtasks for the next stage sequentially.

[0016] Optionally, the at least two stages include at least one pre-communication stage, an inference stage, and at least one post-communication stage; the at least one pre-communication stage and the at least one post-communication stage are symmetrically distributed about the inference stage;

[0017] If the target stage is not the final stage, then in response to the first computing device completing the first subtask in the target stage, the first computing device initiates the execution of the first subtask in the next stage, including:

[0018] If the target stage is the inference stage, then in response to the first computing device completing the first inference subtask in the target stage, the execution progress of the preceding pre-communication stage of the inference stage is determined.

[0019] When the execution progress of the pre-communication phase is such that the last pre-communication subtask has been completed, the first computing device initiates the execution of the first inference subtask in the next post-communication phase of the inference phase.

[0020] Optionally, when the communication times between the at least one pre-communication stage and / or between the at least one post-communication stage are not equal, within the same stage, the at least two computing devices sequentially execute the sub-tasks of that stage in a serial manner, including:

[0021] For computing device N and stage M, in response to computing device N-1 having completed the subtask Mi corresponding to stage M, and computing device N having completed the subtask M-1i+1 corresponding to stage M-1, computing device N starts executing the subtask Mi+1 corresponding to stage M; N, M, and i are all positive integers.

[0022] Optionally, if the total number of computing devices is D, and the cost of any stage is t, then the cost T of each subtask in that stage is: This indicates a rounding up operation; when the cost t of this stage is not an integer multiple of the total number of computing devices D, the tasks of the at least two stages are each divided into several sub-tasks, including:

[0023] At the end of this stage, data is filled in so that the length of each subtask is T, and the length of the filled data is L = D × Tt.

[0024] Optionally, parallel processing of data processing tasks also includes:

[0025] When the computing device executes subtasks in parallel, the padding data portion of the subtasks is skipped.

[0026] Optionally, the various stages in the data processing task include, in sequence: the data splitting pre-communication stage between different expert models, the data splitting pre-communication stage between different computing devices within an expert model, the inference stage of each computing device, the data set post-communication stage between different computing devices within each expert model, and the data set post-communication stage between different expert models.

[0027] This disclosure also provides a data parallel processing apparatus, implemented by at least two parallel computing devices, wherein the data processing task includes at least two stages, and the apparatus includes:

[0028] The splitting module is configured to split each of the at least two phases of tasks into several sub-tasks;

[0029] The parallel processing module is configured to perform parallel processing of data processing tasks based on the at least two parallel computing devices.

[0030] Parallel processing of data processing tasks includes: within the same stage, at least two computing devices sequentially execute the sub-tasks of that stage in a serial manner; and between sub-tasks in different stages, the at least two computing devices process them in parallel.

[0031] This disclosure also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the data parallel processing method described above are performed.

[0032] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the data parallel processing method described above.

[0033] This disclosure provides a data parallel processing method and apparatus that divides a task with at least two stages into several sub-tasks and performs parallel processing of the data processing tasks based on at least two parallel computing devices. In this way, different computing devices can process sub-tasks of different stages in parallel at the same time, thereby significantly reducing the idle time of computing devices, shortening the waiting time of computing devices during the communication stage, reducing the total data processing time, and allowing more time to be devoted to actual inference calculations, thus improving the utilization rate of computing power. Simultaneously, sub-tasks of the same stage are executed sequentially in a serial manner, which is simple and helps ensure the correct execution of data processing tasks.

[0034] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0035] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this disclosure and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 illustrates the total time consumption of a data processing procedure in the prior art provided by an embodiment of this disclosure;

[0037] Figure 2 shows a flowchart of a data parallel processing method provided in an embodiment of this disclosure;

[0038] Figure 3 shows a schematic diagram of the breakdown of a data processing task provided in an embodiment of this disclosure;

[0039] Figure 4 illustrates one of the total time consumption diagrams for parallel data processing provided in an embodiment of this disclosure;

[0040] Figure 5 shows a schematic diagram of a weight matrix for splitting an expert model provided in an embodiment of this disclosure;

[0041] Figure 6 shows a second schematic diagram illustrating the total time consumption of parallel data processing according to an embodiment of this disclosure;

[0042] Figure 7 illustrates the third schematic diagram of the total time consumption of parallel data processing provided in an embodiment of this disclosure;

[0043] Figure 8 shows a schematic diagram of the structure of a data parallel processing apparatus provided in an embodiment of the present disclosure;

[0044] Figure 9 shows a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0045] The preferred embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, so that the advantages and features of this disclosure can be more readily understood by those skilled in the art.

[0046] The above embodiments are only for illustrating the technical concept and features of this disclosure, and are intended to enable those skilled in the art to understand the content of this disclosure and implement it accordingly. They should not be construed as limiting the scope of protection of this disclosure. All equivalent changes or modifications made in accordance with the spirit and essence of this disclosure should be included within the scope of protection of this disclosure.

[0047] Research has revealed that in current related technologies, communication overhead accounts for an excessive proportion of the data processing process. During a single processing run, the limited resources of the computing device are mostly used for communication, with only a small portion dedicated to actual data inference. In classic data processing, communication and actual inference are processed sequentially. As shown in Figure 1, which is a schematic diagram of the total time consumption of a data processing process in a prior art according to an embodiment of this disclosure, the total time consumed by a data processing task is TA = T1a + T2a + Ta + T3a + T4a. However, only Ta is actually used for inference computation, with the vast majority of time (T1a, T2a, T3a, and T4a) spent on communication. In today's world of limited computing resources, this data processing method significantly wastes computing resources and reduces the utilization rate of computing devices.

[0048] Based on this, the present disclosure provides a data parallel processing method and apparatus to solve the problem that excessive communication overhead in the data processing process in the prior art leads to a waste of computing resources.

[0049] The data parallel processing method provided in this disclosure is implemented by at least two parallel computing devices; parallelism means that these computing devices simultaneously execute data processing tasks, which include at least two stages. The implementation details of this data parallel processing method will be described in detail below with reference to specific illustrations.

[0050] Please refer to Figure 2, which is a flowchart of a data parallel processing method provided in an embodiment of this disclosure. As shown in Figure 2, the data parallel processing method provided in this embodiment includes:

[0051] S101. Divide the tasks of the at least two stages into several sub-tasks.

[0052] Here, the number of task breakdowns is determined by the total number of computing devices. Generally, the number of computing devices is a power of 2, such as 2, 4, 8, and 16. Accordingly, the task at each stage can be broken down into subtasks such as 2, 4, 8, and 16.

[0053] Figure 3 is a schematic diagram illustrating the breakdown of a data processing task according to an embodiment of this disclosure. As shown in Figure 3, the data processing task sequentially includes tasks in communication stage 1, communication stage 2, inference stage, communication stage 3, and communication stage 4. In a scenario where the number of devices is evenly distributed (4), the task in each stage is divided into 4 sub-tasks, and each sub-task is executed by a corresponding computing device. For example, sub-task labeled 1 is executed by device 0, sub-task labeled 2 is executed by device 1, sub-task labeled 3 is executed by device 2, and sub-task labeled 4 is executed by device 3. The labels of the devices and sub-tasks are only for distinguishing purposes and have no practical limiting meaning.

[0054] S102. Based on the at least two parallel computing devices, implement parallel processing of data processing tasks.

[0055] Parallel processing of data processing tasks includes: within the same stage, at least two computing devices sequentially execute the sub-tasks of that stage in a serial manner; and between sub-tasks in different stages, the at least two computing devices process them in parallel.

[0056] Figure 4 is one of the schematic diagrams illustrating the total time consumption of parallel data processing provided in an embodiment of this disclosure. As shown in Figure 4, the total time consumption of the data processing task is TA'. Within the same stage, at least two computing devices sequentially execute multiple sub-tasks included in that stage in a serial manner. For example, the four sub-tasks split from communication stage 1 are executed sequentially by devices 0, 1, 2, and 3, respectively. Simultaneously, after completing the sub-task of the current stage, each computing device can continue to execute the sub-task of the next stage, so that the sub-tasks of different stages are processed in parallel by at least two computing devices at the same time. For example, at a certain moment, device 0 executes sub-task 1 of the inference stage, while device 1 executes sub-task 2 of communication stage 2, and device 2 executes sub-task 3 of communication stage 1.

[0057] In this way, the idle time of computing devices can be significantly reduced, and each device can process services at different communication stages simultaneously, thereby shortening the waiting time during communication stages and reducing the total data processing time. A comparison of Figures 1 and 4 shows that the method provided in this embodiment can significantly shorten the total data processing time. This allows more time to be devoted to actual inference calculations, maximizing the proportion of time that computing power devices spend on actual inference calculations, thus improving the utilization rate of computing power devices.

[0058] In one possible implementation, the data parallel processing method can be based on a Mixture of Experts (MoE), a distributed computing technology that is deployed on a distributed cluster for parallel computing, thereby solving the problem of insufficient video memory or limited computing power of a single device.

[0059] The hybrid expert model in this embodiment includes at least two parallel expert models. Here, for different data processing tasks, the hybrid expert model can activate some or all of the expert models to perform the task as needed. Specific methods can be found in the prior art, and will not be elaborated further here.

[0060] Each expert model includes at least two parallel computing devices, and the number of computing devices in each expert model is the same. The number of sub-tasks obtained from the task breakdown at each stage corresponds to the total number of computing devices in all expert models. All computing devices in the hybrid expert model process data processing tasks in parallel, and the parallel processing method can be referred to the description in step S102 above.

[0061] In this way, the embodiments of this disclosure, based on expert parallelism, further realize parallel processing of data processing tasks within each expert model, which is more suitable for large-scale computations that need to process ultra-large-scale data, shortens the waiting time in the communication stage, reduces the total time spent on data processing, and allows more time to be used for actual large-scale computations, thereby improving the utilization rate of computing power equipment.

[0062] In specific implementation, step S101 may include:

[0063] S1011. Obtain the task splitting dimension, and based on the task splitting dimension, transform the weight matrix of the expert model into at least two sub-weight matrices.

[0064] The task splitting dimension here can include either row or column dimensions; preferably, considering the operational characteristics of matrix multiplication, the task splitting dimension is generally column-oriented. For example, the weight matrix W of the expert model can be split column-wise into N sub-weight matrices W1 to WN, each sub-weight matrix having the same shape. Figure 5 is a schematic diagram of splitting the weight matrix of an expert model according to an embodiment of this disclosure. As shown in Figure 5, when the total number of computing devices is 4, splitting the 8×8 weight matrix W column-wise yields four 8×2 sub-weight matrices.

[0065] S1012. Distribute the sub-weight matrices one by one to the computing devices in the corresponding expert model. The number of sub-weight matrices corresponds to the number of computing devices in the expert model. Each computing device performs parallel processing of data processing tasks according to its corresponding sub-weight matrix.

[0066] In this step, each sub-weight matrix is ​​distributed to the corresponding computing device within the expert model. This allows the computation of a single expert model to be split across N computing devices. The computation process is Y = X × W = X × W1 + X × W2 + ... + X × WN, where X is the input matrix and Y is the output result.

[0067] Corresponding to the example above, each expert model includes four parallel computing devices; for any expert model, the weight matrix of the expert model is split into four sub-weight matrices and distributed to the four computing devices one by one, and each computing device performs parallel processing of data processing tasks according to the sub-weight matrix to which it is distributed.

[0068] Building upon the column-based splitting described above, this disclosure also provides an alternative for row-based splitting. The following example illustrates this using a single expert model containing four computing devices.

[0069] When the weight matrix W is split along the row dimension, the 8×8 weight matrix W is divided into four 2×8 sub-weight matrices W1 to W4, which are then distributed to four computing devices. Unlike column-dimensional splitting, the computation process under row-dimensional splitting requires a corresponding splitting of the input matrix X. Specifically, the input matrix X is split into four sub-inputs X1 to X4 along the column dimension and distributed to four computing devices. Each computing device k performs a local matrix multiplication Yk = Xk × Wk, obtaining a local result Yk. Since each computing device only participates in the calculation of a portion of the input and weights, the dimension of each local result Yk is the same as the dimension of the final output Y, but it contains only a partial contribution. Therefore, an all-reduce operation is needed to sum the local results of all computing devices: Y = Y1 + Y2 + Y3 + Y4. The communication mode of row-dimensional splitting includes: before computation, the input matrix needs to be distributed to each device column-wise; after computation, the outputs of each device need to be summed using an all-reduce operation. Compared to column-based partitioning, row-based partitioning can achieve better communication efficiency when bandwidth utilization is high in all-reduce operations.

[0070] In practice, row-dimensional splitting or column-dimensional splitting can be selected based on the shape of the weight matrix, the network topology, and the actual bandwidth of various aggregate communication operations to obtain the optimal overall performance.

[0071] In specific implementation, step S102 may also include:

[0072] Step a: While sequentially executing multiple subtasks of any target stage, if the target stage is not the final stage, in response to the first computing device completing the first subtask in the target stage, the first computing device starts executing the first subtask in the next stage, and continues to implement the sequential execution of multiple subtasks for the next stage.

[0073] In this step, for any target stage that is not the final stage in at least two stages, while at least two computing devices are executing multiple subtasks of the target stage in a serial manner, when the first computing device finishes executing the first subtask in the target stage, the first computing device will continue to start executing the corresponding subtask in the next stage of the target stage, that is, the first subtask of the next stage; thereafter, in the next stage, at least two computing devices continue to execute the subtasks of the stage in a serial manner.

[0074] Referring to Figure 4, for example, with communication phase 2 as the target phase, the four computing devices sequentially execute the four sub-tasks of communication phase 2. When device 0 completes the first sub-task 1 of communication phase 2, device 0 will continue to start executing the next phase, namely the first sub-task 1 of the inference phase; and the four computing devices will continue to implement the sequential execution of the four sub-tasks for the inference phase.

[0075] Furthermore, the multiple stages include at least one pre-communication stage, an inference stage, and at least one post-communication stage; the at least one pre-communication stage and the at least one post-communication stage are symmetrically distributed about the inference stage.

[0076] Here, due to the symmetry of the sending and receiving processes in communication, among at least one preceding communication stage and at least one following communication stage that are symmetrically distributed with respect to the inference stage, the corresponding preceding and following communication stages have the same communication time and corresponding communication types; where "corresponding location" means that the number of communication stages separated from the center of symmetry of the inference stage is the same. "Corresponding communication type" means that the data flow in the communication is opposite, such as aggregation and distribution.

[0077] Corresponding to the above example, communication stage 1, which is the preceding communication stage, and communication stage 4, which is the following communication stage, are symmetrical about the reasoning stage. Communication stage 1 and communication stage 4 have the same time consumption, i.e., T1a = T4a. Communication stage 2, which is the preceding communication stage, and communication stage 3, which is the following communication stage, are symmetrical about the reasoning stage. Communication stage 2 and communication stage 3 have the same time consumption, i.e., T2a = T3a.

[0078] In one example, corresponding to Figure 4, the various stages in the data processing task include: (1) a pre-distribution communication stage between different expert models, such as the all-to-all dispatch stage, where input data is distributed to each activated expert model. (2) a pre-distribution communication stage between different computing devices within an expert model, such as the all-gather stage, where all nodes (processes) send their local data to all other nodes, so that each node has a complete set of global data. (3) an inference stage for each computing device. (4) a post-distribution communication stage between different computing devices within each expert model, such as the reduce-scatter stage, where all nodes participate in the reduction. (5) a post-distribution communication stage between different expert models, such as the all-to-all combine stage, where the output data of different expert models are aggregated.

[0079] Furthermore, if the target stage is not the final stage, then in step a, in response to the first computing device completing the first sub-task in the target stage, the first computing device initiates the execution of the first sub-task in the next stage, including:

[0080] If the target stage is the inference stage, then in response to the first computing device completing the first inference subtask in the target stage, the execution progress of the preceding pre-communication stage of the inference stage is determined; when the execution progress of the pre-communication stage is the completion of the last pre-communication subtask, the first computing device starts the execution of the first inference subtask in the following post-communication stage of the inference stage.

[0081] This is because, in the embodiments of this disclosure, the computing device can execute subtasks of different types simultaneously. For example, the subtasks in communication stage 2 and communication stage 3 are subtasks corresponding to the communication type, and the subtasks in communication stage 1 and communication stage 4 are also subtasks corresponding to the communication type.

[0082] Therefore, in this step, if the target stage is the inference stage, when the first computing device completes the first inference subtask in this target stage, it needs to determine the execution progress of the preceding pre-communication stage. On the one hand, if the execution progress indicates that the last pre-communication subtask has been completed, the first computing device can start executing the first inference subtask in the following post-communication stage of the inference stage and continue to execute multiple subtasks in the following post-communication stage sequentially. On the other hand, if the execution progress indicates that the last pre-communication subtask has not been completed (including before execution has started and during execution), the first computing device needs to wait until the last pre-communication subtask is completed.

[0083] Considering that the data processing task in this embodiment is executed sequentially in multiple stages, and that the pre-communication stage and the post-communication stage are symmetrically distributed with respect to the inference stage, it is only necessary to judge the execution progress of the previous pre-communication stage when the target stage is the inference stage. Then, for other post-communication stages, it is not necessary to judge the execution progress of the corresponding pre-communication stage, which can ensure that the last subtask has been completed, thereby reducing the judgment process and speeding up the processing efficiency.

[0084] Referring again to Figure 4, in this embodiment, each stage takes 4 units of time. At the third unit of time T3, device 0 is executing subtask 1 of the inference stage, device 1 is executing subtask 2 of the communication stage 2, and device 2 is executing subtask 3 of the communication stage 1. However, devices 0 to 3 cannot execute subtasks corresponding to the communication type simultaneously. Again, using Figure 4 as an example, device 0 has already completed subtask 1 of the inference task at the third unit of time T3, but at the fourth unit of time T4 and the fifth unit of time T5, devices 2 and 3 are still executing subtasks 3 and 4 of the communication stage 2, respectively. Only after device 3 has completed subtask 4 of the inference stage 2 at the fifth unit of time T5 can device 0 begin executing subtask 1 of the same task type in the communication stage 3 at the sixth unit of time T6. Therefore, similarly, devices 1, 2, and 3 also need to wait for a period of time before executing the subtasks of the communication stage 3.

[0085] After the target stage is the inference stage and the above judgment is completed, the data processing task continues. When the target stage is communication stage 3, the next subsequent communication stage is communication stage 4, and the preceding communication stage corresponding to the communication type is communication stage 1. Communication stage 1 precedes communication stage 2, so the execution progress of this preceding communication stage obviously meets the requirements. That is to say, for other subsequent communication stages, it is not necessary to judge the execution progress of the corresponding preceding communication stage to ensure that the last subtask of the corresponding preceding communication stage has been completed. Of course, in order to strictly ensure the accuracy and order of data processing, in this embodiment of the disclosure, when the target stage is the inference stage and other subsequent subsequent communication stages, a similar judgment can be made before the first computing device starts executing the first subtask in the next stage.

[0086] Furthermore, when the communication times between the at least one pre-communication stage and / or between the at least one post-communication stage are unequal, within the same stage, the at least two computing devices sequentially execute the sub-tasks of that stage in a serial manner, including:

[0087] For computing device N and stage M, in response to computing device N-1 having completed the subtask Mi corresponding to stage M, and computing device N having completed the subtask M-1i+1 corresponding to stage M-1, computing device N starts executing the subtask Mi+1 corresponding to stage M; N, M, and i are all positive integers.

[0088] Here, when the communication time between at least one pre-communication stage and / or between at least one post-communication stage is not equal, when executing multiple sub-tasks of each stage in sequence, any device N to start executing a sub-task Mi+1 of a certain stage M needs to satisfy at least two conditions: the previous device N-1 has completed the previous sub-task Mi of this stage M and the device N has completed the sub-task M-1i+1 corresponding to the previous stage M-1.

[0089] Figure 6 is a second schematic diagram of the total time consumption of parallel data processing provided in an embodiment of this disclosure. As shown in Figure 6, the total time consumption of the data processing task is TA'. Communication phase 1 takes longer than communication phase 2. Therefore, regarding the sequential execution of multiple sub-tasks of communication phase 2, for device 1, when device 0 has completed the execution of sub-task 1 of communication phase 2 and device 1 has completed the execution of sub-task 2 of communication phase 1, device 1 can start executing sub-task 2 of communication phase 2.

[0090] Furthermore, if the total number of computing devices is D, and the cost of any stage is t, then the cost T of each subtask in that stage is: This indicates the rounding up operation.

[0091] When the cost (time) t of this stage is not an integer multiple of the total number of computing devices D, in S101, the tasks of the at least two stages are each divided into several sub-tasks, including: filling data at the end of this stage, such that the length of each sub-task is T, and the length of the filled data L = D × Tt. Here, the filled data is generally meaningless data, such as 0.

[0092] The purpose of padding data is to help achieve the even distribution of tasks and enable parallel processing across multiple devices. Therefore, after the data is distributed to each parallel device, when processing steps need to be performed, the padding data can be skipped directly and not actually processed by the device, thus saving the overhead of the parallel devices. Therefore, the parallel processing of the data processing task in S102 also includes: when the computing devices execute subtasks in parallel, skipping the padding data portion of the several subtasks.

[0093] Figure 7 is a third schematic diagram of the total time consumption of parallel data processing provided in the embodiments of this disclosure. As shown in Figure 7, the total time consumption of the data processing task is TA'. The time consumption of communication stage 1 and communication stage 4 is 10 units of time, the time consumption of communication stage 2 and communication stage 3 is 7 units of time, and the time consumption of the inference stage is 8 units of time. However, the total number of devices is 4, so it can be seen that the time consumption of each communication stage cannot be divided evenly by the total number of devices, 4.

[0094] Therefore, the time consumption of communication stages 1 and 4 can be padded to 12, with the padded data length being the data length corresponding to 2 units of time. This is then divided into 4 groups, with each subtask having a length of 3. Thus, when parallel processing of the data processing task is achieved, communication stages 1 and 4 are grouped as 3, 3, 3, 1 (the padded 2 is discarded). Alternatively, the time consumption of communication stages 2 and 3 can be padded to 8, with the padded data length being the data length corresponding to 1 unit of time. This is then divided into 4 groups, with each subtask having a length of 2. Thus, when parallel processing of the data processing task is achieved, communication stages 1 and 4 are grouped as 2, 2, 2, 1 (the padded 1 is discarded).

[0095] After the data splitting is complete, parallel processing of data processing tasks is achieved based on parallel computing devices. To ensure the sequential execution of multiple subtasks within the same stage, after a computing device finishes executing a subtask filled with data in a certain stage, it needs to wait for a period of time until the previous computing device finishes executing the subtask corresponding to the next stage before starting to execute the subtask corresponding to the next stage. Corresponding to Figure 7, after device 3 finishes executing the fourth subtask of communication stage 1, it needs to wait one time unit before it can execute the fourth subtask of communication stage 2.

[0096] This disclosure provides a data parallel processing method that divides at least two-stage tasks into several sub-tasks and performs parallel processing of data processing tasks based on at least two parallel computing devices.

[0097] In this way, different computing devices can process subtasks of different stages in parallel at the same time, saving processing time. Taking Figure 4 as an example, at the same time, device 0 executes subtask 1 of the inference stage, while device 1 executes subtask 2 of the communication stage 2, and device 2 executes subtask 3 of the communication stage 1. Each device can process the business of different communication stages at the same time, which can significantly reduce the idle time of computing devices, shorten the waiting time of the communication stage, reduce the total data processing time, and allow more time to be used for actual inference calculations, thereby improving the utilization rate of computing power. At the same time, subtasks of the same stage are executed sequentially in a serial manner, which is simple and helps ensure the correct execution of data processing tasks.

[0098] Based on the same technical concept, this disclosure also provides a data parallel processing device, implemented by at least two parallel computing devices, wherein the data processing task includes at least two stages. Please refer to Figure 8, which is a schematic diagram of the structure of a data parallel processing device provided in this disclosure. As shown in Figure 8, the data parallel processing device 700 includes:

[0099] The split module 710 is configured to split each of the tasks in at least two stages into several subtasks;

[0100] The parallel processing module 720 is configured to perform parallel processing of data processing tasks based on at least two parallel computing devices.

[0101] Parallel processing of data processing tasks includes: within the same stage, at least two computing devices sequentially execute the sub-tasks of that stage in a serial manner; and between sub-tasks in different stages, the at least two computing devices process them in parallel.

[0102] Furthermore, the device is implemented based on a hybrid expert model, which includes at least two expert models, each expert model includes at least two computing devices, and the number of computing devices in each expert model is the same. The number of sub-tasks obtained by splitting the task at each stage corresponds to the number of computing devices in each expert model.

[0103] The parallel processing module 720 is also configured to process data processing tasks in parallel among all computing devices based on a hybrid expert model.

[0104] Furthermore, when the splitting module 710 is configured to split the task of at least two stages into several sub-tasks, the splitting module 710 is configured as follows:

[0105] Obtain the task split dimensions, and based on the task split dimensions, transform the weight matrix of the expert model into at least two sub-weight matrices;

[0106] The sub-weight matrices are distributed one by one to the computing devices in the corresponding expert model. The number of sub-weight matrices corresponds to the number of computing devices in the expert model. Each computing device performs parallel processing of data processing tasks based on its corresponding sub-weight matrix.

[0107] Furthermore, the parallel processing module 720 further includes the following parallel processing of the data processing task:

[0108] While executing multiple subtasks of any target stage sequentially, if the target stage is not the final stage, in response to the first computing device completing the first subtask in the target stage, the first computing device starts executing the first subtask in the next stage, and continues to execute multiple subtasks for the next stage sequentially.

[0109] Furthermore, the multiple stages include at least one pre-communication stage, a reasoning stage, and at least one post-communication stage; the at least one pre-communication stage and the at least one post-communication stage are symmetrically distributed about the reasoning stage;

[0110] If the target stage is not the final stage, then in response to the first computing device completing the first subtask in the target stage, the first computing device initiates the execution of the first subtask in the next stage, including:

[0111] If the target phase is the inference phase, then in response to the first computing device completing the first inference subtask in the target phase, the execution progress of the preceding pre-communication phase of the inference phase is determined.

[0112] When the execution progress of the pre-communication phase is such that the last pre-communication subtask has been completed, the first computing device initiates the execution of the first inference subtask in the next post-communication phase of the inference phase.

[0113] Furthermore, when the communication time between at least one pre-communication stage and / or between at least one post-communication stage is unequal, within the same stage, at least two computing devices sequentially execute the sub-tasks of that stage in a serial manner, including:

[0114] For computing device N and stage M, in response to computing device N-1 having completed the subtask Mi corresponding to stage M, and computing device N having completed the subtask M-1i+1 corresponding to stage M-1, computing device N starts executing the subtask Mi+1 corresponding to stage M; N, M, and i are all positive integers.

[0115] Furthermore, if the total number of computing devices is D, and the cost of any stage is t, then the cost T of each subtask in that stage is: This indicates a rounding up operation; when the cost t of this stage is not an integer multiple of the total number of computing devices D, the splitting module 710, when configured to split the tasks of at least two stages into several subtasks, is configured as follows:

[0116] At the end of this stage, data is filled in so that the length of each subtask is T, and the length of the filled data is L = D × Tt.

[0117] Furthermore, the parallel processing module 720 also includes the following parallel processing methods for data processing tasks:

[0118] When a computing device executes subtasks in parallel, it skips the padding data portions in several subtasks.

[0119] Furthermore, the various stages in the data processing task include, in sequence: the data splitting pre-communication stage between different expert models, the data splitting pre-communication stage between different computing devices within an expert model, the inference stage of each computing device, the data set post-communication stage between different computing devices within each expert model, and the data set post-communication stage between different expert models.

[0120] Please refer to Figure 9, which is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. As shown in Figure 9, the electronic device 800 includes a processor 810, a memory 820, and a bus 830.

[0121] The memory 820 stores machine-readable instructions that can be executed by the processor 810. When the electronic device 800 is running, the processor 810 and the memory 820 communicate through the bus 830. When the machine-readable instructions are executed by the processor 810, the steps of the data parallel processing method as described in the above method embodiment can be executed. For specific implementation details, please refer to the method embodiment, which will not be repeated here.

[0122] This disclosure also provides a computer-readable storage medium storing a computer program. When the computer program is run by a processor, it can execute the steps of the data parallel processing method as described in the above method embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0123] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0124] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0125] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0126] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0127] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0128] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims. Industrial applicability

[0129] This disclosure can significantly reduce the idle time of computing devices, shorten the waiting time of computing devices in the communication phase, reduce the total time spent on data processing, and thus improve the utilization rate of computing devices; at the same time, subtasks in the same phase are executed sequentially in a serial manner, which is simple and helps to ensure the correct execution of data processing tasks.

Claims

1. A data parallel processing method, characterized in that, Implemented by at least two parallel computing devices, the data processing task comprises at least two phases, and the method includes: The tasks of the at least two stages are each divided into several sub-tasks, and the data processing tasks are parallelized based on the at least two parallel computing devices. Parallel processing of data processing tasks includes: within the same stage, at least two computing devices sequentially execute the sub-tasks of that stage in a serial manner; and between sub-tasks in different stages, the at least two computing devices process them in parallel.

2. The method according to claim 1, characterized in that, The method is based on a hybrid expert model, which includes at least two expert models, each expert model includes at least two computing devices, and the number of computing devices in each expert model is the same. The number of sub-tasks obtained by splitting the task at each stage corresponds to the total number of computing devices in all expert models. All computing devices in the hybrid expert model process data processing tasks in parallel.

3. The method according to claim 2, characterized in that, Each of the at least two phases of the task is further divided into several sub-tasks, including: Obtain the task splitting dimension, and based on the task splitting dimension, transform the weight matrix of the expert model into at least two sub-weight matrices; The sub-weight matrices are distributed one by one to the computing devices in the corresponding expert model. The number of sub-weight matrices corresponds to the number of computing devices in the expert model. Each computing device performs parallel processing of data processing tasks based on its corresponding sub-weight matrix.

4. The method according to claim 2, characterized in that, Parallel processing of data processing tasks also includes: While executing multiple subtasks of any target stage sequentially, if the target stage is not the final stage, in response to the first computing device completing the first subtask in the target stage, the first computing device starts executing the first subtask in the next stage, and continues to execute multiple subtasks for the next stage sequentially.

5. The method according to claim 4, characterized in that, The multiple stages include at least one pre-communication stage, an inference stage, and at least one post-communication stage; the at least one pre-communication stage and the at least one post-communication stage are symmetrically distributed about the inference stage; If the target stage is not the final stage, then in response to the first computing device completing the first subtask in the target stage, the first computing device initiates the execution of the first subtask in the next stage, including: If the target stage is the inference stage, then in response to the first computing device completing the first inference subtask in the target stage, the execution progress of the preceding pre-communication stage of the inference stage is determined. When the execution progress of the pre-communication phase is such that the last pre-communication subtask has been completed, the first computing device initiates the execution of the first inference subtask in the next post-communication phase of the inference phase.

6. The method according to claim 5, characterized in that, When the communication times between at least one pre-communication stage and / or between at least one post-communication stage are not equal, within the same stage, the at least two computing devices sequentially execute the sub-tasks of that stage in a serial manner, including: For computing device N and stage M, in response to computing device N-1 having completed the subtask Mi corresponding to stage M, and computing device N having completed the subtask M-1i+1 corresponding to stage M-1, computing device N starts executing the subtask Mi+1 corresponding to stage M; N, M, and i are all positive integers.

7. The method according to any one of claims 1-6, characterized in that, If the total number of computing devices is D, and the cost of any stage is t, then the cost T of each subtask in that stage is: This indicates the rounding up operation; When the overhead t of this stage is not an integer multiple of the total number of computing devices D, the tasks of the at least two stages are each divided into several sub-tasks, including: At the end of this stage, data is filled in so that the length of each subtask is T, and the length of the filled data is L = D × Tt.

8. The method according to claim 7, characterized in that, Parallel processing of data processing tasks also includes: When the computing device executes subtasks in parallel, the padding data portion of the subtasks is skipped.

9. The method according to claim 5, characterized in that, The data processing task includes the following stages in sequence: the data splitting pre-communication stage between different expert models, the data splitting pre-communication stage between different computing devices within an expert model, the inference stage of each computing device, the data set post-communication stage between different computing devices within each expert model, and the data set post-communication stage between different expert models.

10. The method according to claim 2, characterized in that, The number of computing devices is a power of 2.

11. The method according to claim 3, characterized in that, The task splitting dimensions include row dimensions or column dimensions.

12. The method according to claim 11, characterized in that, When the task splitting dimension is a column dimension, the weight matrix of the expert model is split into sub-weight matrices by column according to the number of computing devices in a single expert model, and then distributed to each computing device in the single expert model in turn.

13. The method according to claim 11, characterized in that, When the task is split into rows, the weight matrix of the expert model is split into sub-weight matrices by rows and the input matrix is ​​split into sub-input matrices by columns, based on the number of computing devices in a single expert model. The sub-weight matrices and sub-input matrices are then distributed to the computing devices in the single expert model in sequence. The summation operation sums the local results of the computational devices in a single expert model.

14. A data parallel processing device, characterized in that, Implemented by at least two parallel computing devices, the data processing task includes at least two phases, and the apparatus comprises: The splitting module is configured to split each of the at least two phases of tasks into several sub-tasks; A parallel processing module is configured to perform parallel processing of data processing tasks based on the at least two parallel computing devices. Parallel processing of data processing tasks includes: within the same stage, at least two computing devices sequentially execute the sub-tasks of that stage in a serial manner; and between sub-tasks in different stages, the at least two computing devices process them in parallel.

15. An electronic device, characterized in that, include: The electronic device includes a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and the machine-readable instructions, when executed by the processor, perform the data parallel processing method as described in any one of claims 1-13.

16. A computer-readable storage medium, characterized in that, include: The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the data parallel processing method as described in any one of claims 1-13.