Communication method, apparatus, and system

Abstract

Embodiments of the present application provide a communication method, apparatus, and system. A first communication apparatus sends first data blocks to a second communication apparatus. The second communication apparatus performs computation on each received first data block to obtain a first computation result, and sends the first computation result to the first communication apparatus. The first communication apparatus sends second data blocks associated with the first computation result to the second communication apparatus, and sends fourth data blocks to the second communication apparatus before at least part of third data blocks sent by the second communication apparatus are received. The third data blocks are associated with a second computation result corresponding to a third communication apparatus. The first data blocks, the first computation result, and the second computation result correspond to a first operation, the fourth data blocks correspond to a second operation, and the first operation and the second operation are related to computation. The second data blocks and the third data blocks correspond to a third operation, and the third operation is related to forwarding. Thus, communication time is shortened by performing the second operation and the third operation in parallel.

Description

A communication method, apparatus and system

[0001] This application claims priority to Chinese Patent Application No. 202510021188.6, filed on January 6, 2025, entitled "A Communication Method, Apparatus and System", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of data communication technology, and in particular to a communication method, apparatus and system. Background Technology

[0003] As model size continues to increase, the computational demands for model training far exceed the capabilities of a single chip, making distributed training an inevitable choice. Distributed training requires intensive communication support, leading to the development of Collective Communications. However, current Collective Communications methods suffer from high communication time ratios (i.e., the proportion of communication time to model training time) and low communication efficiency, thus impacting model training efficiency.

[0004] Therefore, there is an urgent need to provide a communication method with higher communication efficiency, which can reduce the proportion of communication time, improve communication efficiency, and thus improve the efficiency of model training. Summary of the Invention

[0005] Based on this, this application provides a communication method, apparatus, and system that can effectively reduce the proportion of communication time and improve communication efficiency, thereby improving the efficiency of model training to a certain extent.

[0006] In a first aspect, this application provides a communication method in which, firstly, a first communication device sends a first data block to be calculated to a second communication device; the second communication device calculates each received first data block to obtain a first calculation result and sends the first calculation result to the first communication device; then, after receiving the first calculation result from the second communication device, the first communication device sends a second data block associated with the first calculation result to the second communication device; next, before receiving at least a portion of the third data block sent by the second communication device, the first communication device sends another fourth data block to be calculated to the second communication device, wherein the third data block is associated with the second calculation result corresponding to the third communication device, the first data block, the first calculation result, and the second calculation result correspond to a first operation, the fourth data block corresponds to a second operation, the first and second operations are related to calculation, the second data block and the third data block correspond to a third operation, and the third operation is related to forwarding. As can be seen, through this method, the first communication device sends a second data block associated with the first calculation result to the second communication device. The first communication device does not need to wait to receive all the third data blocks sent by the second communication device before it can send another data block (i.e., the fourth data block) to be calculated. This is equivalent to the first communication device starting the third operation and then starting the second operation without waiting for the third operation to complete. By operating the second and third operations in parallel, the time spent waiting for the third operation to complete is saved, effectively shortening the communication time. This reduces the proportion of communication time in the collective communication, making rapid distributed training of artificial intelligence (AI) possible.

[0007] It should be noted that the sending of the first data block from the first communication device to the second communication device and the receiving of the first calculation result from the second communication device are two optional steps. That is, this application also provides a communication device, the method of which may include, for example,: the first communication device sending a second data block to the second communication device; the first communication device sending another fourth data block to be calculated to the second communication device before receiving at least a portion of the third data block sent by the second communication device, wherein the third data block is associated with the third communication device, the fourth data block corresponds to a second operation related to calculation, and the second and third data blocks correspond to a third operation related to forwarding. As can be seen, through this method, the first communication device sends the second data block to the second communication device. The first communication device does not need to wait to receive all the third data blocks sent by the second communication device before it can send another data block (i.e., the fourth data block) to be calculated. This is equivalent to the first communication device starting the third operation and then starting the second operation without waiting for the third operation to complete. By operating the second and third operations in parallel, the time spent waiting for the third operation to complete is saved, effectively shortening the communication time. This reduces the proportion of communication time in the collective communication, making rapid distributed AI training possible.

[0008] In some possible implementations, the first calculation result can be obtained by the second communication device calculating the received N first data blocks, where N is the same as the number of communication devices participating in the communication. The communication devices participating in the communication include the first communication device and the third communication device, but do not include the second communication device. Specifically, N is the same as the number of end-side computing devices participating in the communication. In this way, the communication devices participating in the communication can only send and receive data without participating in the calculation. Then, each communication device participating in the communication can send all the data involved in the first operation to the second communication device, which will then complete the calculation part corresponding to each communication device participating in the communication.

[0009] In some other possible implementations, the first calculation result can be obtained by the second communication device calculating (N-1) first data blocks received, where N is the same as the number of communication devices participating in the communication. Since the (N-1) first data blocks do not include the first data block corresponding to the first communication device, after receiving the first calculation result from the second communication device before sending the second data block, the method further includes: the first communication device obtaining the second data block based on the first calculation result and the first data block corresponding to the first communication device. In this way, the second communication device completes part of the calculation, and each participating communication device can complete the remaining calculations locally. It should be noted that in this implementation, for the first operation, the first communication device may not send the first data block corresponding to the first communication device to the second communication device, but instead send the first data blocks corresponding to the other participating communication devices.

[0010] In some possible implementations, the method provided in this application may further include, before the first communication device sends the first data block to the second communication device, sending a fifth data block to the second communication device. This fifth data block corresponds to a fourth operation, which is related to forwarding. Thus, for a single communication task, forwarding-related operations can be executed first, followed by computation-related operations. For example, the fourth operation is the first forwarding-related operation in this communication task, and the first operation is the first computation-related operation in this communication task.

[0011] As an example, the first communication device can send a first data block to the second communication device before receiving at least a portion of the sixth data block sent by the second communication device, the sixth data block corresponding to the fourth operation. In this way, in a single communication task, after completing the sending step in the first forwarding-related operation, the communication flow corresponding to the computation-related operation for the next data block can immediately begin, without waiting for the receiving step in the first forwarding-related operation to complete, effectively improving communication efficiency.

[0012] As another example, before sending the first data block to the second communication device, and after sending the fifth data block to the second communication device, the first communication device can also update the data on the first communication device based on the fifth operation. The fifth operation indicates an operation different from both computation and forwarding. Therefore, the first data block can be the updated data on the first communication device. In this way, in a single communication task, after executing the first forwarding-related operation, it can wait for other operations to complete and then enter the communication flow corresponding to the computation-related operation based on the updated data. This allows for more flexible adaptation to other communication scenarios to improve communication efficiency.

[0013] In some possible implementations, after receiving the first calculation result from the second communication device and before sending the second data block to the second communication device, the first communication device can also obtain the second data block based on the first calculation result and a sixth operation. The sixth operation is used to indicate an operation different from both the calculation and forwarding. The fifth and sixth operations can be related or different. Thus, in a single communication task, after executing the first calculation-related operation, it can wait for other operations to complete and enter the communication flow corresponding to the forwarding-related operation based on the updated data. This allows for more flexible adaptation to other communication scenarios to improve communication efficiency.

[0014] In some possible implementations, before the first communication device sends the first data block to the second communication device, the method may further include: the first communication device obtaining multiple sets of operations, each set of operations including a computation-related operation and a forwarding-related operation, wherein the first operation and the third operation belong to one set of operations in the multiple sets of operations. The source of the multiple sets of operations can be any entity capable of configuring operations to the first communication device, such as a controller or an upper-layer application, and the multiple sets of operations can correspond to a communication task.

[0015] As an example, the first communication device obtains multiple sets of operations, which may include: the first communication device obtains multiple sets of operations based on a seventh operation, which is used to instruct the performance of an all-reduce operation. The seventh operation is the operation obtained by the first communication device for this communication task. In one case, firstly, the first communication device can convert the seventh operation into one first-type operation and one second-type operation. The first-type operation is related to computation, and the second-type operation is related to forwarding. Then, the first communication device divides the first-type operation into multiple first-type operations and the second-type operation into multiple second-type operations through data segmentation. The number of first-type operations and the number of second-type operations after segmentation are the same as the number of data blocks after segmentation. The segmented first-type operations correspond one-to-one with the segmented second-type operations. The first operation is one of the multiple segmented first-type operations, and the third operation is a segmented second-type operation corresponding to the first operation.

[0016] As another example, the first communication device can acquire an eighth operation and a ninth operation, where the eighth operation is related to computation and the ninth operation is related to forwarding. The first communication device can acquire multiple sets of operations in the following ways: The first communication device obtains M first-type operations based on the eighth operation and M second-type operations based on the ninth operation, where M is an integer greater than 1, and the M first-type operations and M second-type operations correspond one-to-one. Therefore, the first communication device records each of the M first-type operations and its corresponding second-type operation as a group of operations to obtain multiple sets of operations. The first communication device can split the eighth and ninth operations by data segmentation, that is, by segmenting the data and assigning each segmented data block a first-type operation and a second-type operation, thus forming a group of operations corresponding to that data block.

[0017] In some possible implementations, for the first operation, any one or more communication devices participating in the communication can send the first operation to the second communication device, so that the second communication device can perform calculations on the data including the first data block based on the received first operation. For example, the method may further include: the first communication device sending the first operation to the second communication device. The first communication device may send the first operation and the corresponding data block to the second communication device in the same message, or it may send them to the second communication device in separate messages.

[0018] As an example, the first communication device sending a first data block to the second communication device may include, for example, the first communication device sending a first message to the second communication device, the first message including the first data block and a first operation.

[0019] As an example, the method may further include: a first communication device obtaining N reduce operations according to a first operation, each of the N reduce operations instructing computation on a first data block corresponding to a communication device participating in the communication, where N is the same as the number of communication devices participating in the communication; thereby, the first communication device sends the N reduce operations to a second communication device. In this example, the first communication device sending the first data block to the second communication device may include: the first communication device sending a second message to the second communication device, the second message including the first data block and the reduce operations.

[0020] In some possible implementations, for the third operation, any one or more communication devices participating in the communication can send the third operation to the second communication device, so that the second communication device can forward the data block associated with the calculation result of the first operation based on the received third operation. For example, the method may also include: the first communication device sending the third operation to the second communication device. The first communication device may send the third operation and the corresponding second data block in the same message to the second communication device, or it may send them in separate messages to the second communication device.

[0021] As an example, the first communication device sending a second data block to the second communication device may include, for example, the first communication device sending a third message to the second communication device, the third message including the second data block and a third operation.

[0022] The first and second operations are scattering reduction operations, and the third operation is an all-gather operation or a broadcast operation.

[0023] As an example, if the second data block corresponds to the All-Gather operation, the method may further include: the first communication device obtaining the corresponding Broadcast operation based on the All-Gather operation, and recording the Broadcast operation as the third operation. Thus, after receiving the Broadcast operation for the second data block, the second communication device can copy the second data block and forward it to other participating communication devices.

[0024] Secondly, this application also provides a first communication device, which includes a transceiver unit. The transceiver unit is configured to: send a first data block to a second communication device, the first data block being data to be calculated; receive a first calculation result from the second communication device, the first calculation result being obtained by the second communication device from each received first data block; send a second data block to the second communication device, the second data block being associated with the first calculation result; and before receiving at least a portion of a third data block sent by the second communication device, send a fourth data block to the second communication device, the fourth data block being data to be calculated, the third data block being associated with a second calculation result corresponding to the third communication device, the first data block, the first calculation result, and the second calculation result corresponding to a first operation, the fourth data block corresponding to a second operation, the first and second operations being related to calculation, the second data block and the third data block corresponding to a third operation, and the third operation being related to forwarding.

[0025] In some possible implementations, the first calculation result is obtained by the second communication device calculating the N first data blocks received, where N is the same as the number of communication devices participating in the communication. The communication devices participating in the communication include the first communication device and the third communication device, but do not include the second communication device.

[0026] In some other possible implementations, the first calculation result is obtained by the second communication device calculating (N-1) first data blocks received, where N is the same as the number of communication devices participating in the communication. The (N-1) first data blocks do not include the first data block corresponding to the first communication device. The first communication device further includes a processing unit. This processing unit is used to obtain the second data block based on the first calculation result and the first data block corresponding to the first communication device, after receiving the first calculation result from the second communication device before sending the second data block to the second communication device.

[0027] In some possible implementations, the transceiver unit is also configured to send a fifth data block to the second communication device before sending the first data block to the second communication device. The fifth data block corresponds to a fourth operation, which is related to forwarding.

[0028] In some possible implementations, the transceiver unit is specifically configured to: send a first data block to the second communication device before receiving at least a portion of the sixth data block sent by the second communication device, the sixth data block corresponding to the fourth operation.

[0029] In some possible implementations, the first communication device further includes a processing unit. This processing unit is configured to update data on the first communication device based on a fifth operation, before sending a first data block to the second communication device and after sending a fifth data block to the second communication device. The fifth operation indicates an operation different from both calculation and forwarding, and the first data block is the updated data on the first communication device.

[0030] In some possible implementations, the first communication device further includes a processing unit. This processing unit is configured to, after receiving the first calculation result from the second communication device and before sending the second data block to the second communication device, obtain the second data block based on the first calculation result and a sixth operation, wherein the sixth operation indicates an operation different from both the calculation and forwarding.

[0031] In some possible implementations, the first communication device further includes a processing unit. This processing unit is configured to obtain multiple sets of operations before sending the first data block to the second communication device. Each set of operations includes a computation-related operation and a forwarding-related operation, with the first and third operations belonging to one set of operations within the multiple sets of operations.

[0032] As an example, the processing unit is specifically used to: obtain multiple sets of operations based on the seventh operation, which is used to instruct All-Reduce to be performed.

[0033] As another example, the processing unit is specifically used to: obtain M first-type operations based on the eighth operation, and obtain M second-type operations based on the ninth operation, where M is an integer greater than 1, and the M first-type operations and M second-type operations correspond one-to-one. The eighth operation is related to computation, and the ninth operation is related to forwarding. Each first-type operation in the M first-type operations and the corresponding second-type operation are recorded as a group of operations to obtain multiple groups of operations.

[0034] In some possible implementations, the transceiver unit is also used to send the first operation to the second communication device.

[0035] As an example, the transceiver unit is specifically used to: send a first message to a second communication device, the first message including a first data block and a first operation.

[0036] As another example, the first communication device further includes a processing unit. This processing unit is configured to obtain N Reduce operations based on the first operation, each of the N Reduce operations instructing computation on a first data block corresponding to one of the participating communication devices, where N is the same as the number of participating communication devices. Then, the transceiver unit is further configured to send the N Reduce operations to the second communication device. In this example, the transceiver unit is specifically configured to send a second message to the second communication device, the second message including the first data block and the Reduce operations.

[0037] In some possible implementations, the transceiver unit is also used to send a third operation to the second communication device.

[0038] As an example, the transceiver unit is specifically used to: send a third message to a second communication device, the third message including a second data block and a third operation.

[0039] The first and second operations are Reduce-Scatter operations, and the third operation is an All-Gather operation or a Broadcast operation.

[0040] As an example, if the second data block corresponds to the All-Gather operation, the processing unit of the device is further configured to: obtain the corresponding Broadcast operation based on the All-Gather operation, and record the Broadcast operation as the third operation.

[0041] It should be noted that for the relevant description of the first communication device in the second aspect, please refer to the corresponding description in the first aspect.

[0042] Thirdly, this application provides a communication device, which includes a processor and a memory; the processor is configured to execute instructions stored in the memory to cause the communication device to implement the method corresponding to the first aspect and its possible implementations. The communication device may, for example, include a graphics processing unit (GPU).

[0043] Fourthly, this application provides a chip including an interface circuit and a processing circuit, wherein the interface circuit is connected to the processing circuit. The interface circuit is used to perform the receiving and transmitting operations in the method corresponding to the first aspect and its possible implementations; the processing circuit is used to perform other operations in the method corresponding to the first aspect and its possible implementations besides the receiving and transmitting operations.

[0044] Fifthly, this application also provides a communication device, which includes an interface and a processor; the interface is used to receive instructions and transmit them to the processor; the processor is used to execute the method corresponding to the first aspect and its possible implementations. The communication device may, for example, include a network interface card (NIC).

[0045] In a sixth aspect, this application also provides a communication system, which includes a first communication device and a second communication device; the first communication device is used to execute the method corresponding to the first aspect and its possible implementations, and to achieve efficient communication through interaction with the second communication device.

[0046] As an example, in this communication system, the interaction between the first communication device and the second communication device includes, but is not limited to:

[0047] A first communication device is used to send a first data block to a second communication device, wherein the first data block is data that needs to be calculated;

[0048] The second communication device is used to calculate each of the received first data blocks to obtain a first calculation result, and to send the first calculation result to the first communication device;

[0049] The first communication device is further configured to receive a first calculation result from the second communication device, obtain a second data block based on the first calculation result, and send the second data block to the second communication device.

[0050] The first communication device is further configured to send a fourth data block to the second communication device before receiving at least a portion of the third data block sent by the second communication device. The fourth data block is data to be calculated. The third data block is associated with a second calculation result corresponding to the third communication device. The first data block, the first calculation result, and the second calculation result correspond to a first operation. The fourth data block corresponds to a second operation. The first and second operations are related to calculation. The second data block and the third data block correspond to a third operation. The third operation is related to forwarding.

[0051] In a seventh aspect, this application also provides a storage medium storing instructions that, when executed on a processor, implement the method corresponding to the first aspect and its possible implementations.

[0052] Eighthly, this application also provides a program product comprising a program that, when run on a processor, implements the method corresponding to the first aspect and its possible implementations. Attached Figure Description

[0053] Figure 1a is a schematic diagram of AR operation in an embodiment of this application;

[0054] Figure 1b is a schematic diagram of RS operation in an embodiment of this application;

[0055] Figure 1c is a schematic diagram of the AG operation in an embodiment of this application;

[0056] Figure 2a is a schematic diagram of RS operation under the network architecture corresponding to the on-net computing technology in the embodiments of this application;

[0057] Figure 2b is a schematic diagram of AG operation under the network architecture corresponding to the on-net computing technology in the embodiments of this application;

[0058] Figure 2c is a schematic diagram illustrating the effect of the amount of data transmitted in an embodiment of this application;

[0059] Figure 2d is a schematic diagram of the execution flow of the two pairs of RS operations + AG operations;

[0060] Figure 2e is a schematic diagram of the execution flow of two pairs of RS operations + AG operations under parallel communication operations in an embodiment of this application;

[0061] Figure 2f is a schematic diagram illustrating the effect of communication time in an embodiment of this application;

[0062] Figure 3 is a flowchart illustrating a communication method 100 in an embodiment of this application;

[0063] Figure 4a is a schematic diagram illustrating the effect of an example of AG operation + RS operation in an embodiment of this application;

[0064] Figure 4b is a schematic diagram illustrating the effect of another example of AG operation + other types of operation + RS operation in the embodiments of this application;

[0065] Figure 4c is a schematic diagram illustrating the effect of an example of RS operation + AG operation in an embodiment of this application.

[0066] Figure 4d is a schematic diagram of the effect of another example of RS operation + AG operation in the embodiments of this application;

[0067] Figure 5 is a schematic diagram of the correspondence between a set of operations and data blocks in an embodiment of this application;

[0068] Figure 6 is a schematic diagram of the structure of a communication device 600 in an embodiment of this application;

[0069] Figure 7 is a schematic diagram of the structure of a communication device 700 in an embodiment of this application;

[0070] Figure 8 is a structural schematic diagram of a communication device 800 according to an embodiment of this application;

[0071] Figure 9 is a schematic diagram of the structure of a chip 900 in an embodiment of this application;

[0072] Figure 10 is a schematic diagram of the structure of a communication system 1000 according to an embodiment of this application. Detailed Implementation

[0073] With the evolution of AI distributed training strategies, efficient ensemble communication has become indispensable for AI distributed training. Currently, ensemble communication typically employs All-Reduce (AR) operations (also known as ensemble communication operations, ensemble communication primitives, ensemble communication operators, or simply primitives or operators). An AR operation is a data reduction, equivalent to broadcasting after reduction. Ultimately, each computing node will have the same reduction result. Reduce can be understood as reducing a set of numbers into a smaller set using a function. For example, for the list of numbers [1,2,3,4,5], the AR operation uses a summation function to reduce this list, producing sum([1,2,3,4,5]) = 15, which is the reduction result that each computing node has after the AR operation. Alternatively, the AR operation uses a multiplication function to reduce this list, producing multiplication([1,2,3,4,5]) = 120, which is also the reduction result that each computing node has after the AR operation. As shown in Figure 1a, compute nodes 1, 2, and 3 participate in set communication, which includes AR operations. The data on compute node 1 includes 2, 4, and 6; the data on compute node 2 includes 1, 2, and 3; and the data on compute node 3 includes 4, 8, and 12. Assuming the AR operation uses the sum function, after the AR operation, the data on compute nodes 1, 2, and 3 all include 7, 14, and 21. Here, 7 is obtained by summing 2 on compute node 1, 1 on compute node 2, and 4 on compute node 3 in this AR operation, i.e., sum(2,1,4) = 7; similarly, 14 is obtained by summing 4 on compute node 1, 2 on compute node 2, and 8 on compute node 3 in this AR operation, i.e., sum(4,2,8) = 14; and 21 is obtained by summing 6 on compute node 1, 3 on compute node 2, and 12 on compute node 3 in this AR operation, i.e., sum(6,3,12) = 21.

[0074] With the development of ensemble communication, in many scenarios, AR operations have been replaced by a combination of Reduce-Scatter (RS) and All-Gather (AG) operations. The RS + AG model has become the primary ensemble communication mode. For example, in data parallel (DP) scenarios, AR operations can degenerate into RS + AG operations; similarly, in sequence parallel (SP) partitioning strategies, AR operations can degenerate into RS + AG operations. The difference between RS and AR operations lies in the fact that each computing node only needs to store a portion of the reduction results. AG operations, on the other hand, synchronize all data from all participating computing nodes to all other participating computing nodes, achieving complete data transparency. The difference between AG and Broadcast operations is that AG operations target data on all participating computing nodes, while Broadcast operations target data on a single computing node. Therefore, AG operations involve a larger amount of data transmission. Furthermore, AG operations are considered complete when the participating communication device receives the data blocks forwarded by all other communication devices, while Broadcast operations are only defined up to the point where the second communication device sends the computation result to other communication devices after the participating communication device sends the computation result to the second communication device, without considering whether data blocks from other communication devices have been received.

[0075] As shown in Figure 1b, compute node 1, compute node 2, and compute node 3 still participate in the set communication. This set communication includes RS operation. The data on compute node 1 includes a1, b1, and c1; the data on compute node 2 includes a2, b2, and c2; and the data on compute node 3 includes a3, b3, and c3. Assuming that the RS operation uses the sum function, after the RS operation, the data on compute node 1 includes a1 + a2 + a3; the data on compute node 2 includes b1 + b2 + b3; and the data on compute node 3 includes c1 + c2 + c3. Taking a1+a2+a3 as A1, b1+b2+b3 as B1, and c1+c2+c3 as C1 as an example, after this RS operation, as shown in Figure 1c, the data on compute node 1 includes A1, the data on compute node 2 includes B1, and the data on compute node 3 includes C1. After the AG operation, compute node 1 synchronizes data A1 to compute nodes 2 and 3. Similarly, compute node 2 synchronizes data B1 to compute nodes 1 and 3, and compute node 3 synchronizes data C1 to compute nodes 1 and 2. At this point, the data on compute nodes 1, 2, and 3 all include A1, B1, and C1. It can be seen that the RS operation + AG operation mode can achieve the same effect as the AR operation in aggregated communication.

[0076] However, using RS operation + AG operation mode for ensemble communication has the problem of an excessively high proportion of communication time to training time (hereinafter referred to as communication time percentage). For example, in the training of transformer-type models, the communication time percentage in RS operation + AG operation mode is about 20%. Such a high proportion of communication time will slow down the training. Therefore, a more efficient ensemble communication method is needed to improve the efficiency of AI distributed training.

[0077] Based on this, this application provides a communication method that improves communication efficiency mainly through the following two aspects: First, by using in-network computing technology, the amount of data exchanged between computing nodes is reduced; second, by using parallel operation between different types of operations during the communication process (which can be understood as allowing other types of operations to start executing while one type of operation is being executed), the waiting time during the communication process is reduced, thereby reducing the total time required for the entire communication process.

[0078] Regarding the first aspect, a communication device for on-network computing can be introduced into the context of aggregated communication. This device can perform specific calculations on fields in the message during message forwarding, thereby offloading the computing task to the communication device. This on-network computing communication device, in addition to traditional forwarding and routing functions, also possesses certain programmability, caching capabilities, and computing power, enabling it to perform certain calculations (e.g., addition) based on cached data. The communication device for on-network computing can be any of the computing nodes in aggregated communication, or it can be a network device or a functional module within a network device specifically designed for on-network computing, such as an on-network computing switch that is not a computing node in aggregated communication. The following description uses the network architecture shown in Figure 2a or Figure 2b as an example.

[0079] In this embodiment, the network architecture may include an on-grid computing switch 10, computing node 1, computing node 2, and computing node 3. Assuming the data on computing node 1 includes a1, b1, and c1; the data on computing node 2 includes a2, b2, and c2; and the data on computing node 3 includes a3, b3, and c3, and the RS operation uses the sum function, then, in one RS operation, as shown in Figure 2a, firstly, computing node 1, computing node 2, and computing node 3 send their respective data to the on-grid computing switch 10; then, the on-grid computing switch 10 aggregates the received data to obtain data A1 (i.e., a1+a2+a3) corresponding to computing node 1, data B1 (i.e., b1+b2+b3) corresponding to computing node 2, and data C1 (i.e., c1+c2+c3) corresponding to computing node 3; then, the on-grid computing switch 10 sends data A1 to computing node 1, data B1 to computing node 2, and data C1 to computing node 3. As shown in Figure 2b, after the RS operation shown in Figure 2a, the AG operation may include, for example, the following: First, compute node 1, compute node 2, and compute node 3 send data A1, B1, and C1 to the on-network compute switch 10, respectively; then, the on-network compute switch 10 forwards data A1 received from compute node 1 to compute node 2 and compute node 3, data B1 received from compute node 2 to compute node 1 and compute node 3, and data C1 received from compute node 3 to compute node 1 and compute node 2.

[0080] As shown in Figure 2c, taking three computing nodes (compute node 1, compute node 2, and compute node 3) as an example, in a network architecture without on-network computing technology, the amount of data that each computing node needs to receive and send in one RS operation can be represented as two sets of data corresponding to the other two computing nodes. Similarly, the amount of data that each computing node needs to receive and send in one AG operation can be represented as two sets of data corresponding to the other two computing nodes. Taking compute node 1 as an example, in one RS operation, it needs to send data b1 and data c1, and receive data a2 and data a3; in one AG operation, it needs to send data A1 to compute nodes 2 and 3. In the network architecture shown in Figure 2a or Figure 2b, the amount of data each computing node needs to receive in a single RS operation is reduced, and the amount of data it needs to send in a single AG operation is reduced. Taking computing node 1 as an example, in a single RS operation, the amount of data sent is still 2 copies, but it only needs to receive 1 copy of data (i.e., data A1) from the on-network computing switch 10. In a single AG operation, the amount of data received is 2 copies, but it only needs to send 1 copy of data (i.e., data A1) to the on-network computing switch 10. It should be noted that in Figure 2c, a minimum unit represents 1 copy of data. Light gray represents the data involved in the RS operation that needs to be sent and received, dark gray represents the data involved in the AG operation that needs to be received and sent, and white represents data that does not need to be sent or received. As can be seen, compared with the aggregated communication using on-network computing technology proposed in the embodiments of this application, the current aggregated communication that does not use on-network computing technology has the following advantages: for RS operation, when the data-carrying message passes through the on-network computing switch 10, it is aggregated by the on-network computing switch 10, and the aggregated result is returned to the individual computing node to save the amount of data received by the computing node; for AG operation, when the data-carrying message passes through the on-network computing switch 10, it is handed over to the on-network computing switch 10 for replication and multicasting to reduce the amount of data sent by the computing node. Thus, the overall amount of data transmitted is reduced.

[0081] Regarding the second aspect, a functional module (e.g., middleware) for implementing interleaved communication can be added to each computing node in the aggregated communication to achieve parallelism of different types of operations during the communication process. For example, the next RS operation can begin before the AG operation is completed. Here, the computing node receives multiple pairs of "RS operation + AG operation". During the AG operation of a pair of "RS operation + AG operation" at the computing node, if the received data is not completed, the next RS operation of the "RS operation + AG operation" can begin. Here, "incomplete data reception" means that the computing node has sent the data corresponding to the AG operation, but has not completed the data reception action for that AG operation.

[0082] Taking the interaction between computing node 1, computing node 2, and the network-connected computing switch 10 participating in the aggregated communication as an example, for the current scheme where different types of operations cannot be parallelized, the process of performing two "RS operation + AG operation" can be seen in Figure 2d. For example, it can include: S11, computing node 1 and computing node 2 send data {a1, b1} and {a2, b2} to the network-connected computing switch 10 respectively; S12, the network-connected computing switch 10 aggregates a1 and a2 to obtain A1, and aggregates b1 and b2 to obtain B1; S13, the network-connected computing switch 10 sends data A1 and B1 to computing node 1 and computing node 2 respectively; S14, computing node 1 sends data A1 to the network-connected computing switch 10, and computing node 2 sends data B1 to the network-connected computing switch 10; S15, the on-network computing switch 10 sends data B1 to computing node 1 and data A1 to computing node 2; S16, computing node 1 and computing node 2 send data {c1, d1} and {d2, d2} to the on-network computing switch 10 respectively; S17, the on-network computing switch 10 aggregates c1 and c2 to obtain C1, and aggregates d1 and d2 to obtain D1; S18, the on-network computing switch 10 sends data C1 and D1 to computing node 1 and computing node 2 respectively; S19, computing node 1 sends data C1 to the on-network computing switch 10, and computing node 2 sends data D1 to the on-network computing switch 10; S20, the on-network computing switch 10 sends data D1 to computing node 1 and data C1 to computing node 2. In this process, S11–S13 correspond to RS operation 1, S14–S15 correspond to AG operation 1, S16–S18 correspond to RS operation 2, and S19–S20 correspond to AG operation 2. RS operation 1 + AG operation 1 is a pair of "RS operation + AG operation", and RS operation 2 + AG operation 2 is another pair of "RS operation + AG operation". It is evident that in the communication process, different types of operations cannot be parallelized; the start of another RS ​​operation requires waiting for the completion of the previous AG operation (i.e., waiting for the data of all communication nodes in the previous AG operation to be synchronized).For the parallel schemes of different types of operations in the embodiments of this application, the process of performing two "RS operation + AG operation" can be seen in Figure 2e. For example, it may include: S21, computing node 1 and computing node 2 send data {a1, b1} and {a2, b2} to the network computing switch 10 respectively; S22, the network computing switch 10 aggregates a1 and a2 to obtain A1, and aggregates b1 and b2 to obtain B1; S23, the network computing switch 10 sends data A1 and B1 to computing node 1 and computing node 2 respectively; S24, computing node 1 sends data A1 to the network computing switch 10, and computing node 2 sends data B1 to the network computing switch 10; S25, computing node 1 and computing node 2 send data to the network computing switch 10 respectively. S26. Computation switch 10 sends data {c1, d1} and {d2, d2}; S27. On-network computing switch 10 sends data B1 to computing node 1 and data A1 to computing node 2; S28. On-network computing switch 10 aggregates c1 and c2 to obtain C1 and aggregates d1 and d2 to obtain D1; S29. On-network computing switch 10 sends data C1 and D1 to computing node 1 and computing node 2 respectively; S20. Computing node 1 sends data C1 to on-network computing switch 10 and computing node 2 sends data D1 to on-network computing switch 10; S31. On-network computing switch 10 sends data D1 to computing node 1 and data C1 to computing node 2. In this embodiment, processes S21-S23 correspond to RS operation 1, processes S24 and S26 correspond to AG operation 1, processes S25 and S27-S28 correspond to RS operation 2, and processes S29-S30 correspond to AG operation 2. RS operation 1 + AG operation 1 is a pair of "RS operation + AG operation", and RS operation 2 + AG operation 2 is another pair of "RS operation + AG operation". It should be noted that for a computing node, S25 can be executed as soon as the sending action of S24 is completed, regardless of whether the computing node has completed all the receiving operations in AG operation 1; the execution order of S26 and S27-S28 is not limited in this embodiment. The difference between Figure 2e and Figure 2d is that RS operation 2 is not performed while waiting for AG operation 1 to be completed.

[0083] As shown in Figure 2f, taking three computing nodes (computing node 1, computing node 2, and computing node 3) as an example, when there are at least two pairs of "RS operation + AG operation", for technical solutions that do not employ on-grid computing technology and where different types of operations cannot be parallelized during communication, the amount of data that each computing node needs to receive and send in one RS operation can be represented as two data sets corresponding to the other two computing nodes. Similarly, the amount of data that each computing node needs to receive and send in one AG operation can be represented as two data sets corresponding to the other two computing nodes. For the technical solution in this application that employs on-grid computing technology and where different types of operations are parallelized during communication, the amount of data that each computing node needs to receive in one RS operation becomes one data set, and the amount of data that needs to be sent in one AG operation becomes one data set. Furthermore, the start of the next RS operation does not need to wait for the end of the previous AG operation. For example, after the sending action in AG operation 1 is executed, the sending action in RS operation 2 begins immediately, without waiting for the receiving action in AG operation 1 to be completed before starting the sending action in RS operation 2. It should be noted that in Figure 2f, a minimum unit represents one piece of data. Light gray represents data involved in the RS operation that needs to be sent and received, and dark gray represents data involved in the AG operation that needs to be sent and received. White represents data that does not need to be sent or received. The white squares outlined by thicker dashed lines represent the interactive data saved (or the saved communication time) compared to the scheme in Figure 2d when communication is completed. As can be seen from the three pairs of "RS operation + AG operation" shown in Figure 2f, the technical solution provided by the embodiments of this application can effectively save communication time (see the area outlined by thicker dashed lines in Figure 2f). Moreover, the more pairs of "RS operation + AG operation" there are, the more significant the effect of saving communication time.

[0084] It should be noted that in this embodiment, the parallel communication of forwarding and computation is used as an example, such as the parallel communication of AG operation and RS operation. However, the parallel operation during communication can take other forms. For example, in a communication scenario involving multiple types of operations, such as RS and AG operations, the AG operation can start before the RS operation is completed. Alternatively, in a communication scenario involving only one type of operation, parallel communication can be performed between two adjacent operations. This embodiment will not elaborate on these points. Starting the AG operation before the RS operation is completed can mean that the computation and forwarding of each computing node are decoupled by the network computing switch. That is, after collecting data related to a computing node, the network computing switch can complete the aggregation corresponding to that computing node and send the aggregation result to that computing node. At this time, the computing node does not need to wait for other computing nodes to complete the RS operation. After receiving the aggregation result corresponding to itself in the RS operation, the computing node can start the AG operation corresponding to the RS operation and send the received aggregation result to the network computing switch.

[0085] In some possible implementations, the communication method provided in this application embodiment may include, for example, the following: First, a first communication device sends a first data block to a second communication device, the first data block being data to be calculated; then, the first communication device receives a first calculation result from the second communication device, the first calculation result being calculated by the second communication device on each of the received first data blocks; next, the first communication device sends a second data block associated with the first calculation result to the second communication device; then, before the first communication device receives at least a portion of the third data block sent by the second communication device, a fourth data block is sent to the second communication device, the fourth data block being data to be calculated, the third data block being associated with the second calculation result corresponding to the third communication device, the first data block, the first calculation result, and the second calculation result corresponding to a first operation, the fourth data block corresponding to a second operation, the first and second operations being related to calculation, the second data block and the third data block corresponding to a third operation, and the third operation being related to forwarding. As can be seen, through this method, the first communication device sends a second data block associated with the first calculation result to the second communication device. The first communication device does not need to wait to receive all the third data blocks sent by the second communication device before it can send another data block (i.e., the fourth data block) to be calculated. This is equivalent to the first communication device starting the third operation and then starting the second operation without waiting for the third operation to complete. By operating the second and third operations in parallel, the time spent waiting for the third operation to complete is saved, effectively shortening the communication time. This reduces the proportion of communication time in the collective communication, making rapid distributed AI training possible.

[0086] It should be noted that sending the first data block to the second communication device and receiving the first calculation result from the second communication device are two optional steps.

[0087] It should be noted that the first communication device and the third communication device in the embodiments of this application can refer to computing nodes in aggregated communication, and the second communication device can refer to an on-net computing node with on-net computing capability in aggregated communication.

[0088] For data center networks, computing nodes can be servers, or processors or network interface cards (NICs) within servers. For example, these can be processors suitable for executing deep learning algorithms (such as graphics processing units (GPUs), tensor processing units (TPUs), neural network processing units (NPUs), and deep learning processing units (DPUs)). It should be noted that the edge device can include at least two processors of the same or different types. For example, server 1 can include two GPUs, two NPUs, or even one GPU and one NPU. Multiple processors can execute deep learning-related algorithms in parallel, such as neural network training and inference. Data transfer between processors can be accomplished via a bus using aggregated communication technology. In this embodiment, the computing nodes participating in the aggregated communication can be multiple servers, for example, server 1 and server 2 are two computing nodes participating in the aggregated communication; or, they can be multiple processors belonging to the same server, for example, processor 1 and processor 2 in server 1 are two computing nodes participating in the aggregated communication; or, they can be multiple processors belonging to different servers, with each server including at least one processor, for example, processor 1 and processor 2 in server 1, and processor 3 in server 2 are three computing nodes participating in the aggregated communication. The on-network computing nodes can be network devices with on-network computing capabilities, such as switches or routers, or they can be chips with on-network computing functions within network devices, such as central processing units (CPUs).

[0089] For GPU clusters, compute nodes can be XPU chips, where XPUs can be, for example, processors mentioned above suitable for executing deep learning algorithms (such as graphics processing units (GPUs), tensor processing units (TPUs), neural network processing units (NPUs), and deep learning processing units (DPUs)). On-network compute nodes can be switching chips.

[0090] It should be noted that the communication method provided in this application embodiment is also applicable to scenarios in other algorithms that require communication with sparse data sets.

[0091] To provide a clearer description of the embodiments of this application, the methods provided in the embodiments of this application will be described below with reference to the accompanying drawings.

[0092] Figure 3 is a schematic flowchart of a communication method 100 provided in an embodiment of this application. This method 100 is described in the form of interaction between a first communication device and a second communication device. The first communication device can be, for example, any computing node participating in the communication, such as computing node 1, computing node 2, or computing node 3 in Figure 2a or 2b, or computing node 1 or computing node 2 in Figure 2d or 2f. The second communication device can be a node with computing capabilities and capable of data interaction with the participating computing nodes, such as a communication device with on-network computing capabilities, such as the on-network computing switch 10 in Figures 2a, 2b, 2d, or 2f.

[0093] As shown in Figure 3, the method 100 may include, for example, the following steps S101 to S108:

[0094] S101, the first communication device sends a first data block to the second communication device. The first data block is data that needs to be calculated. The first data block corresponds to a first operation, which is related to calculation.

[0095] It is understood that the operations in the embodiments of this application can at least indicate an operation type and an operation object. The operation type indicates the specific action to be performed on the operation object, and the operation object indicates the specific data to be performed on the operation type. For example, the operation type of the first operation can be RS (hereinafter referred to as the first operation as RS operation), and the operation object can include a first data block. Specifically, the operation can include indication information corresponding to the operation type and the storage address corresponding to the operation object. The second communication device can obtain the relevant operation object from the corresponding communication device based on the storage address in the operation; and perform the corresponding action on the operation object according to the operation type indicated by the indication information in the operation.

[0096] In some possible implementations, prior to S101, method 100 may further include: the first communication device receiving multiple sets of operations, each set of operations including a computation-related operation and a forwarding-related operation, wherein the first operation and the third operation (described below) belong to one set of operations within the multiple sets of operations. The source of the multiple sets of operations can be any entity capable of configuring operations to the first communication device, such as a controller or an upper-layer application. The following description uses the example of an upper-layer application sending content related to multiple sets of operations to the first communication device.

[0097] It should be noted that each group of operations in the multiple operations can correspond to a data block. The group of operations includes an operation 1 that performs calculations on the data block, and an operation 2 that forwards the calculation result obtained from operation 1.

[0098] As an example, the upper-layer application may send multiple sets of operations to the first communication device. The first communication device can obtain multiple sets of operations by receiving multiple sets of operations from the upper-layer application.

[0099] As another example, the upper-layer application may send a seventh operation to the first communication device to instruct it to perform AR. In this case, the first communication device can obtain multiple sets of operations based on the seventh operation. One scenario involves the first communication device first converting the seventh operation into one first-type operation and one second-type operation. The first-type operation is related to computation, and the second-type operation is related to forwarding. Next, the first communication device segments the first-type operation into multiple first-type operations and the second-type operation into multiple second-type operations through data segmentation. The number of first-type operations and the number of second-type operations after segmentation are the same as the number of data blocks after segmentation. Each segmented first-type operation corresponds one-to-one with a segmented second-type operation. The first operation is one of the multiple segmented first-type operations, and the third operation is a segmented second-type operation corresponding to the first operation.

[0100] As another example, the upper-layer application may send an eighth operation and a ninth operation to the first communication device. The eighth operation is related to computation, and the ninth operation is related to forwarding. The first communication device can obtain multiple sets of operations in the following ways: The first communication device obtains M first-type operations based on the eighth operation and M second-type operations based on the ninth operation, where M is an integer greater than 1, and the M first-type operations and M second-type operations correspond one-to-one. Therefore, the first communication device records each of the M first-type operations and its corresponding second-type operation as a group of operations to obtain multiple sets of operations. The first communication device can split the eighth and ninth operations by dividing the data into blocks, that is, dividing the data into blocks and assigning each block a first-type operation and a second-type operation, thus forming a group of operations for that data block.

[0101] In this implementation, for the first operation, any one or more communication devices participating in the communication can send the first operation to the second communication device, so that the second communication device can perform calculations on the data including the first data block based on the received first operation. The communication device sending the first operation to the second communication device can either carry the first operation and the corresponding data block in the same message or send them separately in different messages.

[0102] Taking the example where the communication device sending the first operation to the second communication device includes the first communication device, the method 100 may further include: the first communication device sending the first operation to the second communication device. This step can be performed before S101, simultaneously with S101, or after S101. As an example, the first communication device can send the first operation and the first data block in the same message to the second communication device. That is, S101 may include, for example, the first communication device sending a first message to the second communication device, which includes the first data block and the first operation. As another example, the first communication device can send the first operation and the first data block in different messages to the first communication device. That is, the first communication device sends message 1 and message 2 to the second communication device. Message 1 may carry the first data block, and message 2 may carry the first operation.

[0103] As an example, the first operation could be an RS operation for a first data block. In one scenario, the first communication device sending the first operation to the second communication device could include, for example, the first communication device sending an RS operation for the first data block to the second communication device. In another scenario, before sending the first operation to the second communication device, the first communication device could obtain N Reduce (R) operations based on the first operation. Each of the N Reduce operations is used to instruct computation on the first data block corresponding to one of the participating communication devices, where N is the same as the number of participating communication devices. In this case, the first communication device sending the first operation to the second communication device could include, for example, the first communication device sending the N Reduce operations to the second communication device. If the first data block and the Reduce operation are carried in the same message and sent to the second communication device, S101 could include, for example, the first communication device sending N second messages to the second communication device, each second message including one Reduce operation from the N Reduce operations and the corresponding first data block.

[0104] In other possible implementations, the multiple sets of operations can be obtained by the second communication device, and the source of the multiple sets of operations can be any entity capable of configuring operations to the second communication device, such as a controller or an upper-layer application. In this way, the communication device that does not need to participate in the communication sends operations to the second communication device, and the second communication device can obtain the data stored at the storage address indicated by the storage address of the operation object in the operation to be executed from the communication device participating in the communication. For example, before S101, the second communication device sends message 3 to the first communication device. Message 3 may carry the storage address of the operation object (including the first data block) corresponding to the first operation. Message 3 is used to request the first communication device to provide the data stored at that storage address. After receiving message 3, the first communication device, in response to message 3, executes S101, that is, the first communication device sends the first data block to the second communication device.

[0105] It is understood that the first operation can be an RS operation; the third operation can be any forwarding-related operation, such as an operation with multicast functionality, and the operation type can include, but is not limited to, AG operations or Broadcast operations. It should be noted that if the first communication device includes an AG operation during the execution of a communication task, then the first communication device can break down the AG operation into a series of Broadcast operations. For example, in the case where the third operation is a Broadcast operation, after obtaining the AG operation, the first communication device can obtain the corresponding Broadcast operation based on the AG operation. Corresponding to the description below, after obtaining the second data block, the first communication device can send a Broadcast operation for the second data block to the second communication device. After receiving the Broadcast operation, the second communication device can copy the second data block and multicast the copied second data block to other communication devices participating in the communication.

[0106] For a single collective communication task, computation-related operations can be performed first, or forwarding-related operations can be performed first; this application embodiment does not specifically limit this. Figure 3 illustrates an example of performing computation-related operations first and then forwarding-related operations. As shown in Figure 3, this communication task can begin with a computation-related operation (such as the first operation), and during the process, computation-related operations and forwarding-related operations are performed in parallel, ending with the last forwarding-related operation. The first operation can be, for example, RS operation 1, and the third operation can be, for example, AG operation 1. Then, the communication task corresponding to Figure 3 can include, for example, the following operation sequence: {RS operation 1, AG operation 1, RS operation 2, AG operation 2, ..., RS operation n, AG operation n}, where n is an integer greater than 1, and RS operation 2 can correspond to the second operation mentioned below.

[0107] For a single collective communication task, forwarding-related operations can be performed first, followed by computation-related operations. Taking the first operation as the first computation-related operation in this collective communication task, before S101, the method 100 may further include: the first communication device sending a fifth data block to the second communication device. This fifth data block corresponds to the fourth operation, which is related to forwarding. The fourth operation could be, for example, AG operation 0, the first operation could be, for example, RS operation 1, and the third operation could be, for example, AG operation 1. Therefore, a single communication task in this implementation could include, for example, the following operation sequence: {AG operation 0, RS operation 1, AG operation 1, RS operation 2, AG operation 2, ..., RS operation n, AG operation n}, where n is an integer greater than 1, and RS operation 2 can correspond to the second operation described below. The operation objects of AG operation 0 and RS operation 1 can correspond to different data blocks. Thus, the method provided by this implementation can improve communication efficiency in scenarios where "it begins with an operation related to forwarding (such as the fifth operation), computation-related operations run in parallel with forwarding-related operations, and ends with the last forwarding-related operation." It should be noted that AG operation n is an optional operation, meaning that the communication task can also end with RS operation n.

[0108] As an example, after the first communication device sends the fifth data block to the second communication device, the first operation can begin. That is, S101 may include: the first communication device sending a first data block to the second communication device in response to sending the fifth data block. In other words, S101 may include: the first communication device sending a first data block to the second communication device before receiving at least a portion of the sixth data block sent by the second communication device. This sixth data block corresponds to the fourth operation, and the sixth data block can be understood as data to be forwarded by other communication devices participating in the communication for the fourth operation. Thus, in a single collective communication task, after completing the sending step in the first forwarding-related operation, the communication flow corresponding to the computation-related operation for the next data block can immediately begin, without waiting for the receiving step in the first forwarding-related operation to be completed. This effectively improves communication efficiency, and the corresponding effect can be seen in Figure 4a.

[0109] It should be noted that in Figure 4a and Figures 4b, 4c, and 4d below, a minimum unit represents one piece of data. Light gray represents data involved in RS operations that need to be sent and received, dark gray represents data involved in AG operations that need to be received and sent, white represents data that does not need to be sent or received, and black represents other types of operations. These other types of operations are different from the forwarding-related operations and computation-related operations mentioned in the embodiments of this application. The dark gray grids drawn with thicker dashed lines represent optional AG operations in this communication task. For example, in Figures 4a and 4d, the dark gray grids drawn with thicker dashed lines indicate that AG operation n may be included in some communication tasks, while AG operation n may not be included in other communication tasks.

[0110] As another example, after the first communication device sends the fifth data block to the second communication device, before S101, other types of operations may be included. In this case, the execution of S101 needs to wait for the completion of these other types of operations before implementing S101 with the latest data. For example, after the first communication device sends the fifth data block to the second communication device, before S101, the method 100 may also include: updating the data on the first communication device based on the fifth operation, which indicates an operation different from the calculation and forwarding in the embodiments of this application. In this case, the first data block sent by the first communication device to the second communication device in S101 belongs to the updated data on the first communication device. Thus, in a single communication task, after the first forwarding-related operation is completed, other operations can be waited for to complete, and the communication flow corresponding to the calculation-related operation can be entered based on the updated data. This allows for more flexible adaptation to other communication scenarios to improve communication efficiency. The corresponding effect can be seen in Figure 4b.

[0111] For a single group communication task, the number of groups in multiple operations can be the same as the number of data blocks, and the number of data regions included in each data block can be the same as the number of communication devices participating in the communication.

[0112] For example, referring to Figure 5, assuming that for the collective communication task shown in Figure 3, the communication devices involved in the communication include a first communication device, a third communication device, and a fourth communication device, and the corresponding multiple sets of operations include {RS operation 1, AG operation 1, RS operation 2, AG operation 2, RS operation 3, AG operation 3}, then the data on the first communication device may include data block 11, data block 12, and data block 13. Data block 11 may include a10, b10, and c10; data block 12 may include a11, b11, and c11; and data block 13 may include a12, b12, and c12. The data on the third communication device may include data block 31, data block 32, and data block 33. Data block 31 may include a30, b30, and c30; data block 32 may include a31, b31, and c31; and data block 33 may include a32, b32, and c32. The data on the fourth communication device may include data block 41, data block 42 and data block 43, wherein data block 41 may include a40, b40 and c40, data block 42 may include a41, b41 and c41, and data block 43 may include a42, b42 and c42.

[0113] For RS operation 1 of this collective communication task, the corresponding data blocks may include: data block 11, data block 31, and data block 41. Specifically, a10 in data block 11, a30 in data block 31, and a40 in data block 41 can correspond to the first communication device; that is, the calculation results of a10, a30, and a40 need to be forwarded to the first communication device as the result of RS operation 1 performed by the first communication device. Similarly, b10 in data block 11, b30 in data block 31, and b40 in data block 41 can correspond to the third communication device; that is, the calculation results of b10, b30, and b40 need to be forwarded to the third communication device as the result of RS operation 1 performed by the third communication device. Furthermore, c10 in data block 11, c30 in data block 31, and c40 in data block 41 can correspond to the fourth communication device; that is, the calculation results of c10, c30, and c40 need to be forwarded to the fourth communication device as the result of RS operation 1 performed by the fourth communication device.

[0114] For RS operation 2 of this collective communication task, the corresponding data blocks may include: data block 12, data block 32, and data block 42. Specifically, a11 in data block 12, a31 in data block 32, and a41 in data block 42 can correspond to the first communication device; that is, the calculation results of a11, a31, and a41 need to be forwarded to the first communication device as the result of RS operation 2 performed by the first communication device. Similarly, b11 in data block 12, b31 in data block 32, and b41 in data block 42 can correspond to the third communication device; that is, the calculation results of b11, b31, and b41 need to be forwarded to the third communication device as the result of RS operation 2 performed by the third communication device. Furthermore, c11 in data block 12, c31 in data block 32, and c41 in data block 42 can correspond to the fourth communication device; that is, the calculation results of c11, c31, and c41 need to be forwarded to the fourth communication device as the result of RS operation 2 performed by the fourth communication device.

[0115] For RS operation 3 of this collective communication task, the corresponding data blocks may include: data block 13, data block 33, and data block 43. Specifically, a12 in data block 13, a32 in data block 33, and a42 in data block 43 can correspond to the first communication device; that is, the calculation results of a12, a32, and a42 need to be forwarded to the first communication device as the result of RS operation 3 performed by the first communication device. Similarly, b12 in data block 13, b32 in data block 33, and b42 in data block 43 can correspond to the third communication device; that is, the calculation results of b12, b32, and b42 need to be forwarded to the third communication device as the result of RS operation 3 performed by the third communication device. Furthermore, c12 in data block 13, c32 in data block 33, and c42 in data block 43 can correspond to the fourth communication device; that is, the calculation results of c12, c32, and c42 need to be forwarded to the fourth communication device as the result of RS operation 3 performed by the fourth communication device.

[0116] If the first operation corresponds to RS operation 1 in Figure 5, then the first data block in S101 may correspond to data block 11, or a portion of the data in data block 11. If the first operation corresponds to RS operation 2 in Figure 5, then the first data block in S101 may correspond to data block 12, or a portion of the data in data block 12. If the first operation corresponds to RS operation 3 in Figure 5, then the first data block in S101 may correspond to data block 13, or a portion of the data in data block 13.

[0117] S102, the second communication device receives the first data block.

[0118] S103, the second communication device calculates each of the received first data blocks to obtain a first calculation result, and the first calculation result corresponds to the first operation.

[0119] The calculation can correspond to different calculation operations based on the operation type of the first operation. For example, if the operation type of the first operation is RS, then the calculation in S103 can be aggregation.

[0120] As an example, the communication devices participating in the communication can only send and receive data without participating in calculations. In this case, each participating communication device can send all the data involved in the first operation to the second communication device, which then completes the calculation portion corresponding to each participating communication device. For example, S103 may include: the second communication device performs calculations on the N received first data blocks corresponding to the first communication device to obtain a first calculation result corresponding to the first communication device. Here, N is the same as the number of participating communication devices, which includes the first and third communication devices, but not the second communication device. In this example, the first communication device can use the first calculation result received from the second communication device as the second data block in S106 below. Alternatively, after receiving the second data block, the first communication device can also update the data on the first communication device based on other types of operations prior to the third operation. In this case, the second data block can also be data updated based on the first calculation result and other types of operations.

[0121] Corresponding to the embodiment shown in Figure 5, if the first operation corresponds to RS operation 1, then the process of the second communication device performing the first operation may include: the second communication device aggregating a10, a30, and a40 to obtain calculation result 11; aggregating b10, b30, and b40 to obtain calculation result 13; and aggregating c10, c30, and c40 to obtain calculation result 14. The first calculation result in S103 can correspond to the above-mentioned calculation result 11, calculation result 13 is the calculation result corresponding to the third communication device, and calculation result 14 is the calculation result corresponding to the fourth communication device. The second communication device can send calculation result 13 to the third communication device and calculation result 14 to the fourth communication device.

[0122] If the first operation corresponds to RS operation 2 in Figure 5, then the process of the second communication device executing the first operation may include: the second communication device aggregating a11, a31, and a41 to obtain calculation result 21; aggregating b11, b31, and b41 to obtain calculation result 23; and aggregating c11, c31, and c41 to obtain calculation result 24. The first calculation result in S103 can correspond to the above-mentioned calculation result 21, calculation result 23 is the calculation result corresponding to the third communication device, and calculation result 24 is the calculation result corresponding to the fourth communication device. The second communication device can send calculation result 23 to the third communication device and calculation result 24 to the fourth communication device.

[0123] If the first operation corresponds to RS operation 3 in Figure 5, then the process of the second communication device executing the first operation may include: the second communication device aggregating a12, a32, and a42 to obtain calculation result 31; aggregating b12, b32, and b42 to obtain calculation result 33; and aggregating c12, c32, and c42 to obtain calculation result 34. The first calculation result in S103 may correspond to the aforementioned calculation result 31, calculation result 33 is the calculation result corresponding to the third communication device, and calculation result 34 is the calculation result corresponding to the fourth communication device. The second communication device may send calculation result 33 to the third communication device and calculation result 34 to the fourth communication device.

[0124] As another example, the second communication device performs some calculations, while each participating communication device can perform other calculations locally. For example, the second communication device performs calculations on the received (N-1) first data blocks corresponding to each participating communication device to obtain the calculation results corresponding to each participating communication device in the first operation. Here, N is the same as the number of participating communication devices, and the (N-1) first data blocks do not include the first data block corresponding to each participating communication device itself. In this example, each participating communication device may not send its own data block in the first operation to the second communication device. For example, the first data block in S101 may not include the first data block corresponding to the first communication device in the first operation. That is, for the first operation, the first communication device may not send its own first data block to the second communication device, but instead send the first data blocks corresponding to the other participating communication devices. S103 may include, for example, the second communication device performing calculations on the received (N-1) first data blocks corresponding to the first communication device to obtain the first calculation result corresponding to the first communication device. Where N is the same as the number of communication devices participating in the communication, and "(N-1) first data blocks corresponding to the first communication device" does not include the first data block corresponding to the first communication device. In this example, after the first communication device receives the first calculation result, the method may further include: the first communication device obtaining a second data block based on the first calculation result and the first data block corresponding to the first communication device.

[0125] Corresponding to the embodiment shown in Figure 5, if the first operation corresponds to RS operation 1, then the process of the second communication device performing the first operation may include: the second communication device aggregating a30 and a40 to obtain calculation result 11'; aggregating b30 and b40 to obtain calculation result 13'; and aggregating c30 and c40 to obtain calculation result 14'. The first calculation result in S103 can correspond to the above-mentioned calculation result 11', calculation result 13' is the calculation result corresponding to the third communication device, and calculation result 14' is the calculation result corresponding to the fourth communication device. The second communication device can send calculation result 13' to the third communication device and calculation result 14' to the fourth communication device. Taking the first communication device as an example, after receiving the calculation result 11' sent by the second communication device, it can aggregate a10 and calculation result 11' to obtain calculation result 11, which corresponds to the second data block in S106 below.

[0126] If the first operation corresponds to RS operation 2 in Figure 5, then the process of the second communication device performing the first operation may include: the second communication device aggregating a31 and a41 to obtain calculation result 21'; aggregating b31 and b41 to obtain calculation result 23'; and aggregating c31 and c41 to obtain calculation result 24'. The first calculation result in S103 can correspond to the above calculation result 21', calculation result 23' is the calculation result corresponding to the third communication device, and calculation result 24' is the calculation result corresponding to the fourth communication device. The second communication device can send calculation result 23' to the third communication device and calculation result 24' to the fourth communication device. Taking the first communication device as an example, after receiving the calculation result 21' sent by the second communication device, it can aggregate a11 and calculation result 21' to obtain calculation result 21, which corresponds to the second data block in S106 below.

[0127] If the first operation corresponds to RS operation 3 in Figure 5, then the process of the second communication device performing the first operation may include: the second communication device aggregating a32 and a42 to obtain calculation result 31'; aggregating b32 and b42 to obtain calculation result 33'; and aggregating c32 and c42 to obtain calculation result 34'. The first calculation result in S103 can correspond to the above calculation result 31', calculation result 33' is the calculation result corresponding to the third communication device, and calculation result 34' is the calculation result corresponding to the fourth communication device. The second communication device can send calculation result 33' to the third communication device and calculation result 34' to the fourth communication device. Taking the first communication device as an example, after receiving the calculation result 31' sent by the second communication device, it can aggregate a12 and calculation result 31' to obtain calculation result 31, which corresponds to the second data block in S106 below.

[0128] S104, the second communication device sends the first calculation result to the first communication device.

[0129] S105, the first communication device receives the first calculation result.

[0130] S106, the first communication device obtains the second data block based on the first calculation result.

[0131] It should be noted that the acquisition of the second data block by the first communication device can be understood as the completion of the first operation for the first communication device.

[0132] It should be noted that the relevant descriptions in S104 to S106 can be found in the corresponding descriptions in S103 and the examples in Figure 5.

[0133] In some possible implementations, other types of operations may be included between the first operation and the third operation described below. That is, before S106, after the first communication device receives the first calculation result, or after the first communication device obtains the second data block, the method 100 may further include: the first communication device updates the data on the first communication device based on the first calculation result (or the second data block) and the sixth operation, which is used to indicate an operation that is different from both the calculation and forwarding in the embodiments of this application, and may be the same as or different from the fifth operation mentioned above. Then, the second data block sent by the first communication device to the second communication device in S107 belongs to the updated data on the first communication device. In this way, in a task of collective communication, after the first calculation-related operation is completed, it is possible to wait for other operations to complete, and enter the communication process corresponding to the forwarding-related operation based on the updated data, which can be more flexibly adapted to other communication scenarios to improve communication efficiency. For example, the corresponding effect can be seen in Figure 4c.

[0134] S107, the first communication device sends a second data block to the second communication device, and the second data block corresponds to the third operation.

[0135] It is understandable that S101 to S106 above can be interpreted as the communication flow corresponding to the first operation. After S106, a third operation corresponding to the first operation can be executed. The third operation can be any forwarding-related operation, such as an AG operation or a Broadcast operation.

[0136] In some possible implementations, for the third operation, any one or more communication devices participating in the communication can send the third operation to the second communication device, so that the second communication device can forward the second data block based on the received third operation. The communication device sending the third operation to the second communication device can either carry the third operation and the corresponding data block in the same message or send them separately in different messages.

[0137] Taking a scenario where the communication device sending the third operation to the second communication device includes the first communication device, the method 100 may further include: the first communication device sending the third operation to the second communication device. This step can be performed before S107, simultaneously with S107, or after S107. As an example, the first communication device can send the third operation and the second data block in the same message to the second communication device; that is, S107 may include, for example, the first communication device sending a third message to the second communication device, which includes the second data block and the third operation. As another example, the first communication device can send the third operation and the second data block in different messages to the first communication device; that is, the first communication device sends messages 3 and 4 to the second communication device, where message 3 may carry the second data block and message 4 may carry the third operation.

[0138] As an example, the third operation could be an AG operation for the second data block. In this case, the first communication device sending the third operation to the second communication device could include, for example, the first communication device sending an AG operation for the second data block to the second communication device.

[0139] In some other possible implementations, if the second communication device obtains the third operation corresponding to the first operation in advance, the source of the third operation can be any entity capable of configuring operations to the second communication device, such as a controller or an upper-layer application. In this way, the communication device that does not need to participate in the communication sends the third operation to the second communication device, and each communication device participating in the communication can start executing the third operation after knowing that the first operation has been completed, for example, by executing S107.

[0140] It should be noted that after receiving the second data block that requires the third operation, the second communication device can copy the second data block and send it to the other communication devices participating in the communication. In the scenario shown in Figure 5, after S107, the second communication device receives the second data block and sends it to the third and fourth communication devices. Similarly, the third communication device can send the calculation result corresponding to the first operation to the second communication device, which then forwards the calculation result to the first and fourth communication devices; the fourth communication device can send the calculation result corresponding to the first operation to the second communication device, which then forwards the calculation result to the first and third communication devices.

[0141] S108, before receiving at least part of the third data block sent by the second communication device, the first communication device sends a fourth data block to the second communication device. The fourth data block is data that needs to be calculated. The third data block is associated with the second calculation result corresponding to the third communication device. The second calculation result corresponds to the first operation, the fourth data block corresponds to the second operation, the second operation is related to the calculation, and the third data block corresponds to the third operation.

[0142] It is understandable that, in order to improve communication efficiency, after S107 is executed, the first communication device can begin the second operation related to the calculation without waiting for the completion of the third operation. The completion of the third operation, as described in S108, corresponds to the first communication device receiving all data blocks from all participating communication devices for the third operation. Therefore, in this embodiment, the phrase "the first communication device sends a fourth data block to the second communication device before receiving at least a portion of the third data block sent by the second communication device" is used to express that once the first communication device completes the action of sending the second data block to the second communication device, it can send the fourth data block corresponding to the second operation to the second communication device.

[0143] The third data block can refer to any communication device other than the first communication device among the communication devices participating in the communication. After the third communication device sends the calculation result of the first operation to the second communication device, the second communication device may send it to the first communication device once or multiple times. Therefore, after sending the second data block, the first communication device will continuously receive data blocks forwarded by other communication devices through the second communication device. Before at least some data blocks are received, the first communication device sends the fourth data to the second communication device to achieve the purpose of parallelizing the forwarding related operations and the calculation related operations, thereby making the efficiency of the collective communication higher. The corresponding effect can be seen in Figure 4d.

[0144] It should be noted that after S108, the second communication device can perform calculations on the received fourth data block to obtain a third calculation result; the second communication device sends the third calculation result to the first communication device, and the first communication device obtains a seventh data block based on the third calculation result. The seventh data block is the calculation result corresponding to the first communication device in the second operation. In the ninth operation related to forwarding that is adjacent to the second operation after the second operation, the first communication device sends the seventh data block to the second communication device; if there is another calculation-related operation in this collective communication task, then the first communication device sends the ninth data block to the second communication device before receiving at least part of the eighth data block sent by the second communication device, thus initiating the tenth operation related to calculation that is adjacent to the ninth operation after the ninth operation, wherein the eighth data block corresponds to the fourth calculation result of the fourth communication device; if there is no next calculation-related operation in this collective communication task, then the first communication device is considered to have completed the ninth operation upon receiving all of the ninth data blocks sent by the second communication device.

[0145] Corresponding to the embodiment shown in Figure 5, if the first operation corresponds to RS operation 1, the third operation corresponds to AG operation 1, and the second operation can correspond to RS operation 2, then the process of S108 may include: after the first communication device sends the calculation result 11 to the second communication device, it immediately sends the data block 12 to the second communication device.

[0146] If the first operation corresponds to RS operation 2 in Figure 5, the third operation corresponds to AG operation 2, and the second operation can correspond to RS operation 3, then the process of S108 can include: after the first communication device sends the calculation result 21 to the second communication device, it immediately sends the data block 13 to the second communication device.

[0147] If the first operation corresponds to RS operation 3 in Figure 5, the third operation corresponds to AG operation 3, and there is no subsequent second operation in this collective communication task, then the process of S108 may include: the first communication device receiving the third data block sent by the second communication device from the other communication devices participating in the communication, thus completing the collective communication task.

[0148] It should be noted that in method 100, steps S101 to S106 are optional.

[0149] It should be noted that, for ease of understanding and description, the technical solutions provided in the embodiments of this application are described in method 100 by means of interaction between the first communication device and the second communication device. However, the steps performed by the first communication device can exist as separate embodiments in this application and have corresponding technical effects; the steps performed by the second communication device can also exist as separate embodiments in this application and have corresponding technical effects.

[0150] In this way, through method 100, the first communication device sends a second data block associated with the first calculation result to the second communication device. The first communication device does not need to wait to receive all the third data blocks sent by the second communication device before it can send another data block (i.e., the fourth data block) to be calculated. This is equivalent to the first communication device starting the third operation and then starting the second operation without waiting for the third operation to complete. By operating the second and third operations in parallel, the time waiting for the third operation to complete is saved, effectively shortening the communication time. This reduces the proportion of communication time in the collective communication, making rapid distributed AI training possible.

[0151] It should be noted that the technical solutions provided in the embodiments of this application are applicable to scenarios including but not limited to: Scenario 1, RS operation 1 + AG operation 1 + ... RS operation n + AG operation n; Scenario 2, RS operation 1 + other type of operation 1 + AG operation 1 + ... RS operation n + other type of operation 2, AG operation n, wherein the number and specific type of other type of operation are not limited, and can be between any RS operation and AG operation; Scenario 3, AG operation 0 + RS operation 1 + AG operation 1 + ... RS operation n + AG operation n, wherein AG operation n is an optional operation; Scenario 4, AG operation 0 + other type of operation 3 + RS operation 1 + AG operation 1 + ... RS operation n + AG operation n, wherein AG operation n is an optional operation, and the number and specific type of other type of operation are not limited, and can be between any RS operation and AG operation or between AG operation and RS operation.

[0152] Accordingly, this application also provides a communication device 600, as shown in FIG6. The device 600 is applied to the first communication device and may include at least a transceiver unit 601.

[0153] The transceiver unit 601 is configured to: send a first data block to a second communication device, the first data block being data to be calculated; receive a first calculation result from the second communication device, the first calculation result being obtained by the second communication device from each received first data block; send a second data block to the second communication device, the second data block being associated with the first calculation result; and before receiving at least a portion of the third data block sent by the second communication device, send a fourth data block to the second communication device, the fourth data block being data to be calculated, the third data block being associated with the second calculation result corresponding to the third communication device. The first data block, the first calculation result, and the second calculation result correspond to a first operation, the fourth data block corresponds to a second operation, the first and second operations are related to calculation, the second data block and the third data block correspond to a third operation, and the third operation is related to forwarding. This function of the transceiver unit 601 corresponds to the description related to the first communication device in Figure 3.

[0154] In some possible implementations, the first calculation result is obtained by the second communication device calculating the N first data blocks received, where N is the same as the number of communication devices participating in the communication. The communication devices participating in the communication include the first communication device and the third communication device, but do not include the second communication device.

[0155] In some possible implementations, the first calculation result is obtained by the second communication device calculating (N-1) first data blocks received, where N is the same as the number of communication devices participating in the communication, and the (N-1) first data blocks do not include the first data block corresponding to the first communication device. Therefore, the first communication device 600 further includes a processing unit 602. This processing unit 602 is used to obtain the second data block based on the first calculation result and the first data block corresponding to the first communication device after receiving the first calculation result from the second communication device before sending the second data block to the second communication device.

[0156] In some possible implementations, the transceiver unit 601 is further configured to send a fifth data block to the second communication device before sending the first data block to the second communication device. The fifth data block corresponds to a fourth operation, which is related to forwarding.

[0157] In some possible implementations, transceiver unit 601 is specifically configured to: send a first data block to the second communication device before receiving at least a portion of the sixth data block sent by the second communication device, the sixth data block corresponding to the fourth operation.

[0158] In some possible implementations, the first communication device 600 further includes a processing unit 602. This processing unit 602 is configured to update the data on the first communication device based on a fifth operation, after sending a fifth data block to the second communication device, before sending the first data block to the second communication device. The fifth operation indicates an operation different from both calculation and forwarding, and the first data block is the updated data on the first communication device.

[0159] In some possible implementations, the first communication device 600 further includes a processing unit 602. This processing unit 602 is configured to, after receiving the first calculation result from the second communication device and before sending the second data block to the second communication device, obtain the second data block based on the first calculation result and a sixth operation, the sixth operation being used to indicate an operation different from both the calculation and forwarding.

[0160] In some possible implementations, the first communication device 600 further includes a processing unit 602. This processing unit 602 is configured to obtain multiple sets of operations before sending the first data block to the second communication device. Each set of operations includes a computation-related operation and a forwarding-related operation, with the first and third operations belonging to one set of operations within the multiple sets of operations.

[0161] As an example, processing unit 602 is specifically used to: obtain multiple sets of operations based on the seventh operation, which is used to instruct All-Reduce to be performed.

[0162] As another example, the processing unit 602 is specifically used to: obtain M first-type operations based on the eighth operation, and obtain M second-type operations based on the ninth operation, where M is an integer greater than 1, and the M first-type operations and M second-type operations correspond one-to-one. The eighth operation is related to computation, and the ninth operation is related to forwarding. Each first-type operation in the M first-type operations and the corresponding second-type operation are recorded as a group of operations to obtain multiple groups of operations.

[0163] In some possible implementations, the transceiver unit 601 is also used to send the first operation to the second communication device.

[0164] As an example, transceiver unit 601 is specifically used to: send a first message to a second communication device, the first message including a first data block and a first operation.

[0165] As another example, the first communication device 600 further includes a processing unit 602. This processing unit 602 is configured to obtain N Reduce operations based on the first operation, each of the N Reduce operations instructing computation on a first data block corresponding to one of the participating communication devices, where N is the same as the number of participating communication devices. Then, the transceiver unit is further configured to send the N Reduce operations to the second communication device. In this example, the transceiver unit is specifically configured to send a second message to the second communication device, the second message including the first data block and the Reduce operations.

[0166] In some possible implementations, the transceiver unit 601 is also used to send a third operation to the second communication device.

[0167] As an example, transceiver unit 601 is specifically used to send a third message to a second communication device, the third message including a second data block and a third operation.

[0168] The first and second operations are Reduce-Scatter operations, and the third operation is an All-Gather operation or a Broadcast operation.

[0169] As an example, if the second data block corresponds to the All-Gather operation, the processing unit 602 of the device 600 is further configured to: obtain the corresponding Broadcast operation based on the All-Gather operation, and record the Broadcast operation as the third operation.

[0170] It should be noted that for relevant descriptions of the first communication device 600, please refer to the corresponding description in the method provided in the embodiments of this application.

[0171] It should be noted that the division of units in this embodiment is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. The functional units in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. For example, in the above embodiment, the processing unit and the transceiver unit can be the same unit or different units. The integrated unit can be implemented in hardware or as a software functional unit.

[0172] Accordingly, this application also provides a communication device 700, as shown in FIG7. The communication device 700 includes a processor 701 and a memory 702; the processor 701 is used to execute instructions stored in the memory 702 so that the communication device 700 implements the method provided in this application.

[0173] Accordingly, this application also provides a communication device 800, as shown in FIG8. The communication device 800 includes an interface 801 and a processor 802; the interface 801 is used to receive instructions and transmit them to the processor 802; the processor 802 is used to implement the method provided in this application embodiment.

[0174] Accordingly, this application provides a chip 900, as shown in FIG9. The chip 900 includes an interface circuit 901 and a processing circuit 902, with the interface circuit 901 connected to the processing circuit 902. The interface circuit 901 is used to execute the receiving and transmitting operations in the method provided in this application embodiment; the processing circuit 902 is used to execute other operations in the method provided in this application embodiment besides the receiving and transmitting operations.

[0175] Furthermore, this application also provides a communication system 1000, as shown in FIG10. The communication system 1000 includes a first communication device 1001 and a second communication device 1002; the first communication device 1001 is used to implement the operations related to the first communication device in the method provided in this application embodiment; the second communication device is used to implement the operations related to the second communication device in the method provided in this application embodiment.

[0176] As an example, in this communication system 1000, the interaction between the first communication device 1001 and the second communication device 1002 includes, but is not limited to:

[0177] The first communication device 1001 is used to send a first data block to the second communication device 1002, wherein the first data block is data that needs to be calculated.

[0178] The second communication device 1002 is used to calculate each of the received first data blocks to obtain a first calculation result, and send the first calculation result to the first communication device 1001;

[0179] The first communication device 1001 is further configured to receive a first calculation result from the second communication device 1002, obtain a second data block based on the first calculation result, and send the second data block to the second communication device 1002.

[0180] The first communication device 1001 is further configured to send a fourth data block to the second communication device 1002 before receiving at least a portion of the third data block sent by the second communication device 1002. The fourth data block is data to be calculated. The third data block is associated with a second calculation result corresponding to the third communication device. The first data block, the first calculation result, and the second calculation result correspond to a first operation. The fourth data block corresponds to a second operation. The first and second operations are related to calculation. The second data block and the third data block correspond to a third operation. The third operation is related to forwarding.

[0181] It should be noted that the modules or units with corresponding functions in the above communication device can be understood as interchangeable. For example, the transceiver unit 601 can correspond to the interface 801 or the interface circuit 901; the processing unit 602 can correspond to the memory 701, the memory 802, or the processing circuit 902.

[0182] Furthermore, this application embodiment also provides a storage medium storing program code or instructions, which, when run on a processor, cause the processor to execute the method under any of the above embodiments.

[0183] Furthermore, this application also provides a program product that, when run on a processor, causes the processor to execute the method under any of the aforementioned implementations.

[0184] It should be understood that "determining B based on A" mentioned in the embodiments of this application does not mean determining B solely based on A, but also determining B based on A and / or other information.

[0185] It should be understood that the network architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0186] In this application, ordinal numbers such as “1”, “2”, “3”, “first”, “second”, and “third” are used to distinguish multiple objects, not to limit the order of multiple objects.

[0187] The reference to "A and / or B" in this application should be understood to include the following situations: including only A, including only B, or including both A and B.

[0188] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus a general-purpose hardware platform. Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a router) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0189] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system and device embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device and system embodiments described above are merely illustrative. Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0190] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A communication method, characterized in that, Applied to a first communication device, the method includes: Send a first data block to the second communication device, wherein the first data block is data that needs to be calculated; The second communication device receives a first calculation result, which is obtained by the second communication device from calculating each first data block received. Send a second data block to the second communication device, the second data block being associated with the first calculation result; Before receiving at least a portion of the third data block sent by the second communication device, a fourth data block is sent to the second communication device. The fourth data block is data that needs to be calculated. The third data block is associated with a second calculation result corresponding to the third communication device. The first data block, the first calculation result, and the second calculation result correspond to a first operation. The fourth data block corresponds to a second operation. The first operation and the second operation are related to calculation. The second data block and the third data block correspond to a third operation. The third operation is related to forwarding.

2. The method according to claim 1, characterized in that, The first calculation result is obtained by the second communication device from the N first data blocks received, where N is the same as the number of communication devices participating in the communication. The communication devices participating in the communication include the first communication device and the third communication device, but do not include the second communication device.

3. The method according to claim 1, characterized in that, The first calculation result is obtained by the second communication device calculating (N-1) first data blocks received, where N is the same as the number of communication devices participating in the communication. The (N-1) first data blocks do not include the first data block corresponding to the first communication device. Therefore, before sending the second data block to the second communication device and after receiving the first calculation result from the second communication device, the method further includes: The second data block is obtained based on the first calculation result and the first data block corresponding to the first communication device.

4. The method according to any one of claims 1-3, characterized in that, Before sending the first data block to the second communication device, the method further includes: A fifth data block is sent to the second communication device, the fifth data block corresponding to the fourth operation, the fourth operation being related to forwarding.

5. The method according to claim 4, characterized in that, Sending the first data block to the second communication device includes: Before receiving at least a portion of the sixth data block sent by the second communication device, the first data block, corresponding to the fourth operation, is sent to the second communication device.

6. The method according to claim 5, characterized in that, Before sending the first data block to the second communication device, and after sending the fifth data block to the second communication device, the method further includes: Based on the fifth operation, the data on the first communication device is updated. The fifth operation is used to indicate an operation that is different from both the calculation and the forwarding. The first data block is the updated data on the first communication device.

7. The method according to any one of claims 1-6, characterized in that, After receiving the first calculation result from the second communication device and before sending the second data block to the second communication device, the method further includes: Based on the first calculation result and the sixth operation, the second data block is obtained, wherein the sixth operation is used to indicate an operation that is different from both the calculation and the forwarding.

8. The method according to any one of claims 1-7, characterized in that, Before sending the first data block to the second communication device, the method further includes: Multiple sets of operations are obtained, each set of operations including an operation related to the computation and an operation related to the forwarding, wherein the first operation and the third operation belong to one set of operations in the multiple sets of operations.

9. The method according to claim 8, characterized in that, The acquisition of multiple sets of operations includes: The plurality of operations are obtained according to the seventh operation, which is used to instruct the performance of All-Reduce.

10. The method according to claim 8, characterized in that, The acquisition of multiple sets of operations includes: M first-type operations are obtained according to the eighth operation, and the M second-type operations are obtained according to the ninth operation, where M is an integer greater than 1. The M first-type operations and the M second-type operations correspond one-to-one. The eighth operation is related to the calculation, and the ninth operation is related to the forwarding. Each of the M first-type operations and its corresponding second-type operation is denoted as a group of operations to obtain the multiple groups of operations.

11. The method according to any one of claims 1-10, characterized in that, The method further includes: Send the first operation to the second communication device.

12. The method according to claim 11, characterized in that, Sending the first data block to the second communication device includes: Send a first message to the second communication device, the first message including the first data block and the first operation.

13. The method according to any one of claims 1-10, characterized in that, The method further includes: Based on the first operation, N reduction operations are obtained, each of the N reduction operations being used to instruct the computation of a first data block corresponding to a communication device participating in the communication, where N is the same as the number of communication devices participating in the communication.

14. The method according to claim 13, characterized in that, Sending the first data block to the second communication device includes: A second message is sent to the second communication device, the second message including the first data block and the Reduce operation.

15. The method according to any one of claims 1-14, characterized in that, The method further includes: The third operation is sent to the second communication device.

16. The method according to claim 15, characterized in that, Sending the second data block to the second communication device includes: A third message is sent to the second communication device, the third message including the second data block and the third operation.

17. The method according to any one of claims 1-16, characterized in that, The first and second operations are Reduce-Scatter operations, and the third operation is an All-Gather operation or a Broadcast operation.

18. The method according to claim 17, characterized in that, The second data block corresponds to the All-Gather operation, and the method further includes: The corresponding Broadcast operation is obtained based on the All-Gather operation, and the Broadcast operation is denoted as the third operation.

19. A communication device, characterized in that, Applied to a first communication device, the first communication device comprising: A transceiver unit is configured to perform the receiving and transmitting operations in the method described in any one of claims 1-18 above; The processing unit is used to perform other processing operations besides the receiving operation and the sending operation in the method described in any one of claims 1-18.

20. A communication device, characterized in that, The communication device includes a memory and a processor; The memory is used to store instructions; The processor is configured to execute the instructions in the memory and perform the method described in any one of claims 1-18.

21. A chip, characterized in that, The chip includes an interface circuit and a processing circuit, and the interface circuit is connected to the processing circuit. The interface circuit is used to perform the receiving and sending operations in the method described in any one of claims 1-18 above; The processing circuit is used to perform operations other than receiving and sending operations in the method described in any one of claims 1-18.

22. A communication device, characterized in that, The communication device includes an interface and a processor; The interface is used to receive instructions and transmit them to the processor; The processor is configured to perform the method described in any one of claims 1-18.

23. A communication system, characterized in that, The communication system includes a first communication device and a second communication device; the first communication device is used to implement the operation related to the first communication device in any one of claims 1-18; the second communication device is used to implement the operation related to the second communication device in claims 1-18.

24. A storage medium, characterized in that, The storage medium includes instructions that, when executed on a processor, cause the processor to perform the method described in any one of claims 1-18.

25. A program product, characterized in that, The program product includes a program that, when run on a processor, causes the processor to perform the method described in any one of claims 1-18.

Images