Collective communication method and computing device cluster

Abstract

The present application relates to the field of computer technology, and discloses a collective communication method and a computing device cluster. The method comprises: during the process of N computing units in a computing device cluster executing a collective communication operation, a first computing unit receives first data from a second computing unit and / or sends second data to a third computing unit, wherein when data in each computing unit among the N computing units is greater than a first threshold, the data in each computing unit is segmented into N pieces of slice data, the number of pieces of slice data in the second computing unit comprised in the first data is inversely proportional to a physical distance between the first computing unit and the second computing unit, and the number of pieces of slice data in the first computing unit comprised in the second data is inversely proportional to a physical distance between the first computing unit and the third computing unit. Thus, cross‑switch traffic conflicts during the process of N computing units executing a collective communication operation can be reduced, and the efficiency of collective communication can be improved.

Description

A clustered communication method and computing device cluster

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese Patent Application No. 202411518348.X, filed on October 28, 2024, entitled "A Cluster Communication Method and Computing Device Cluster", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of computer technology, and in particular to a clustered communication method and a cluster of computing devices. Background Technology

[0004] In the process of training artificial intelligence (AI) models or high-performance computing (HPC), collective communication operations are required to complete data distribution or synchronization between multiple computing units in a computing device cluster.

[0005] Currently, aggregate communication operations can be implemented using communication algorithms such as the ring algorithm and the halving doubling (HD) algorithm. The ring algorithm connects multiple computing units into a ring structure, where communication occurs between these units. Each unit only communicates with its left and right adjacent units. During ring communication, data is transmitted along the ring structure until communication is complete and the data reaches its destination unit. The HD algorithm, on the other hand, involves pairwise communication between multiple computing units. The communication distance and the amount of data transmitted are halved or doubled at each step of the HD algorithm's implementation.

[0006] However, for aggregate communication operations based on ring algorithms, the number of communication steps and communication latency are directly proportional to the number of computing units in the computing device cluster. Therefore, the more computing units in the cluster, the more communication steps and latency there are, resulting in lower communication efficiency. For aggregate communication operations based on the HD algorithm, computing device clusters with a power of two computing units have a theoretically optimal number of communication steps at the logarithmic level and lower communication latency. However, for computing device clusters with a non-power of two computing units, there is no theoretically optimal number of communication steps at the logarithmic level, resulting in higher communication latency. Furthermore, in each step of the HD algorithm, the communication object changes, which can easily lead to cross-switch traffic conflicts in high-traffic scenarios, resulting in lower communication efficiency. Summary of the Invention

[0007] This application provides a collective communication method and a computing device cluster to improve the efficiency of collective communication.

[0008] In a first aspect, embodiments of this application provide a collective communication method applied to a computing device cluster comprising N computing units, where N is a positive integer. The method includes: during the collective communication operation performed by the N computing units, a first computing unit receives first data from a second computing unit and / or sends second data to a third computing unit; wherein, when the data in each of the N computing units exceeds a first threshold, the data in each computing unit is divided into N slices, the number of slices from the second computing unit included in the first data being inversely proportional to a first physical distance, and the number of slices from the first computing unit included in the second data being inversely proportional to a second physical distance, where the first physical distance is the physical distance between the first computing unit and the second computing unit, and the second physical distance is the physical distance between the first computing unit and the third computing unit.

[0009] In this embodiment, if the data in each of the N computing units is large, the data in each computing unit can be divided into N slices. For each of the N slices in the computing unit, during the execution of the aggregated communication operation, the communication step with the largest amount of communication data (i.e., the largest number of slices sent and / or received) occurs between the computing units with the smallest physical distance, and the communication step with the smallest amount of communication data (i.e., the smallest number of slices sent and / or received) occurs between the computing units with the largest physical distance. Since the smaller the physical distance between computing units, the fewer switches are needed for communication between computing units, and the larger the physical distance between computing units, the more switches are needed for communication between computing units, this reduces cross-switch traffic conflicts during the aggregated communication operation of the N computing units, thus improving the efficiency of aggregated communication.

[0010] In one possible implementation, the N computing units are adjacent in number or the computing units with the smallest and largest in number are considered adjacent computing units, and the physical distance between any one of the N computing units and an adjacent computing unit is less than the physical distance between any one computing unit and a non-adjacent computing unit.

[0011] This embodiment provides multiple methods for determining the physical distance between computing units. For example, for two computing units with adjacent serial numbers among N computing units, the physical distance between these two computing units is small; or, for two computing units with the smallest and largest serial numbers among N computing units, the physical distance between these two computing units is small. Besides these, there are other methods for determining the physical distance between computing units, and these are not limited.

[0012] In one possible implementation, the data of each of the N computing units is greater than the first threshold, and the aggregate communication operation includes a global collection operation, which comprises P steps. As a rounding operator, the method further includes: during the (K+1)th step of the global collection operation performed by the N computing units, the first computing unit receives the first data from the second computing unit and sends the second data to the third computing unit; wherein, the first computing unit is the (i+1)th computing unit among the N computing units, and the second computing unit is the (i-2)th computing unit among the N computing units. P-K-1 +N)%N+1 calculation units, the third calculation unit is the (i+2)th calculation unit among the N calculation units. P-K-1 The system has N+1 computational units, where K is a natural number less than or equal to P-1, i is a natural number less than or equal to N-1, and % is the modulo operator; the first data includes D slices of data from the second computational unit, and the second data includes D slices of data from the first computational unit. This is the rounding operator.

[0013] In this embodiment, one possible set of steps for N computing units to perform a global collection operation is provided. For example, a global collection operation performed by 6 computing units may include 3 steps: In the first step, the communication interval is 4, and the communication data volume is 1 slice of data, such as the first computing unit receiving 1 slice of data from the third computing unit and sending 1 slice of data to the fifth computing unit; In the second step, the communication interval is 2, and the communication data volume is 1 slice of data, such as the first computing unit receiving 1 slice of data from the fifth computing unit and sending 1 slice of data to the third computing unit; In the third step, the communication interval is 1, and the communication data volume is 3 slices of data, such as the first computing unit receiving 3 slices of data from the sixth computing unit and sending 3 slices of data to the second computing unit. In other words, during the global collection operation, the communication steps with the largest amount of communication data occur between computing units with the smallest communication interval. These computing units with the smallest communication interval are also the computing units with the smallest physical distance. Since the smaller the physical distance between computing units, the fewer switches they need to traverse, thus reducing cross-switch traffic conflicts during the global collection operation. Furthermore, whether the number of computing units in the computing device cluster is a power of 2 or not, it has a theoretically optimal number of communication steps at the logarithmic level, resulting in lower communication latency.

[0014] In one possible implementation, the data in each of the N computing units is greater than the first threshold, and the aggregate communication operation includes a reduction-distribution operation, which comprises P steps. As a rounding operator, the method further includes: during the (K+1)th step of the reduction and scattering operation performed by the N computing units, the first computing unit receives the first data from the second computing unit and sends the second data to the third computing unit; wherein, the first computing unit is the (i+1)th computing unit among the N computing units, and the second computing unit is the (i+2)th computing unit among the N computing units. K The third calculation unit is the (i-2)th calculation unit among the N calculation units. k +N)%N+1 computational units, where K is a natural number less than or equal to P-1, i is a natural number less than or equal to N-1, and % is the modulo operator; the first data includes D slice data from the second computational unit, and the second data includes D slice data from the first computational unit. This is the rounding operator.

[0015] In this embodiment, one possible set of steps for N computing units to perform a reduction distribution operation is provided. For example, a reduction distribution operation performed by 6 computing units may include 3 steps: In the first step, the communication interval is 1, and the communication data volume is 3 slices of data, such as the first computing unit receiving 3 slices of data from the second computing unit and sending 3 slices of data to the sixth computing unit; In the second step, the communication interval is 2, and the communication data volume is 1 slice of data, such as the first computing unit receiving 1 slice of data from the third computing unit and sending 1 slice of data to the fifth computing unit; In the third step, the communication interval is 4, and the communication data volume is 1 slice of data, such as the first computing unit receiving 1 slice of data from the fifth computing unit and sending 1 slice of data to the third computing unit. In other words, during the reduction and distribution operation, the communication steps with the largest amount of communication data occur between computing units with the smallest communication interval. These computing units with the smallest communication interval are also the computing units with the smallest physical distance. Since the smaller the physical distance between computing units, the fewer switches are needed for communication between them, thus reducing cross-switch traffic conflicts during the reduction and distribution operation. Furthermore, whether the number of computing units in the computing device cluster is a power of 2 or not, it has a theoretically optimal number of communication steps at the logarithmic level, resulting in lower communication latency.

[0016] In one possible implementation, the data in each of the N computing units is greater than the first threshold, and the aggregate communication operation includes a distribution operation, which comprises P steps. The method further includes, as a rounding operator, the following steps: during the (K+1)th step of the scattering operation performed by the N computing units, if the index distance between the first computing unit and the fourth computing unit is greater than or equal to a second threshold and less than a third threshold, then the first computing unit receives the first data from the second computing unit; or, during the (K+1)th step of the scattering operation performed by the N computing units, if the index distance between the first computing unit and the fourth computing unit is less than a second threshold, then the first computing unit sends the second data to the third computing unit; wherein, the first computing unit is the (i+1)th computing unit among the N computing units, and the second computing unit is the (i+2)th computing unit among the N computing units. K The third calculation unit is the (i-2)th calculation unit among the N calculation units. K+N)%N+1 computation units, wherein the fourth computation unit is a computation unit that stores data in the N computation units before the distributed operation is performed by the N computation units, K is a natural number less than or equal to P-1, i is a natural number less than or equal to N-1, and % is the modulo operator; the first data includes D slice data in the second computation unit, and the second data includes D slice data in the first computation unit. round(*) is the rounding operator.

[0017] This implementation provides one possible set of steps for N computing units to perform distributed operations. For example, distributed operations performed by 6 computing units may include 3 steps: in the first step, the communication interval is 1, and the communication data volume is 3 slices of data; in the second step, the communication interval is 2, and the communication data volume is 1 slice of data; in the third step, the communication interval is 4, and the communication data volume is 1 slice of data. That is, during the distributed operation, the communication steps with the largest communication data volume occur between computing units with the smallest communication interval. The computing unit with the smallest communication interval is also the computing unit with the smallest physical distance. Since the smaller the physical distance between computing units, the fewer switches are needed for communication between computing units, thus reducing cross-switch traffic conflicts during the distributed operation. Furthermore, whether the number of computing units in the computing device cluster is a power of 2 or not, it has a theoretically optimal number of communication steps at the logarithmic level, resulting in lower communication latency.

[0018] In one possible implementation, when the fourth computing unit is the (j+1)th computing unit among the N computing units, the index distance between the first computing unit and the fourth computing unit satisfies the following relationship: W1=(j-i+N)%N

[0019] Where W1 is the sequence distance between the first calculation unit and the fourth calculation unit, and j is a natural number less than or equal to N-1.

[0020] When N is not a power of 2 and K equals P-1, the second threshold and the third threshold satisfy the following relationship:

[0021] When N is a power of 2 or K is not equal to P-1, the second threshold and the third threshold satisfy the following relationship:

[0022] Wherein, Q1 is the second threshold and Q2 is the third threshold.

[0023] In this embodiment, specific calculation formulas are provided for determining the sequence distance, second threshold, and third threshold between the first and fourth computing units during the execution of distributed operations, thereby improving the determination efficiency.

[0024] In one possible implementation, the first data includes the (d1-2)th data in the second computing unit. P-K ·m)%N+1 slice data, the second data includes the (d2-2)th slice data in the first computing unit. P-K ·m)%N+1 slices of data; where d1=(i-2 P-K-1 +N)%N, d2=i, m is a natural number less than D.

[0025] In this embodiment, a specific calculation method is provided for determining the sequence numbers of D slice data in the second computing unit, which are included in the first data, and the sequence numbers of D slice data in the first computing unit, which are included in the second data, during the global collection operation, thereby improving the determination efficiency.

[0026] In one possible implementation, the first data includes the (d1-2)th data in the second computing unit. K+1 ·m)%N+1 slice data, the second data includes the (d2-2)th slice data in the first computing unit. K+1 ·m)%N+1 slices of data; where d1=i, d2=(i-2 K +N)%N, where m is a natural number less than D.

[0027] In this embodiment, a specific calculation method is provided for determining the sequence numbers of the D slice data in the second computing unit included in the first data, and the sequence numbers of the D slice data in the first computing unit included in the second data, during the execution of the scattering operation and the reduction scattering operation, thereby improving the determination efficiency.

[0028] In one possible implementation, all or part of the D slice data are slice data with consecutive sequence numbers.

[0029] In this embodiment, in order to reduce the number of communication tasks issued during the execution of reduction-distribution operation, global collection operation, global reduction operation, distribution operation, or broadcast operation, the D slice data in the second computing unit included in the first data and the D slice data in the first computing unit included in the second data can be reordered so that all or part of the slice data in these D slice data are slice data with consecutive order numbers, thereby improving communication performance.

[0030] In one possible implementation, the data of each of the N computing units is less than or equal to the first threshold, and the set communication operation includes a one-step reduction operation, which comprises P steps. As a rounding operator, the method further includes: during the (K+1)th step of the one-step reduction operation performed by the N computing units, if the ordinal distance between the first computing unit and the first computing unit among the N computing units satisfies a first relationship, then the first computing unit receives the first data from the second computing unit; or, during the (K+1)th step of the one-step reduction operation performed by the N computing units, if the ordinal distance between the first computing unit and the first computing unit among the N computing units satisfies a second relationship, then the first computing unit sends the second data to the third computing unit; wherein, the first computing unit is the (i+1)th computing unit among the N computing units, and the second computing unit is the (i+2)th computing unit among the N computing units. K The third calculation unit is the (i-2)th calculation unit among the N calculation units. K +N)%N+1 calculation units, K is a natural number less than or equal to P-1, i is a natural number less than or equal to N-1, and % is the modulo operator; the first data includes all data in the second calculation unit, and the second data includes all data in the first calculation unit.

[0031] This implementation provides one possible set of steps for N computing units to perform a one-step reduction operation. When the amount of communication data between the N computing units is small, the communication data can be left unsegmented during the one-step reduction operation, allowing the communication data to be contiguous in memory. This reduces the number of communication tasks issued during the one-step reduction operation, thus improving communication performance. Furthermore, whether the number of computing units in the computing device cluster is a power of 2 or not, it achieves a theoretically optimal number of communication steps at the logarithmic level, resulting in lower communication latency.

[0032] In one possible implementation, the first relation is specifically:

[0033] The second relationship is specifically as follows:

[0034] Wherein, W2 is the sequence distance between the first computing unit and the first computing unit among the N computing units.

[0035] In this embodiment, specific calculation formulas for determining the first and second relations are provided during the one-step reduction operation, thereby improving the determination efficiency.

[0036] In one possible implementation, the data of each of the N computing units is less than or equal to the first threshold, and the aggregate communication operation includes a one-step broadcast operation, which comprises P steps. As a rounding operator, the method further includes: during the (K+1)th step of the one-step broadcast operation performed by the N computing units, if the sequence distance between the first computing unit and the fifth computing unit satisfies a third relationship, then the first computing unit receives the first data from the second computing unit; or, during the (K+1)th step of the one-step broadcast operation performed by the N computing units, if the sequence distance between the first computing unit and the fifth computing unit satisfies a fourth relationship, then the first computing unit sends the second data to the third computing unit; wherein, the first computing unit is the (i+1)th computing unit among the N computing units, and the second computing unit is the (i-2)th computing unit among the N computing units. P-K-1 +N)%N+1 calculation units, the third calculation unit is the (i+2)th calculation unit among the N calculation units. P-K-1 The system comprises N+1 computational units, wherein the fifth computational unit is a computational unit that stores the data to be broadcast in the N computational units before the N computational units perform the broadcast operation, K is a natural number less than or equal to P-1, i is a natural number less than or equal to N-1, and % is the modulo operator; the first data includes all the data in the second computational unit, and the second data includes all the data in the first computational unit.

[0037] This implementation provides one possible set of steps for N computing units to perform a one-step broadcast operation. When the amount of communication data between the N computing units is small, the communication data can be kept contiguous in memory during the one-step broadcast operation without being segmented, thus reducing the number of communication tasks sent during the operation and improving communication performance. Furthermore, whether the number of computing units in the computing device cluster is a power of 2 or not, it achieves a theoretically optimal number of communication steps at the logarithmic level, resulting in lower communication latency.

[0038] In one possible implementation, when the fifth computational unit is the (j+1)th computational unit among the N computational units, the third relation is specifically as follows:

[0039] The fourth relationship is as follows:

[0040] Where j is a natural number less than or equal to N-1, and W2 is the ordinal distance between the first calculation unit and the fifth calculation unit.

[0041] In this embodiment, specific calculation formulas for determining the third and fourth relationships are provided during the execution of a one-step broadcast operation, thereby improving the determination efficiency.

[0042] In one possible implementation, the collection communication operation includes one or more of the following: reduction-distribution operation, global collection operation, global reduction operation, distribution operation, broadcast operation, one-step reduction operation, and one-step broadcast operation;

[0043] Wherein, if the data in each of the N computing units is greater than the first threshold, the global reduction operation includes the reduction-distribution operation and the global collection operation, and the broadcast operation includes the distribution operation and the global collection operation; or...

[0044] When the data in each of the N computing units is less than or equal to the first threshold, the global reduction operation includes the one-step reduction operation and the one-step broadcast operation, and the broadcast operation includes the one-step broadcast operation.

[0045] This implementation provides various possibilities for aggregated communication operations, such as reduction-distribution operations, global collection operations, global reduction operations, distribution operations, broadcast operations, one-step reduction operations, and one-step broadcast operations. In addition, aggregated communication operations can also include other operations without limitation. Furthermore, when the amount of communication data between N computing units is small, the global reduction operation can include one-step reduction operations and one-step broadcast operations, and the broadcast operation can include one-step broadcast operations. That is, the communication data does not need to be segmented, allowing the communication data to be contiguous in memory, thereby reducing the number of communication tasks issued during the execution of global reduction and broadcast operations and improving communication performance.

[0046] Secondly, this application also provides a clustered communication device, which can exist independently or be located within computing units of a computing device cluster. This device has the function of implementing the methods described in the first aspect or any possible design of the first aspect. This function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described functions, such as an interface unit, a processing unit, and a storage unit. The interface unit can be a human-computer interaction interface, a human-computer interaction interface, or an input / output port of a device, etc., and this application embodiment does not limit its scope. The processing unit can be a processor or a processing chip, etc., and this application embodiment does not limit its scope either. The storage unit can be used to store multiple indexes; of course, the storage unit can also be located in other devices, and this application embodiment does not limit its scope either.

[0047] An interface unit is configured to, during the execution of a collective communication operation by N computing units in a computing device cluster, have a first computing unit receive first data from a second computing unit and / or send second data to a third computing unit; wherein N is a positive integer, and when the data in each of the N computing units is greater than a first threshold, the data in each computing unit is divided into N slices of data, the number of slices of data in the second computing unit included in the first data is inversely proportional to a first physical distance, and the number of slices of data in the first computing unit included in the second data is inversely proportional to a second physical distance, wherein the first physical distance is the physical distance between the first computing unit and the second computing unit, and the second physical distance is the physical distance between the first computing unit and the third computing unit.

[0048] Thirdly, embodiments of this application also provide a computing device cluster, including at least one computing device, each computing device comprising a processor and a memory. The processor is configured to support the computing unit in executing the methods described in the first aspect or any possible design of the first aspect. The memory is coupled to the processor and stores program instructions and data necessary for the communication device. The communication device may further include a communication interface for communicating with other devices.

[0049] Fourthly, embodiments of this application also provide a computer-readable storage medium comprising computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in the first aspect or any possible design of the first aspect.

[0050] Fifthly, embodiments of this application also provide a computer program product containing instructions that, when the instructions are run by a computing device cluster, cause the computing device cluster to perform the method described in the first aspect or any possible design of the first aspect.

[0051] In a sixth aspect, embodiments of this application also provide a chip that can be coupled to a memory for calling a computer program stored in the memory to execute the method described in the first aspect or any possible design of the first aspect.

[0052] The beneficial effects of the second to sixth aspects and their possible designs can be referred to the description of the beneficial effects of the methods described in the first aspect and any of its possible designs. Attached Figure Description

[0053] Figure 1 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0054] Figure 2a is a schematic diagram of a global reduction operation, a reduction distribution operation, and a global collection operation provided in an embodiment of this application;

[0055] Figure 2b is a schematic diagram of performing a global reduction operation based on a ring algorithm according to an embodiment of this application;

[0056] Figure 2c is a schematic diagram of performing a global reduction operation based on the HD algorithm according to an embodiment of this application;

[0057] Figure 3 is a flowchart illustrating a collection communication method provided in an embodiment of this application;

[0058] Figure 4a is a schematic diagram of adjacent computing units and non-adjacent computing units in a computing device cluster provided in an embodiment of this application;

[0059] Figure 4b is a schematic diagram of performing a reduction and dispersion operation according to an embodiment of this application;

[0060] Figure 4c is a schematic diagram of performing a global collection operation according to an embodiment of this application;

[0061] Figure 4d is a schematic diagram of performing a distributed operation according to an embodiment of this application;

[0062] Figure 4e is a schematic diagram of another reduction and dispersion operation provided by an embodiment of this application;

[0063] Figure 4f is a schematic diagram of another global collection operation provided by an embodiment of this application;

[0064] Figure 4g is a schematic diagram of performing a one-step reduction operation according to an embodiment of this application;

[0065] Figure 4h is a schematic diagram of performing a one-step broadcast operation according to an embodiment of this application;

[0066] Figure 5 is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0067] Figure 6 is a schematic diagram of another computing device cluster provided in an embodiment of this application;

[0068] Figure 7 is a schematic diagram of a connection method between computing device clusters provided in an embodiment of this application. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the embodiments of this application will be described in detail below with reference to the accompanying drawings. The terminology used in the implementation section of this application is only for explaining specific embodiments of this application and is not intended to limit this application.

[0070] The technical solutions provided in this application can be applied to computing device clusters, which may include one or more computing devices (or computing nodes) and one or more aggregation switches. The aggregation switches are used to connect multiple computing devices in the computing device cluster. It is understood that the computing device cluster provided in this application may include other devices besides computing devices and aggregation switches, and this application does not limit this. Furthermore, any device in this application can integrate its functions onto a single physical unit or distribute its functions across multiple independent physical units, and this application does not limit this either.

[0071] The computing device can be any device with computing and communication capabilities, including but not limited to servers, desktop computers, laptops or portable computers, handheld devices, netbooks, internet devices, tablets, and mobile devices (such as mobile phones, personal digital assistants, smartphones, etc.). This application embodiment does not limit the scope of the computing device. The computing device may include one or more computing units and one or more access switches. The computing units are used to perform aggregated communication operations and include, but are not limited to, graphics processing units (GPUs), neural network processing units (NPUs), data processing units (DPUs), and tensor processing units (TPUs). This application embodiment does not limit the scope of these components. The access switches are used to connect multiple computing units within the computing device. It is understood that the computing device provided in this application embodiment may include other devices besides computing units and access switches, and this application embodiment does not limit the scope of these devices.

[0072] Any two computing units in a computing device cluster can be connected via an access switch, or via both an access switch and an aggregation switch. In other words, multiple computing units that can be connected via only an access switch are considered as computing units within the same computing device, while multiple computing units that can be connected via both an access switch and an aggregation switch are considered as computing units within different computing devices.

[0073] In this computing device cluster, the first layer from bottom to top includes L1 computing units, the second layer includes L2 access switches, and the third layer includes L3 aggregation switches, where L1, L2, and L3 are all positive integers. In one possible example, the number of access switches L2 can be determined by the number of downlink ports of the access switches, the number of uplink ports of the computing units, and the number of computing units L1; the number of aggregation switches L3 can be determined by the number of downlink ports of the aggregation switches, the number of uplink ports of the access switches, and the number of access switches L2.

[0074] For example, Figure 1 is a schematic diagram of a computing device cluster provided in an embodiment of this application. As shown in Figure 1, the computing device cluster includes two computing devices and two aggregation switches, namely computing device 0, computing device 1, aggregation switch 0, and aggregation switch 1.

[0075] The computing device 0 includes four computing units and three access switches, namely computing units 0 to 3 and access switches 0 to 2. The computing device 1 includes four computing units and three access switches, namely computing units 4 to 7 and access switches 3 to 4.

[0076] In this computing device cluster, computing units 0 through 3 are all connected to any one of the access switches: access switch 0, access switch 1, and access switch 2; computing units 4 through 7 are all connected to any one of the access switches: access switch 3, access switch 4, and access switch 5. Access switches 0 through 5 are all connected to either aggregation switch 0 or aggregation switch 1.

[0077] It is understood that the number of computing devices, computing units, access switches and aggregation switches in Figure 1 are exemplary, and the computing device cluster may include a greater number of computing devices, computing units, access switches and aggregation switches.

[0078] The above briefly introduces the computing device clusters applicable to the embodiments of this application. The following describes the relevant technical solutions involved in the embodiments of this application.

[0079] In the process of training artificial intelligence (AI) models or high-performance computing (HPC), collective communication operations are required to complete data distribution or synchronization between multiple computing units in a computing device cluster.

[0080] Collective communication operations may include broadcast operations, scatter operations, gather operations, all-gather operations, reduce operations, all-reduce operations, reduce-scatter operations, etc., and the embodiments of this application do not limit them.

[0081] In a broadcast operation, there is one data sender and multiple data receivers, which can send data from one computing unit in a computing device cluster to multiple computing units. For example, computing unit 0 contains data 0, and computing unit 0 sends data 0 to computing units 1, 2, and 3.

[0082] Distributed operations have one data sender and multiple data receivers. They can divide data in a computing unit within a computing device cluster into different data slices, and then send these slices to multiple computing units. For example, computing unit 0 contains data 0. Computing unit 0 divides data 0 into four data slices, denoted as A0, B0, C0, and D0. Computing unit 0 sends B0 to computing unit 1, C0 to computing unit 2, and D0 to computing unit 3.

[0083] The collection operation has multiple data senders and one data receiver, which can send data from multiple computing units in a computing device cluster to one computing unit. For example, computing unit 0, computing unit 1, computing unit 2, and computing unit 3 respectively include data 0, data 1, data 2, and data 3, and computing unit 1, computing unit 2, and computing unit 3 send data 1, data 2, and data 3 to computing unit 0 respectively.

[0084] A reduction operation involves multiple data senders and one data receiver. Data from multiple computing units in a computing device cluster can be sent to a single computing unit, which then performs a reduction operation on the data to obtain the reduction result. Common reduction operations include summation, product, maximum value, and minimum value. This application does not limit these operations; for ease of explanation, this application uses summation as an example. For instance, computing units 0, 1, 2, and 3 respectively include data 0, data 1, data 2, and data 3. Computing units 1, 2, and 3 send data 1, data 2, and data 3 to computing unit 0, and computing unit 0 obtains data 0123 (i.e., the reduction result of data 0, data 1, data 2, and data 3).

[0085] Reduce-distribute operations involve multiple data senders and multiple data receivers. Data from multiple computing units in a computing cluster can be sent to a single computing unit. This single computing unit performs a reduction operation on the data to obtain a reduction result. This reduction result is then divided into multiple data slices, which are subsequently sent to multiple computing units. In essence, reduce-distribute operations can include both reduction and distribution operations.

[0086] A global collection operation involves multiple data senders and multiple data receivers. It can send data from multiple computing units in a computing device cluster to a single computing unit, and then send the data from that single computing unit to multiple computing units. It can be understood that a global collection operation can include both collection and broadcast operations.

[0087] A global reduction operation involves multiple data senders and multiple data receivers. Within a computing device cluster, data from multiple computing units can be sent to a single computing unit. This single computing unit performs a reduction operation on the data in its own unit to obtain the reduction result, which is then sent to multiple computing units. It can be understood that a global reduction operation can include reduction operations and broadcast operations, or it can include reduction distribution operations and global collection operations.

[0088] For example, Figure 2a is a schematic diagram of a global reduction operation, a reduction distribution operation, and a global collection operation provided in an embodiment of this application. As shown in Figure 2a, the computing device cluster includes four computing units, which are respectively represented as computing unit 0, computing unit 1, computing unit 2, and computing unit 3. Computing units 0, 1, 2, and 3 include data 0, data 1, data 2, and data 3, respectively. Computing unit 0 divides data 0 into four data fragments, represented as A0, B0, C0, and D0; computing unit 1 divides data 1 into four data fragments, represented as A1, B1, C1, and D1; computing unit 2 divides data 2 into four data fragments, represented as A2, B2, C2, and D2; and computing unit 3 divides data 3 into four data fragments, represented as A3, B3, C3, and D3.

[0089] When the four computing units perform the global reduction operation, computing unit 0, computing unit 1, computing unit 2 and computing unit 3 all obtain the data 0123 (that is, the reduction result of data 0, data 1, data 2 and data 3).

[0090] If the global reduction operation is divided into two operations, such as the reduction distribution operation and the global collection operation, when the four computing units perform the reduction distribution operation, computing unit 0 obtains A0123 (i.e., the reduction results of A0, A1, A2, and A3), computing unit 1 obtains B0123 (i.e., the reduction results of B0, B1, B2, and B3), computing unit 2 obtains C0123 (i.e., the reduction results of C0, C1, C2, and C3), and computing unit 3 obtains D0123 (i.e., the reduction results of D0, D1, D2, and D3). When the four computing units perform the global collection operation, computing unit 0, computing unit 1, computing unit 2 and computing unit 3 all obtain A0123, B0123, C0123 and D0123, that is, the reduction results of A0, A1, A2, A3, B0, B1, B2, B3, C0, C1, C2, C3, D0, D1, D2 and D3, which can be understood as the reduction results of data 0, data 1, data 2 and data 3.

[0091] Currently, global reduction operations can be implemented using communication algorithms such as the ring algorithm and the halving doubling (HD) algorithm. These will be introduced below.

[0092] 1) Global reduction operation based on ring algorithm. The ring algorithm refers to connecting multiple computing units into a ring structure, and the multiple computing units communicate with each other in a ring. That is, each computing unit only communicates with the computing units on its left and right. During the ring communication process, data is transmitted along the ring structure until the communication is completed and the data is transmitted to the destination computing unit.

[0093] For example, Figure 2b is a schematic diagram of performing a global reduction operation based on a ring algorithm according to an embodiment of this application.

[0094] To perform a global reduction operation based on a ring algorithm, the N computing units in the computing device cluster need to divide their respective data into N data fragments. For example, as shown in Figure 2b(1), the computing device cluster includes four computing units, which are represented as computing unit 0, computing unit 1, computing unit 2, and computing unit 3, respectively. These four computing units form a ring, where computing unit 1 is the next computing unit after computing unit 0, computing unit 0 is the next computing unit after computing unit 3, and so on. Computing units 0, 1, 2, and 3 respectively include data 0, data 1, data 2, and data 3. Calculation unit 0 divides data 0 into 4 data segments, denoted as A0, B0, C0, and D0 respectively; calculation unit 1 divides data 1 into 4 data segments, denoted as A1, B1, C1, and D1 respectively; calculation unit 2 divides data 2 into 4 data segments, denoted as A2, B2, C2, and D2 respectively; and calculation unit 3 divides data 3 into 4 data segments, denoted as A3, B3, C3, and D3 respectively.

[0095] In each step of the reduction distribution operation implemented based on the ring algorithm, each computing unit in the computing device cluster sends a fragment of data to the next computing unit, so that each computing unit can obtain an incomplete reduction result. For example, as shown in (2) of Figure 2b, in step 1, computing unit 0 sends A0 to computing unit 1, and computing unit 1 obtains A01 based on A0 and A1; computing unit 1 sends B1 to computing unit 2, and computing unit 2 obtains B12 based on B1 and B2; computing unit 2 sends C2 to computing unit 3, and computing unit 3 obtains C23 based on C2 and C3; computing unit 3 sends D3 to computing unit 0, and computing unit 0 obtains D03 based on D3 and D0. Similarly, the fragmented data transmitted between each computing unit in steps 2 and 3 can be seen in (2) of Figure 2b. After step 3 is completed, calculation unit 0 obtains B0123 (i.e., the reduction result of B0, B1, B2, and B3), calculation unit 1 obtains C0123 (i.e., the reduction result of C0, C1, C2, and C3), calculation unit 2 obtains D0123 (i.e., the reduction result of D0, D1, D2, and D3), and calculation unit 3 obtains A0123 (i.e., the reduction result of A0, A1, A2, and A3).

[0096] In each step of the global collection operation implemented based on the ring algorithm, each computing unit in the computing device cluster sends the incomplete reduction result it has obtained to the next computing unit, so that each computing unit can obtain the complete reduction result. For example, as shown in (3) of Figure 2b, combined with the incomplete reduction result obtained by each computing unit in (2) of Figure 2b, in step 1, computing unit 0 sends B0123 to computing unit 1, and computing unit 1 obtains B0123 accordingly; computing unit 1 sends C0123 to computing unit 2, and computing unit 2 obtains C0123 accordingly; computing unit 2 sends D0123 to computing unit 3, and computing unit 3 obtains D0123 accordingly; computing unit 3 sends A0123 to computing unit 0, and computing unit 0 obtains A0123 accordingly. Similarly, in steps 2 and 3, the fragmented data transmitted between each computing unit can be seen in (3) of Figure 2b. After step 3 is completed, each calculation unit obtains A0123, B0123, C0123 and D0123.

[0097] When N computing units in a computing device cluster perform a global reduction operation based on a ring algorithm, if L is the delay between any two computing units, S is the size of the data transmitted between any two computing units, and B is the bandwidth between any two computing units, i.e., the size of the data transmitted per unit time, then the number of communication steps is 2(N-1). The communication delay is obtained by accumulating the static delay and the transmission delay, which is 2(N-1)L+(2(N-1)S) / (NB).

[0098] It is evident that for global reduction operations based on ring algorithms, the number of communication steps and communication latency are directly proportional to the number of computing units in the computing device cluster; the more computing units in the cluster, the higher the number of communication steps and the higher the communication latency. Furthermore, since computing units in a computing device cluster need to segment their data into fragments when performing global reduction operations based on ring algorithms, this approach is more suitable for large data packet scenarios and less suitable for small data packet scenarios.

[0099] 2) Global reduction operation based on HD algorithm. HD algorithm refers to the communication between multiple computing units, where the communication distance and the amount of communication data are halved or doubled in each step of the HD algorithm implementation.

[0100] For example, Figure 2c is a schematic diagram of performing a global reduction operation based on the HD algorithm provided in an embodiment of this application.

[0101] To perform a global reduction operation based on the HD algorithm, the N computing units in the computing device cluster need to divide their respective data into N data fragments. For example, as shown in Figure 2c (1), the computing device cluster includes 4 computing units, which are represented as computing unit 0, computing unit 1, computing unit 2, and computing unit 3. Computing units 0, 1, 2, and 3 include data 0, data 1, data 2, and data 3, respectively. Computing unit 0 divides data 0 into 4 data fragments, represented as A0, B0, C0, and D0; computing unit 1 divides data 1 into 4 data fragments, represented as A1, B1, C1, and D1; computing unit 2 divides data 2 into 4 data fragments, represented as A2, B2, C2, and D2; and computing unit 3 divides data 3 into 4 data fragments, represented as A3, B3, C3, and D3.

[0102] In each step of the reduction distribution operation based on the HD algorithm, each computing unit in the computing device cluster sends half of the fragmented data to another nearby computing unit, and the communication distance and the amount of communication data are halved in each step, so that each computing unit can obtain an incomplete reduction result. For example, as shown in (1) of Figure 2c, in step 1, computing unit 0 sends C0 and D0 to computing unit 2, and computing unit 2 obtains C02 according to C0 and C2 and D02 according to D0 and D2; computing unit 1 sends C1 and D1 to computing unit 3, and computing unit 3 obtains C13 according to C1 and C3 and D13 according to D1 and D3; computing unit 2 sends A1 and B1 to computing unit 0, and computing unit 0 obtains A02 according to A0 and A2 and B02 according to B0 and B2; computing unit 3 sends A3 and B3 to computing unit 1, and computing unit 1 obtains A13 according to A1 and A3 and B13 according to B1 and B3. In step 2, calculation unit 0 sends B02 to calculation unit 1, and correspondingly, calculation unit 1 obtains B0123 (i.e., the reduction result of B0, B1, B2, and B3) based on B02 and B13; calculation unit 1 sends A13 to calculation unit 0, and correspondingly, calculation unit 0 obtains A0123 (i.e., the reduction result of A0, A1, A2, and A3) based on A02 and A13; calculation unit 2 sends D02 to calculation unit 3, and correspondingly, calculation unit 0 obtains D0123 (i.e., the reduction result of D0, D1, D2, and D3) based on D02 and D13; calculation unit 3 sends C13 to calculation unit 2, and correspondingly, calculation unit 2 obtains C0123 (i.e., the reduction result of C0, C1, C2, and C3) based on C02 and C13.

[0103] In each step of the global collection operation based on the HD algorithm, each computing unit in the computing device cluster sends the incomplete reduction result to another nearby computing unit. The communication distance and the amount of communication data double in each step, so that each computing unit can obtain the complete reduction result. As shown in (2) of Figure 2c, combined with the incomplete reduction result obtained by each computing unit in (1) of Figure 2c, in step 1, computing unit 0 sends A0123 to computing unit 1, and computing unit 1 obtains A0123 accordingly; computing unit 1 sends B0123 to computing unit 0, and computing unit 0 obtains B0123 accordingly; computing unit 2 sends D0123 to computing unit 3, and computing unit 3 obtains D0123 accordingly; computing unit 3 sends C0123 to computing unit 2, and computing unit 2 obtains C0123 accordingly. In step 2, calculation unit 0 sends A0123 and B0123 to calculation unit 2, and calculation unit 2 receives A0123 and B0123 accordingly; calculation unit 1 sends A0123 and B0123 to calculation unit 3, and calculation unit 3 receives A0123 and B0123 accordingly; calculation unit 2 sends C0123 and D0123 to calculation unit 0, and calculation unit 0 receives C0123 and D0123 accordingly; calculation unit 3 sends C0123 and D0123 to calculation unit 1, and calculation unit 1 receives C0123 and D0123 accordingly. After step 2 ends, each calculation unit obtains A0123, B0123, C0123, and D0123.

[0104] When N computing units in a computing device cluster perform a global reduction operation based on the HD algorithm, if L is the delay between any two computing units, S is the size of the data transmitted between any two computing units, and B is the bandwidth between any two computing units, i.e., the size of the data transmitted per unit time, then the number of communication steps is 2log2 N, and the communication delay is obtained by accumulating the static delay and the transmission delay, which is (2log2 N)L+((2log2 N)S) / (NB).

[0105] It is evident that for global reduction operations based on the HD algorithm, when the number of computing units in the computing device cluster is a power of two, it achieves a theoretically optimal number of communication steps at the logarithmic level with low communication latency. However, when the number of computing units in the cluster is not a power of two, additional communication steps and latency are introduced, leading to a situation where the collective communication performance of an N-1 scale computing device cluster is lower than that of an N scale computing device cluster. Furthermore, because the communication objects change in each step when the computing units in the computing device cluster perform global reduction operations based on the HD algorithm, cross-switch traffic conflicts are prone to occur in high-traffic scenarios. Additionally, since the computing units in the computing device cluster need to split their data into fragments when performing global reduction operations based on the HD algorithm, it is more suitable for large data packet scenarios and less suitable for small data packet scenarios.

[0106] Therefore, embodiments of this application provide a collection communication method to improve the efficiency of collection communication.

[0107] In the embodiments of this application, "when," "if," and "if" all refer to the device taking corresponding actions under certain objective circumstances, and are not time-limited, nor do they require the device to perform a judgment action, nor do they imply any other limitations. Unless otherwise specified, "if" and "if" can be substituted, and "when" and "in the case of" can be substituted. "When" and "if" / "if" can be substituted.

[0108] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0109] In this application embodiment, the number of nouns, unless otherwise specified, refers to "singular nouns or plural nouns," that is, "one or more." "At least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, or B exists alone, where A and B can be singular or plural. The character " / " can indicate that the related objects before and after are in an "or" relationship. For example, A / B means: A or B. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c means: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0110] In this application, the ordinal numbers such as "first" and "second" are used to distinguish multiple objects, and are not used to limit the size, content, order, timing, priority, or importance of the multiple objects. For a technical feature, the technical features within that technical feature are distinguished by "A", "B", "C", and "D", and there is no sequential or size order among the technical features described by "A", "B", "C", and "D".

[0111] The technical solutions provided by the embodiments of this application are described in detail below with reference to the accompanying drawings. For example, Figure 3 is a schematic flowchart of a collective communication method provided by an embodiment of this application. This collective communication method can be applied to the computing device cluster shown in Figure 1 or with a similar functional structure. This application embodiment does not impose specific limitations here. The specific flow of the collective communication method is described below. In the following description, the method is applied to the computing device cluster shown in Figure 1 as an example. For example, the computing unit mentioned below can be a computing unit in the system shown in Figure 1. To facilitate understanding of the embodiments of this application, unless otherwise specified, any two computing units in the computing device cluster are taken as computing units within different computing devices.

[0112] S301. During the collective communication operation performed by N computing units in a computing device cluster, the first computing unit receives first data from the second computing unit and / or sends second data to the third computing unit; wherein, N is a positive integer, and when the data in each of the N computing units is greater than a first threshold, the data in each computing unit is divided into N slices of data, the number of slices of data in the second computing unit included in the first data is inversely proportional to the first physical distance, and the number of slices of data in the first computing unit included in the second data is inversely proportional to the second physical distance, the first physical distance being the physical distance between the first computing unit and the second computing unit, and the second physical distance being the physical distance between the first computing unit and the third computing unit.

[0113] To facilitate understanding of the embodiments of this application, the following will first introduce the N computing units and the collective communication operations performed by the N computing units.

[0114] (1) N computing units

[0115] In this embodiment of the application, among the N computing units, the computing units with adjacent serial numbers or the computing units with the smallest and largest serial numbers are considered adjacent computing units. The physical distance between any computing unit and an adjacent computing unit is less than the physical distance between any computing unit and a non-adjacent computing unit.

[0116] For example, Figure 4a is a schematic diagram of adjacent computing units and non-adjacent computing units in a computing device cluster provided by an embodiment of this application. As shown in Figure 4a, the computing device cluster includes 6 computing units, which are respectively represented as computing unit 0, computing unit 1, computing unit 2, computing unit 3, computing unit 4 and computing unit 5. These 6 computing units form a ring.

[0117] On this ring, calculation unit 1 is the next calculation unit after calculation unit 0, and calculation unit 5 is the previous calculation unit of calculation unit 0. Therefore, calculation unit 1 and calculation unit 5 are adjacent calculation units of calculation unit 0, and calculation units 2, 3 and 4 are non-adjacent calculation units of calculation unit 0. The physical distance between calculation unit 0 and calculation unit 1 or calculation unit 5 is physical distance 1, which is less than the physical distance between calculation unit 0 and calculation unit 2, calculation unit 3 or calculation unit 4. Physical distance 1 is less than physical distance 2.

[0118] Similarly, on this ring, computation unit 0 and computation unit 2 are adjacent computation units of computation unit 1, and computation units 3, 4, and 5 are non-adjacent computation units of computation unit 1. Computation unit 1 and computation unit 3 are adjacent computation units of computation unit 2, and computation units 4, 5, and 0 are non-adjacent computation units of computation unit 2. Computation unit 2 and computation unit 4 are adjacent computation units of computation unit 3, and computation units 5, 0, and 1 are non-adjacent computation units of computation unit 3. Computation unit 3 and computation unit 5 are adjacent computation units of computation unit 4, and computation units 0, 1, and 2 are non-adjacent computation units of computation unit 4. Computation unit 4 and computation unit 0 are adjacent computation units of computation unit 5, and computation units 1, 2, and 3 are non-adjacent computation units of computation unit 5.

[0119] It can be understood that the physical distance between any two computing units is the same as the physical distance between the computing devices to which those two computing units belong. For example, the physical distance between computing unit 1 and computing unit 2 is the same as the physical distance between computing device 1 (to which computing device 1 belongs) and computing device 2 (to which computing unit 2 belongs). The greater the physical distance between any two computing units, the more switches are needed for communication between them. Conversely, the smaller the physical distance between any two computing units, the fewer switches are needed for communication between them.

[0120] (2) Collective communication operations performed by N computing units

[0121] In this embodiment, the collective communication operation performed by the N computing units may include one or more of the following: reduction-distribution operation, global collection operation, global reduction operation, distribution operation, broadcast operation, one-step reduction operation, and one-step broadcast operation. This embodiment does not limit the specific operations performed by the N computing units.

[0122] In this context, when the data in each of the N computing units exceeds a first threshold, the global reduction operation can include a reduction-distribution operation and a global collection operation, and the broadcast operation can include a distribution operation and a global collection operation. It can be understood that N computing units performing a reduction-distribution operation and a global collection operation constitutes N computing units performing a global reduction operation. N computing units performing a distribution operation and a global collection operation constitutes N computing units performing a broadcast operation.

[0123] When the data in each of the N computational units is less than or equal to a first threshold, the global reduction operation can include one reduction operation and one broadcast operation, and the broadcast operation can include one broadcast operation. It can be understood that N computational units performing one reduction operation and one broadcast operation constitutes performing a global reduction operation on the N computational units. N computational units performing one broadcast operation constitutes performing a broadcast operation on the N computational units.

[0124] The first threshold can be pre-configured or defined by a standard; this application does not limit this.

[0125] The following section describes the set communication operations performed by the N computing units based on whether the data of each of the N computing units is greater than the first threshold.

[0126] In case A, when the data in each of the N computing units is greater than the first threshold, the data in each of the N computing units can be divided into N slices.

[0127] In the embodiments of this application, during the process of N computing units performing reduction-distribution operation, global collection operation, global reduction operation, distribution operation or broadcast operation, the first computing unit may receive first data from the second computing unit and / or send second data to the third computing unit.

[0128] The first data may include some data from the second computing unit, and the second data may include some data from the first computing unit.

[0129] The number of slice data in the second computing unit included in the first data is inversely proportional to the first physical distance. The number of slice data in the first computing unit included in the second data is inversely proportional to the second physical distance. The first physical distance is the physical distance between the first computing unit and the second computing unit, and the second physical distance is the physical distance between the first computing unit and the third computing unit.

[0130] It is understandable that the greater the physical distance between the first computing unit and the second computing unit, the fewer slice data from the second computing unit are included in the first data sent by the second computing unit to the first computing unit; conversely, the smaller the physical distance between the first computing unit and the second computing unit, the more slice data from the second computing unit are included in the first data sent by the second computing unit to the first computing unit.

[0131] The greater the physical distance between the first computing unit and the third computing unit, the fewer slice data from the first computing unit are included in the second data sent from the first computing unit to the third computing unit; the smaller the physical distance between the first computing unit and the second computing unit, the more slice data from the first computing unit are included in the second data sent from the first computing unit to the third computing unit.

[0132] In other words, during the execution of reduction-distribution, global collection, global reduction, distribution, or broadcast operations by N computing units, the communication steps with the largest amount of data occur between the computing units with the smallest physical distance, while the communication steps with the smallest amount of data occur between the computing units with the largest physical distance. Since the smaller the physical distance between computing units, the fewer switches are needed for communication between them, and the larger the physical distance, the more switches are needed, this reduces cross-switch traffic conflicts during the execution of reduction-distribution, global collection, global reduction, distribution, or broadcast operations by N computing units.

[0133] The following describes the specific steps for performing reduction-distribution, global collection, global reduction, distribution, or broadcast operations on N computing units. Specifically, the steps for global reduction can include those for reduction-distribution and global collection operations, and the steps for broadcast can include those for distribution and global collection operations.

[0134] For example, a global reduction operation performed by N computing units may include 2P steps. Specifically, a reduction distribution operation performed by N computing units may include the first P steps, and a global collection operation performed by N computing units may include the last P steps. This is the rounding operator. For example, a global reduction operation performed by 6 computational units may include 6 steps, where the reduction distribution operation may include the first 3 steps, and the global collection operation may include the last 3 steps.

[0135] For example, a broadcast operation performed by N computing units can include 2P steps. The distributed operation performed by the N computing units can include the first P steps, and the global collection operation performed by the N computing units can include the last P steps. This is the rounding operator. For example, a broadcast operation performed by 6 computing units may include 6 steps, where the scatter operation may include the first 3 steps and the global collection operation may include the last 3 steps.

[0136] A1. Reduction and distribution operations performed by N computing units.

[0137] Specifically, during the K+1 step of the reduction and distribution operation performed by N computing units, the first computing unit can receive first data from the second computing unit and send second data to the third computing unit.

[0138] The first computation unit can be the (i+1)th computation unit among the N computation units. The second computation unit can be the (i+2)th computation unit among the N computation units. K The third computational unit can be the (i-2)th computational unit among the N computational units. k +N)%N+1 computational units. K is a natural number less than or equal to P-1, i is a natural number less than or equal to N-1, and % is the modulo operator.

[0139] The first data may include D slice data from the second computing unit. The second data may include D slice data from the first computing unit. This is the rounding operator.

[0140] In one possible implementation, the first data may include the (d1-2)th data in the second computing unit. K+1 The second data may include the (d2-2)th slice data in the first calculation unit. K+1 ·m)%N+1 slices of data. Where, d1=i, d2=(i-2 K +N)%N, where m is a natural number less than D. That is, the indices of the D slices of data in the second computational unit included in the first data can be determined according to (d1-2). K+1 The sequence numbers of the D slice data in the first calculation unit included in the second data can be determined according to (d2-2). K+1 ·m)%N+1 is determined.

[0141] For example, Figure 4b is a schematic diagram of performing a reduction distributed operation according to an embodiment of this application. Taking the computing device cluster shown in Figure 4a as an example, computing unit 0, computing unit 1, computing unit 2, computing unit 3, computing unit 4 and computing unit 5 respectively include data 0, data 1, data 2, data 3, data 4 and data 5. To perform the reduction and distribution operation, computing unit 0 splits data 0 into 6 data fragments, denoted as A0, B0, C0, D0, E0, and F0; computing unit 1 splits data 1 into 6 data fragments, denoted as A1, B1, C1, D1, E1, and F1; computing unit 2 splits data 2 into 6 data fragments, denoted as A2, B2, C2, D2, E2, and F2; computing unit 3 splits data 3 into 6 data fragments, denoted as A3, B3, C3, D3, E3, and F3; computing unit 4 splits data 4 into 6 data fragments, denoted as A4, B4, C4, D4, E4, and F4; and computing unit 5 splits data 5 into 6 data fragments, denoted as A5, B5, C5, D5, E5, and F5.

[0142] In the first step of the reduction and distribution operation, computing unit 0 sends B0, D0, and F0 to computing unit 5. Accordingly, computing unit 5 obtains B05 based on B0 and B5, D05 based on D0 and D5, and F05 based on F0 and F5. Computing unit 1 sends A1, C1, and E1 to computing unit 0. Accordingly, computing unit 1 obtains A01 based on A0 and A1, and C0 based on C0 and C1. 1. Calculation unit 2 sends B2, D2, and F2 to calculation unit 1. Accordingly, calculation unit 1 obtains B12 based on B1 and B2, D12 based on D1 and D2, and F12 based on F1 and F2. Calculation unit 3 sends A3, C3, and E3 to calculation unit 2. Accordingly, calculation unit 2 obtains A23 based on A2 and A3, C23 based on C2 and C3, and E23 based on E2 and E3. Calculation unit 4 sends B4, D4, and F4 to calculation unit 3. Accordingly, calculation unit 3 obtains B34 based on B3 and B4, D34 based on D3 and D4, and F34 based on F3 and F4. Calculation unit 5 sends A5, C5, and E5 to calculation unit 4. Accordingly, calculation unit 4 obtains A45 based on A4 and A5, C45 based on C4 and C5, and E45 based on E4 and E5.

[0143] In the second step of the reduction and distribution operation, computing unit 0 sends E01 to computing unit 4, and computing unit 4 obtains B0145 based on B01 and B45; computing unit 1 sends F12 to computing unit 5, and computing unit 5 obtains F0125 based on F12 and F05; computing unit 2 sends A23 to computing unit 0, and computing unit 0 obtains A0123 based on A23 and A01; computing unit 3 sends B34 to computing unit 1, and computing unit 1 obtains B1234 based on B34 and B12; computing unit 4 sends C45 to computing unit 2, and computing unit 1 obtains C2345 based on C45 and C23; computing unit 5 sends D05 to computing unit 3, and computing unit 3 obtains D0345 based on D05 and D34.

[0144] In the third step of the reduction distribution operation, computation unit 0 sends C01 to computation unit 2, and computation unit 2 accordingly obtains C012345 (i.e., the reduction result of C0, C1, C2, C3, C4, and C5) based on C01 and C2345; computation unit 1 sends D12 to computation unit 3, and computation unit 3 accordingly obtains D012345 (i.e., the reduction result of D0, D1, D2, D3, D4, and D5) based on D12 and D0345; computation unit 2 sends E23 to computation unit 4, and computation unit 4 accordingly obtains E012345 (i.e., E0, E1, E2, E3, E4, and E0145) based on E23 and E0145. The calculation unit 3 sends F34 to the calculation unit 5, and the calculation unit 5 obtains F012345 (i.e., the reduction results of F0, F1, F2, F3, F4, and F5) based on F34 and F0125; the calculation unit 4 sends A45 to the calculation unit 0, and the calculation unit 0 obtains A012345 (i.e., the reduction results of A0, A1, A2, A3, A4, and A5) based on A45 and A0123; the calculation unit 5 sends B05 to the calculation unit 1, and the calculation unit 1 obtains B012345 (i.e., the reduction results of B0, B1, B2, B3, B4, and B5) based on B05 and B1234.

[0145] It can be understood that the reduction and distribution operation performed by the six computing units can include three steps. In the first step, the communication interval is 1, and the communication data volume is 3 slices of data. For example, computing unit 0 receives 3 slices of data from computing unit 1 and sends 3 slices of data to computing unit 5. In the second step, the communication interval is 2, and the communication data volume is 1 slice of data. For example, computing unit 0 receives 1 slice of data from computing unit 2 and sends 1 slice of data to computing unit 4. In the third step, the communication interval is 4, and the communication data volume is 1 slice of data. For example, computing unit 0 receives 1 slice of data from computing unit 4 and sends 1 slice of data to computing unit 2. That is to say, in the process of performing the reduction and distribution operation, the communication step with the largest amount of communication data occurs between computing units with the smallest communication interval. The computing unit with the smallest communication interval is the computing unit with the smallest physical distance. Since the smaller the physical distance between computing units, the fewer switches are needed for communication between computing units, thus reducing cross-switch traffic conflicts during the reduction and distribution operation.

[0146] A2. Global collection operation performed by N computing units.

[0147] Specifically, during the (K+1)th step of the global collection operation performed by N computing units, the first computing unit can receive first data from the second computing unit and send second data to the third computing unit.

[0148] The first computational unit can be the (i+1)th computational unit among the N computational units. The second computational unit can be the (i-2)th computational unit among the N computational units. P-K-1 +N)%N+1 computational units. The third computational unit can be the (i+2)th computational unit among the N computational units. P-K-1 )%N+1 computational units. K is a natural number less than or equal to P-1, i is a natural number less than or equal to N-1, and % is the modulo operator.

[0149] The first data may include D slice data from the second computing unit. The second data may include D slice data from the first computing unit. This is the rounding operator.

[0150] In one possible implementation, the first data may include the (d1-2)th data in the second computing unit. P-K The second data may include the (d2-2)th slice data in the first calculation unit. P-K ·m)%N+1 slices of data. Where, d1=(i-2 P-K-1+N)%N, d2=i, m is a natural number less than D. That is, the indices of the D slices of data in the second computational unit included in the first data can be determined according to (d1-2). P-K The sequence numbers of the D slice data in the first calculation unit included in the second data can be determined according to (d2-2). P-K ·m)%N+1 is determined.

[0151] For example, Figure 4c is a schematic diagram of performing a global collection operation according to an embodiment of this application. Taking the computing device cluster shown in Figure 4a as an example, after the six computing units execute the third step in Figure 4b, computing unit 0 includes six data fragments, represented as A012345, B0, C01, D0, E01, and F0 respectively; computing unit 1 includes six data fragments, represented as A1, B012345, C1, D12, E1, and F12 respectively; and computing unit 2 includes six data fragments, represented as A23, B245, C01, D12, E1, and F12 respectively. The calculation unit 3 includes 6 data segments, represented as A3, B34, C3, D012345, E3, and F34 respectively; the calculation unit 4 includes 6 data segments, represented as A45, B4, C45, D4, E012345, and F4 respectively; the calculation unit 5 includes 6 data segments, represented as A5, B05, C5, D05, E5, and F012345 respectively.

[0152] In the first step of the global collection operation, computing unit 0 sends A012345 to computing unit 4, and computing unit 4 receives A012345 accordingly; computing unit 1 sends B012345 to computing unit 5, and computing unit 5 receives B012345 accordingly; computing unit 2 sends C012345 to computing unit 0, and computing unit 0 receives C012345 accordingly; computing unit 3 sends D012345 to computing unit 1, and computing unit 1 receives D012345 accordingly; computing unit 4 sends E012345 to computing unit 2, and computing unit 2 receives E012345 accordingly; computing unit 5 sends F012345 to computing unit 3, and computing unit 3 receives F012345 accordingly.

[0153] In the second step of the global collection operation, computing unit 0 sends A012345 to computing unit 2, and computing unit 2 receives A012345 accordingly; computing unit 1 sends B012345 to computing unit 3, and computing unit 3 receives B012345 accordingly; computing unit 2 sends C012345 to computing unit 4, and computing unit 4 receives C012345 accordingly; computing unit 3 sends D012345 to computing unit 5, and computing unit 5 receives D012345 accordingly; computing unit 4 sends E012345 to computing unit 0, and computing unit 0 receives E012345 accordingly; computing unit 5 sends F012345 to computing unit 1, and computing unit 1 receives F012345 accordingly.

[0154] In the third step of the global data collection operation, computing unit 0 sends A012345, C012345, and E012345 to computing unit 1, and computing unit 1 receives A012345, C012345, and E012345 accordingly; computing unit 1 sends B012345, D012345, and F012345 to computing unit 2, and computing unit 2 receives B012345, D012345, and F012345 accordingly; computing unit 2 sends A012345, C012345, and E012345 to computing unit 3, and computing unit 3 receives A012345, C012345, and E012345 accordingly. E012345; Calculation unit 3 sends B012345, D012345, and F012345 to calculation unit 4, and correspondingly, calculation unit 4 obtains B012345, D012345, and F012345; Calculation unit 4 sends A012345, C012345, and E012345 to calculation unit 5, and correspondingly, calculation unit 5 obtains A012345, C012345, and E012345; Calculation unit 5 sends B012345, D012345, and F012345 to calculation unit 0, and correspondingly, calculation unit 0 obtains B012345, D012345, and F012345.

[0155] After the third step of the global collection operation, computing units 0, 1, 2, 3, 4 and 5 all obtain A012345, B012345, C012345, D012345, E012345 and F012345, which are the reduction results of data 0, data 1, data 2, data 3, data 4 and data 5.

[0156] It can be understood that the global collection operation performed by the six computing units can include three steps. In the first step, the communication interval is 4, and the communication data volume is 1 slice of data. For example, computing unit 0 receives 1 slice of data from computing unit 2 and sends 1 slice of data to computing unit 4. In the second step, the communication interval is 2, and the communication data volume is 1 slice of data. For example, computing unit 0 receives 1 slice of data from computing unit 4 and sends 1 slice of data to computing unit 2. In the third step, the communication interval is 1, and the communication data volume is 3 slices of data. For example, computing unit 0 receives 3 slices of data from computing unit 5 and sends 3 slices of data to computing unit 1. In other words, during the execution of the global collection operation, the communication step with the largest amount of communication data occurs between computing units with the smallest communication interval. The computing unit with the smallest communication interval is also the computing unit with the smallest physical distance. Since the smaller the physical distance between computing units, the fewer switches are needed for communication between computing units, thus reducing cross-switch traffic conflicts during the execution of the global collection operation. Furthermore, whether the number of computing units in the computing device cluster is a power of 2 or not, it has the theoretically optimal number of communication steps at the log level, with less communication latency.

[0157] A3, Distributed operations performed by N computing units

[0158] Specifically, during the (K+1)th step of the distributed operation performed by the N computing units, the first computing unit can determine whether to receive the first data from the second computing unit and / or send data to the third computing unit based on the sequence distance between the first computing unit and the fourth computing unit (or the root computing unit, or the source computing unit). The fourth computing unit can be a computing unit that stores data in the N computing units before the distributed operation is performed by the N computing units.

[0159] If the sequence distance between the first computing unit and the fourth computing unit is greater than or equal to the second threshold and less than the third threshold, then the first computing unit can receive the first data from the second computing unit.

[0160] If the sequence number distance between the first computing unit and the fourth computing unit is less than the second threshold, the first computing unit can send the second data to the third computing unit.

[0161] If the sequence number distance between the first computing unit and the fourth computing unit is greater than or equal to the third threshold, the first computing unit may not receive the first data from the second computing unit and may not send the second data to the third computing unit.

[0162] The first computational unit can be the (i+1)th computational unit among the N computational units. The second computational unit can be the (i+2)th computational unit among the N computational units.K The third computational unit can be the (i-2)th computational unit among the N computational units. K +N)%N+1 computational units. K is a natural number less than or equal to P-1, i is a natural number less than or equal to N-1, and % is the modulo operator.

[0163] The first data may include D slice data from the second computing unit. The second data may include D slice data from the first computing unit. This is the rounding operator.

[0164] In one possible implementation, when the fourth computational unit is the (j+1)th computational unit out of N computational units, the index distance between the first computational unit and the fourth computational unit can satisfy the following relationship: W1=(j-i+N)%N

[0165] Where W1 is the sequence distance between the first and fourth calculation units, and j is a natural number less than or equal to N-1.

[0166] In one possible implementation, when N is not a power of 2 and K equals P-1, the second threshold and the third threshold can satisfy the following relationship:

[0167] When N is a power of 2 or K is not equal to P-1, the second and third thresholds can satisfy the following relationship:

[0168] Q1 is the second threshold, and Q2 is the third threshold.

[0169] In one possible implementation, the first data may include the (d1-2)th data in the second computing unit. K+1 The second data may include the (d2-2)th slice data in the first calculation unit. K+1 ·m)%N+1 slices of data. Where, d1=i, d2=(i-2 K +N)%N, where m is a natural number less than D. That is, the indices of the D slices of data in the second computational unit included in the first data can be determined according to (d1-2). K+1 The sequence numbers of the D slice data in the first calculation unit included in the second data can be determined according to (d2-2). K+1 ·m)%N+1 is determined.

[0170] For example, Figure 4d is a schematic diagram of performing a distributed operation according to an embodiment of this application. Taking the computing device cluster shown in Figure 4a as an example, computing unit 0 includes data 0. To perform the distributed operation, computing unit 0 divides data 0 into 6 data fragments, which are represented as A0, B0, C0, D0, E0 and F0, respectively.

[0171] In the first step of the distributed operation, computing unit 0 sends B0, D0 and F0 to computing unit 5, and computing unit 5 receives B0, D0 and F0 accordingly.

[0172] In the second step of the distributed operation, computing unit 0 sends E0 to computing unit 4, and computing unit 4 receives E0 accordingly; computing unit 5 sends D0 to computing unit 3, and computing unit 3 receives D0 accordingly.

[0173] In the third step of the distributed operation, computing unit 0 sends C0 to computing unit 2, and computing unit 2 receives C0 accordingly; computing unit 5 sends B0 to computing unit 1, and computing unit 1 receives B0 accordingly.

[0174] It can be understood that the distributed operation performed by 6 computing units can include 3 steps. In the first step, the communication interval is 1, and the communication data volume is 3 slices of data; in the second step, the communication interval is 2, and the communication data volume is 1 slice of data; in the third step, the communication interval is 4, and the communication data volume is 1 slice of data. That is to say, during the distributed operation, the communication step with the largest communication data volume occurs between computing units with the smallest communication interval. The computing unit with the smallest communication interval is also the computing unit with the smallest physical distance. Since the smaller the physical distance between computing units, the fewer switches are needed for communication between computing units, thus reducing cross-switch traffic conflicts during the distributed operation. Furthermore, whether the number of computing units in the computing device cluster is a power of 2 or not, it has a theoretically optimal number of communication steps at the logarithmic level, resulting in lower communication latency.

[0175] In one possible implementation, as shown in Figures 4b, 4c, and 4d above, for the D slice data in the second computing unit included in the first data, and the D slice data in the first computing unit included in the second data, all the slice data in these D slice data are non-contiguous in sequence, that is, all the slice data in these D slice data are non-contiguous in memory. Since communication of non-contiguous slice data in memory requires issuing communication tasks separately, for example, if slice data 1, slice data 2, and slice data 3 are non-contiguous in memory, then 3 communication tasks need to be issued. Therefore, in order to reduce the number of communication tasks issued during the execution of reduction-distribution operations, global collection operations, global reduction operations, distribution operations, or broadcast operations, the D slice data in the second computing unit included in the first data and the D slice data in the first computing unit included in the second data can be rearranged in sequence, so that all or part of the slice data in these D slice data are slice data with consecutive sequence numbers, thereby improving communication performance.

[0176] For example, Figure 4e is a schematic diagram of another reduction distribution operation provided by an embodiment of this application. Taking the computing device cluster shown in Figure 4a as an example, computing unit 0, computing unit 1, computing unit 2, computing unit 3, computing unit 4 and computing unit 5 respectively include data 0, data 1, data 2, data 3, data 4 and data 5. To perform the reduction and distribution operation, computing unit 0 splits data 0 into 6 data fragments, denoted as A0, B0, C0, D0, E0, and F0; computing unit 1 splits data 1 into 6 data fragments, denoted as A1, B1, C1, D1, E1, and F1; computing unit 2 splits data 2 into 6 data fragments, denoted as A2, B2, C2, D2, E2, and F2; computing unit 3 splits data 3 into 6 data fragments, denoted as A3, B3, C3, D3, E3, and F3; computing unit 4 splits data 4 into 6 data fragments, denoted as A4, B4, C4, D4, E4, and F4; and computing unit 5 splits data 5 into 6 data fragments, denoted as A5, B5, C5, D5, E5, and F5.

[0177] In the first step of the reduction and distribution operation, computing unit 0 sends D0, E0, and F0 to computing unit 5. Accordingly, computing unit 5 obtains D05 based on D0 and D5, E05 based on E0 and E5, and F05 based on F0 and F5. Computing unit 1 sends A1, B1, and C1 to computing unit 0. Accordingly, computing unit 1 obtains A01 based on A0 and A1, and B0 based on B0 and B1. 1. Calculation unit 2 sends D2, E2, and F2 to calculation unit 1. Accordingly, calculation unit 1 obtains D12 based on D1 and D2, E12 based on E1 and E2, and F12 based on F1 and F2. Calculation unit 3 sends A3, B3, and C3 to calculation unit 2. Accordingly, calculation unit 2 obtains A23 based on A2 and A3, B23 based on B2 and B3, and C23 based on C2 and C3. Calculation unit 4 sends D4, E4, and F4 to calculation unit 3. Accordingly, calculation unit 3 obtains D34 based on D3 and D4, E34 based on E3 and E4, and F34 based on F3 and F4. Calculation unit 5 sends A5, B5, and C5 to calculation unit 4. Accordingly, calculation unit 4 obtains A45 based on A4 and A5, B45 based on B4 and B5, and C45 based on C4 and C5.

[0178] In the second step of the reduction and distribution operation, computing unit 0 sends C01 to computing unit 4, and computing unit 4 obtains C0145 based on C01 and C45; computing unit 1 sends F12 to computing unit 5, and computing unit 5 obtains F0125 based on F12 and F05; computing unit 2 sends A23 to computing unit 0, and computing unit 0 obtains A0123 based on A23 and A01; computing unit 3 sends D34 to computing unit 1, and computing unit 1 obtains D1234 based on D34 and D12; computing unit 4 sends B45 to computing unit 2, and computing unit 1 obtains B2345 based on B45 and B23; computing unit 5 sends E05 to computing unit 3, and computing unit 3 obtains E0345 based on E05 and E34.

[0179] In the third step of the reduction distribution operation, computation unit 0 sends B01 to computation unit 2, and computation unit 2 accordingly obtains B012345 (i.e., the reduction result of B0, B1, B2, B3, B4, and B5) based on B01 and B2345; computation unit 1 sends E12 to computation unit 3, and computation unit 3 accordingly obtains E012345 (i.e., the reduction result of E0, E1, E2, E3, E4, and E5) based on E12 and E0345; computation unit 2 sends C23 to computation unit 4, and computation unit 4 accordingly obtains C012345 (i.e., the reduction result of C0, C1, C2, C3, C4, and C5) based on C23 and C0145. The calculation unit 3 sends F34 to the calculation unit 5, and the calculation unit 5 obtains F012345 (i.e., the reduction results of F0, F1, F2, F3, F4, and F5) based on F34 and F0125; the calculation unit 4 sends A45 to the calculation unit 0, and the calculation unit 0 obtains A012345 (i.e., the reduction results of A0, A1, A2, A3, A4, and A5) based on A45 and A0123; the calculation unit 5 sends D05 to the calculation unit 1, and the calculation unit 1 obtains B012345 (i.e., the reduction results of D0, D1, D2, D3, D4, and D5) based on D05 and D1234.

[0180] It can be understood that the reduction and distribution operation performed by the six computing units can include three steps. In the first step, the communication interval is 1, and the communication data volume is 3 slice data, which are slice data with consecutive sequence numbers. For example, computing unit 0 receives 3 slice data (i.e., A1, B1, and C1) from computing unit 1 and sends 3 slice data (i.e., D0, E0, and F0) to computing unit 5. In the second step, the communication interval is 2, and the communication data volume is 1 slice data. For example, computing unit 0 receives 1 slice data from computing unit 2 and sends 1 slice data to computing unit 4. In the third step, the communication interval is 4, and the communication data volume is 1 slice data. For example, computing unit 0 receives 1 slice data from computing unit 4 and sends 1 slice data to computing unit 2. In other words, during the reduction and distribution operation, the communication steps with the largest amount of communication data occur between computing units with the smallest communication interval. These computing units with the smallest communication interval are also the computing units with the smallest physical distance. Since the smaller the physical distance between computing units, the fewer switches they need to traverse, thus reducing cross-switch traffic conflicts during the reduction and distribution operation. Furthermore, the communication data is contiguous in memory, reducing the number of communication tasks issued during the reduction and distribution operation and improving communication performance.

[0181] For example, Figure 4f is a schematic diagram of another global collection operation provided by an embodiment of this application. Taking the computing device cluster shown in Figure 4a as an example, after the 6 computing units execute the third step in Figure 4e, computing unit 0 includes 6 data fragments, represented as A012345, B01, C01, D0, E01, and F0 respectively; computing unit 1 includes 6 data fragments, represented as A1, B1, C1, D012345, E12, and F12 respectively; computing unit 2 includes 6 data fragments, represented as A23, B01, C01, D012345, E12, and F12 respectively. 12345, C23, D2, E2, F2; Calculation unit 3 includes 6 data segments, represented as A3, B3, C3, D34, E012345, F34 respectively; Calculation unit 4 includes 6 data segments, represented as A45, B45, C012345, D4, E4, F4 respectively; Calculation unit 5 includes 6 data segments, represented as A5, B5, C5, D05, E05, F012345 respectively.

[0182] In the first step of the global collection operation, computing unit 0 sends A012345 to computing unit 4, and computing unit 4 receives A012345 accordingly; computing unit 1 sends D012345 to computing unit 5, and computing unit 5 receives D012345 accordingly; computing unit 2 sends B012345 to computing unit 0, and computing unit 0 receives B012345 accordingly; computing unit 3 sends E012345 to computing unit 1, and computing unit 1 receives E012345 accordingly; computing unit 4 sends C012345 to computing unit 2, and computing unit 2 receives C012345 accordingly; computing unit 5 sends F012345 to computing unit 3, and computing unit 3 receives F012345 accordingly.

[0183] In the second step of the global collection operation, computing unit 0 sends A012345 to computing unit 2, and computing unit 2 receives A012345 accordingly; computing unit 1 sends D012345 to computing unit 3, and computing unit 3 receives D012345 accordingly; computing unit 2 sends B012345 to computing unit 4, and computing unit 4 receives B012345 accordingly; computing unit 3 sends E012345 to computing unit 5, and computing unit 5 receives E012345 accordingly; computing unit 4 sends C012345 to computing unit 0, and computing unit 0 receives C012345 accordingly; computing unit 5 sends F012345 to computing unit 1, and computing unit 1 receives F012345 accordingly.

[0184] In the third step of the global collection operation, computing unit 0 sends A012345, B012345, and C012345 to computing unit 1, and computing unit 1 receives A012345, B012345, and C012345 accordingly; computing unit 1 sends D012345, E012345, and F012345 to computing unit 2, and computing unit 2 receives D012345, E012345, and F012345 accordingly; computing unit 2 sends A012345, B012345, and C012345 to computing unit 3, and computing unit 3 receives A012345, B012345, and C012345 accordingly. C012345; Calculation unit 3 sends D012345, E012345, and F012345 to calculation unit 4, and correspondingly, calculation unit 4 obtains D012345, E012345, and F012345; Calculation unit 4 sends A012345, B012345, and C012345 to calculation unit 5, and correspondingly, calculation unit 5 obtains A012345, B012345, and C012345; Calculation unit 5 sends D012345, E012345, and F012345 to calculation unit 0, and correspondingly, calculation unit 0 obtains D012345, E012345, and F012345.

[0185] After the third step of the global collection operation, computing units 0, 1, 2, 3, 4 and 5 all obtain A012345, B012345, C012345, D012345, E012345 and F012345, which are the reduction results of data 0, data 1, data 2, data 3, data 4 and data 5.

[0186] It can be understood that the global collection operation performed by the six computing units can include three steps. In the first step, the communication interval is 4, and the communication data volume is 1 slice of data. For example, computing unit 0 receives 1 slice of data from computing unit 2 and sends 1 slice of data to computing unit 4. In the second step, the communication interval is 2, and the communication data volume is 1 slice of data. For example, computing unit 0 receives 1 slice of data from computing unit 4 and sends 1 slice of data to computing unit 2. In the third step, the communication interval is 1, and the communication data volume is 3 slices of data. For example, computing unit 0 receives 3 slices of data (i.e., D012345, E012345, and F012345) from computing unit 5 and sends 3 slices of data (i.e., A012345, B012345, and C012345) to computing unit 1. In other words, during the global data collection operation, the communication steps with the largest amount of data occur between computing units with the smallest communication interval. These computing units with the smallest communication interval are also the ones with the smallest physical distance. Since the smaller the physical distance between computing units, the fewer switches they need to traverse, thus reducing cross-switch traffic conflicts during the global data collection operation. Furthermore, the communication data is contiguous in memory, reducing the number of communication tasks issued during the global data collection operation and improving communication performance.

[0187] Case B: When the data in each of the N computing units is less than or equal to the first threshold, the data in each of the N computing units can be left undivided into N slices.

[0188] During the execution of a one-step reduction operation, a one-step broadcast operation, a global reduction operation, and a broadcast operation by N computing units, the first computing unit may receive first data from the second computing unit and / or send second data to the third computing unit.

[0189] The first data may include all the data in the second computing unit, and the second data may include all the data in the first computing unit.

[0190] It is understandable that when the amount of communication data between N computing units is small, the communication data can be left unsegmented, allowing the communication data to be continuous in memory. This reduces the number of communication tasks issued during the execution of one-step reduction, one-step broadcast, global reduction, and broadcast operations, thereby improving communication performance.

[0191] The specific steps for performing a one-step reduction operation, a one-step broadcast operation, a global reduction operation, and a broadcast operation on N computational units are described below. The specific steps for a global reduction operation can include the specific steps for a one-step reduction operation and a one-step broadcast operation, and the specific steps for a broadcast operation can include the specific steps for a one-step broadcast operation.

[0192] For example, a global reduction operation performed by N computing units can include 2P steps. Specifically, a one-step reduction operation performed by N computing units can include the first P steps, and a one-step broadcast operation performed by N computing units can include the last P steps. This is the rounding operator. For example, a global reduction operation performed by 6 computation units can include 6 steps, where a single reduction operation can include the first 3 steps, and a single broadcast operation can include the last 3 steps.

[0193] B1. A one-step reduction operation performed by N computing units.

[0194] Specifically, during the K+1th step of the reduction operation performed by N computing units, the first computing unit can determine whether to receive the first data from the second computing unit and / or send data to the third computing unit based on the sequence distance between the first computing unit and the first computing unit (or the root computing unit, or the source computing unit) among the N computing units.

[0195] If the sequence distance between the first computing unit and the first computing unit among the N computing units satisfies relation A, then the first computing unit may not receive the first data from the second computing unit, nor send the second data to the third computing unit.

[0196] If the sequence distance between the first computing unit and the first computing unit among the N computing units satisfies the first relationship, then the first computing unit can receive the first data from the second computing unit.

[0197] If the sequence distance between the first computing unit and the first computing unit among the N computing units satisfies the second relationship, then the first computing unit can send the second data to the third computing unit.

[0198] The first computational unit can be the (i+1)th computational unit among the N computational units. The second computational unit can be the (i+2)th computational unit among the N computational units. K The third computational unit can be the (i-2)th computational unit among the N computational units. K +N)%N+1 computational units. K is a natural number less than or equal to P-1, i is a natural number less than or equal to N-1, and % is the modulo operator.

[0199] The first data may include all the data in the second computing unit, and the second data may include all the data in the first computing unit.

[0200] In one possible implementation, relation A can be specifically defined as:

[0201] The first relation can be specifically defined as:

[0202] The second relationship can be specifically defined as follows:

[0203] Where, round(*) is the rounding operator, j is a natural number less than or equal to N-1, and W2 is the ordinal distance between the first calculation unit and the first calculation unit among N calculation units.

[0204] For example, Figure 4g is a schematic diagram of performing a one-step reduction operation according to an embodiment of this application. Taking the computing device cluster shown in Figure 4a as an example, computing unit 0, computing unit 1, computing unit 2, computing unit 3, computing unit 4 and computing unit 5 respectively include data 0, data 1, data 2, data 3, data 4 and data 5.

[0205] In the first step of the reduction operation, calculation unit 1 sends data 1 to calculation unit 0, and correspondingly, calculation unit 0 obtains data 01 (i.e., the reduction result of data 0 and data 1) based on data 0 and data 1; calculation unit 3 sends data 3 to calculation unit 2, and correspondingly, calculation unit 2 obtains data 23 (i.e., the reduction result of data 2 and data 3) based on data 2 and data 3; calculation unit 5 sends data 5 to calculation unit 4, and correspondingly, calculation unit 4 obtains data 45 (i.e., the reduction result of data 4 and data 5) based on data 4 and data 5.

[0206] In the second step of the reduction operation, the calculation unit 2 sends data 23 to the calculation unit 0. Accordingly, the calculation unit 0 obtains data 0123 (i.e., the reduction result of data 0, data 1, data 2 and data 3) based on data 01 and data 23.

[0207] In the third step of the reduction operation, the calculation unit 4 sends data 45 to the calculation unit 0. Accordingly, the calculation unit 0 obtains data 012345 (i.e., the reduction result of data 0, data 1, data 2, data 3, data 4 and data 5) based on data 0123 and data 45.

[0208] B2. A one-step broadcast operation performed by N computing units.

[0209] Specifically, during the (K+1)th step of a broadcast operation performed by N computing units, the first computing unit can determine whether to receive the first data from the second computing unit and / or send data to the third computing unit based on the sequence distance between the first computing unit and the fifth computing unit (or the root computing unit, or the source computing unit). The fifth computing unit can be a computing unit that stores the data to be broadcast in the N computing units before the broadcast operation is performed by the N computing units.

[0210] If the sequence distance between the first computing unit and the first computing unit among the N computing units satisfies relation B, then the first computing unit may not receive the first data from the second computing unit, nor send the second data to the third computing unit.

[0211] If the sequence distance between the first computing unit and the fifth computing unit satisfies the third relationship, then the first computing unit can receive the first data from the second computing unit.

[0212] If the sequence distance between the first computing unit and the fifth computing unit satisfies the fourth relationship, then the first computing unit can send the second data to the third computing unit.

[0213] The first computational unit can be the (i+1)th computational unit among the N computational units. The second computational unit can be the (i-2)th computational unit among the N computational units. P-K-1 +N)%N+1 computational units. The third computational unit can be the (i+2)th computational unit among the N computational units. P-K-1 )%N+1 computational units. K is a natural number less than or equal to P-1, i is a natural number less than or equal to N-1, and % is the modulo operator.

[0214] The first data may include all the data in the second computing unit, and the second data may include all the data in the first computing unit.

[0215] In one possible implementation, when the fifth computational unit is the (j+1)th computational unit out of N computational units, relation B can be specifically defined as follows:

[0216] The third relationship can be specifically defined as follows:

[0217] The fourth relationship can be specifically defined as:

[0218] Where j is a natural number less than or equal to N-1, and W2 is the ordinal distance between the first and fifth calculation units.

[0219] For example, Figure 4h is a schematic diagram of performing a one-step broadcast operation according to an embodiment of this application. Taking the computing device cluster shown in Figure 4a as an example, after the six computing units execute the third step in Figure 4g, computing unit 0 includes data 012345, computing unit 1 includes data 1, computing unit 2 includes data 23, computing unit 3 includes data 3, computing unit 1 includes data 45, and computing unit 5 includes data 5.

[0220] Taking computing unit 0 as the root node as an example, in the first step of a broadcast operation, computing unit 0 sends data 012345 to computing unit 4, and computing unit 4 receives data 012345 accordingly.

[0221] In the second step of the broadcast operation, computing unit 0 sends data 012345 to computing unit 2, and computing unit 2 receives data 012345 accordingly.

[0222] In the third step of the broadcast operation, calculation unit 0 sends data 012345 to calculation unit 1, and calculation unit 1 receives data 012345 accordingly; calculation unit 2 sends data 012345 to calculation unit 3, and calculation unit 3 receives data 012345 accordingly; calculation unit 4 sends data 012345 to calculation unit 5, and calculation unit 5 receives data 012345 accordingly.

[0223] It is understood that the above embodiments of this application can be implemented individually or in combination with each other, and the embodiments of this application are not limited.

[0224] Based on the above embodiments, this application also provides a computing device, as shown in FIG5, which is a schematic diagram of the structure of a computing device provided in this application embodiment. The computing device 500 includes: a bus 502, a processor 504, a memory 506, and a communication interface 508. The processor 504, the memory 506, and the communication interface 508 communicate with each other through the bus 502. It should be understood that this application does not limit the number of processors and memories in the computing device 500.

[0225] Bus 502 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 5, but this does not imply that there is only one bus or one type of bus. Bus 502 can include pathways for transmitting information between various components of computing device 500 (e.g., memory 506, processor 504, communication interface 508).

[0226] Processor 504 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0227] Memory 506 may include volatile memory, such as random access memory (RAM). Memory 506 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0228] The memory 506 stores executable program code, and the processor 504 executes this executable program code to implement the functions of the communication module and the processing module, thereby realizing the collective communication method in the above method embodiments. That is, the memory 506 stores instructions for executing the collective communication method in the above method embodiments.

[0229] The communication interface 508 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 500 and other devices or communication networks.

[0230] Based on the above embodiments, this application also provides a computing device cluster. The computing device cluster includes at least one computing device. This computing device may be an electronic device.

[0231] As shown in Figure 6, the computing device cluster includes at least one computing device 500. The memory 506 of one or more computing devices 500 in the computing device cluster may store the same instructions for executing the aggregate communication method in the above method embodiments.

[0232] In some possible implementations, the memory 506 of one or more computing devices 500 in the computing device cluster may also store partial instructions for executing the collective communication method in the above method embodiments. In other words, a combination of one or more computing devices 500 can jointly execute instructions for executing the collective communication method in the above method embodiments.

[0233] It is understandable that the memories 506 in different computing devices 500 within a computing device cluster can store different instructions, each used to execute a portion of the electronic device's functions. That is, the instructions stored in the memories 506 of different computing devices 500 can implement the functions of one or more modules in the communication module and processing module.

[0234] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 7 illustrates one possible implementation. As shown in Figure 7, two computing devices 500A and 500B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 506 in computing device 500A stores instructions for performing the functions of the communication module. Simultaneously, the memory 506 in computing device 500B stores instructions for performing the functions of the processing module.

[0235] It should be understood that the functions of computing device 500A shown in Figure 7 can also be performed by multiple computing devices 500. Similarly, the functions of computing device 500B can also be performed by multiple computing devices 500.

[0236] Based on the above embodiments, this application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the collection communication method in the above method embodiments.

[0237] Based on the above embodiments, this application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored in any usable medium. When the computer program product runs on at least one computing device, it causes the at least one computing device to execute the collective communication method in the above method embodiments.

[0238] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0239] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0240] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0241] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0242] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A collective communication method, characterized in that, Applied to a computing device cluster, the computing device cluster comprising N computing units, where N is a positive integer; the method includes: During the collective communication operation performed by the N computing units, the first computing unit receives first data from the second computing unit and / or sends second data to the third computing unit; Wherein, if the data in each of the N computing units is greater than a first threshold, the data in each computing unit is divided into N slices of data. The number of slices of data in the second computing unit included in the first data is inversely proportional to a first physical distance, and the number of slices of data in the first computing unit included in the second data is inversely proportional to a second physical distance. The first physical distance is the physical distance between the first computing unit and the second computing unit, and the second physical distance is the physical distance between the first computing unit and the third computing unit.

2. The method according to claim 1, characterized in that, Among the N computing units, those with adjacent serial numbers or those with the smallest and largest serial numbers are considered adjacent computing units. The physical distance between any one of the N computing units and its adjacent computing units is less than the physical distance between any one of the computing units and its non-adjacent computing units.

3. The method according to claim 2, characterized in that, The data in each of the N computing units is greater than the first threshold. The aggregate communication operation includes a global collection operation, which comprises P steps. As a rounding operator, the method further includes: During the (K+1)th step of the global collection operation performed by the N computing units, the first computing unit receives the first data from the second computing unit and sends the second data to the third computing unit; Wherein, the first computing unit is the (i+1)th computing unit among the N computing units, and the second computing unit is the (i-2)th computing unit among the N computing units. P-K-1 +N)%N+1 calculation units, the third calculation unit is the (i+2)th calculation unit among the N calculation units. P-K-1 )%N+1 computational units, where K is a natural number less than or equal to P-1, i is a natural number less than or equal to N-1, and % is the modulo operator; The first data includes D slice data from the second computing unit, and the second data includes D slice data from the first computing unit. round(*) is the rounding operator.

4. The method according to claim 2, characterized in that, The data in each of the N computing units is greater than the first threshold. The aggregate communication operation includes a reduction-distribution operation, which comprises P steps. As a rounding operator, the method further includes: During the K+1 step of the reduction and distribution operation performed by the N computing units, the first computing unit receives the first data from the second computing unit and sends the second data to the third computing unit; Wherein, the first calculation unit is the (i+1)th calculation unit among the N calculation units, and the second calculation unit is the (i+2)th calculation unit among the N calculation units. K The third calculation unit is the (i-2)th calculation unit among the N calculation units. k +N)%N+1 calculation units, K is a natural number less than or equal to P-1, i is a natural number less than or equal to N-1, and % is the modulo operator; The first data includes D slice data from the second computing unit, and the second data includes D slice data from the first computing unit. round(*) is the rounding operator.

5. The method according to claim 2, characterized in that, The data in each of the N computing units is greater than the first threshold. The aggregate communication operation includes a distribution operation, which comprises P steps. As a rounding operator, the method further includes: During the (K+1)th step of the distributed operation performed by the N computing units, if the sequence distance between the first computing unit and the fourth computing unit is greater than or equal to a second threshold and less than a third threshold, then the first computing unit receives the first data from the second computing unit; or, During the (K+1)th step of the distributed operation performed by the N computing units, if the sequence number distance between the first computing unit and the fourth computing unit is less than the second threshold, then the first computing unit sends the second data to the third computing unit. Wherein, the first calculation unit is the (i+1)th calculation unit among the N calculation units, and the second calculation unit is the (i+2)th calculation unit among the N calculation units. K The third calculation unit is the (i-2)th calculation unit among the N calculation units. K +N)%N+1 computing units, wherein the fourth computing unit is a computing unit that stores data in the N computing units before the N computing units perform the distributed operation, K is a natural number less than or equal to P-1, i is a natural number less than or equal to N-1, and % is the modulo operator; The first data includes D slice data from the second computing unit, and the second data includes D slice data from the first computing unit. round(*) is the rounding operator.

6. The method according to claim 5, characterized in that, When the fourth calculation unit is the (j+1)th calculation unit among the N calculation units, the index distance between the first calculation unit and the fourth calculation unit satisfies the following relationship: W1=(j-i+N)%N Where W1 is the sequence distance between the first calculation unit and the fourth calculation unit, and j is a natural number less than or equal to N-1; When N is not a power of 2 and K equals P-1, the second threshold and the third threshold satisfy the following relationship: When N is a power of 2 or K is not equal to P-1, the second threshold and the third threshold satisfy the following relationship: Wherein, Q1 is the second threshold and Q2 is the third threshold.

7. The method according to claim 3, characterized in that, The first data includes the (d1-2)th data in the second calculation unit. P-K ·m)%N+1 slice data, the second data includes the (d2-2)th slice data in the first computing unit. P-K ·m)%N+1 slices of data; Where, d1=(i-2 P-K-1 +N)%N, d2=i, m is a natural number less than D.

8. The method according to any one of claims 4-5, characterized in that, The first data includes the (d1-2)th data in the second calculation unit. K+1 ·m)%N+1 slice data, the second data includes the (d2-2)th slice data in the first computing unit. K+1 ·m)%N+1 slices of data; Where d1 = i, d2 = (i - 2) K +N)%N, where m is a natural number less than D.

9. The method according to any one of claims 3-5, characterized in that, All or part of the D slice data are slice data with consecutive serial numbers.

10. The method according to claim 2, characterized in that, The data in each of the N computing units is less than or equal to the first threshold. The set communication operation includes a one-step reduction operation, which comprises P steps. As a rounding operator, the method further includes: During the (K+1)th step of the reduction operation performed by the N computing units, if the sequence distance between the first computing unit and the first computing unit among the N computing units satisfies a first relationship, then the first computing unit receives the first data from the second computing unit; or, During the (K+1)th step of the reduction operation performed by the N computing units, if the sequence distance between the first computing unit and the first computing unit among the N computing units satisfies the second relationship, then the first computing unit sends the second data to the third computing unit. Wherein, the first calculation unit is the (i+1)th calculation unit among the N calculation units, and the second calculation unit is the (i+2)th calculation unit among the N calculation units. K The third calculation unit is the (i-2)th calculation unit among the N calculation units. K +N)%N+1 calculation units, K is a natural number less than or equal to P-1, i is a natural number less than or equal to N-1, and % is the modulo operator; The first data includes all the data in the second computing unit, and the second data includes all the data in the first computing unit.

11. The method according to claim 10, characterized in that, The first relationship is specifically: The second relationship is specifically as follows: Wherein, W2 is the sequence distance between the first computing unit and the first computing unit among the N computing units.

12. The method according to claim 2, characterized in that, The data in each of the N computing units is less than or equal to the first threshold. The aggregate communication operation includes a one-step broadcast operation, which comprises P steps. As a rounding operator, the method further includes: During the (K+1)th step of the broadcast operation performed by the N computing units, if the sequence distance between the first computing unit and the fifth computing unit satisfies the third relationship, then the first computing unit receives the first data from the second computing unit; or, During the (K+1)th step of the broadcast operation performed by the N computing units, if the sequence distance between the first computing unit and the fifth computing unit satisfies the fourth relationship, then the first computing unit sends the second data to the third computing unit. Wherein, the first computing unit is the (i+1)th computing unit among the N computing units, and the second computing unit is the (i-2)th computing unit among the N computing units. P-K-1 +N)%N+1 calculation units, the third calculation unit is the (i+2)th calculation unit among the N calculation units. P-K-1 )%N+1 calculation units, wherein the fifth calculation unit is a calculation unit that stores the data to be broadcast in the N calculation units before the N calculation units perform the one-step broadcast operation, K is a natural number less than or equal to P-1, i is a natural number less than or equal to N-1, and % is the modulo operator; The first data includes all the data in the second computing unit, and the second data includes all the data in the first computing unit.

13. The method according to claim 12, characterized in that, When the fifth calculation unit is the (j+1)th calculation unit among the N calculation units, the third relation is specifically as follows: The fourth relationship is as follows: Where j is a natural number less than or equal to N-1, and W2 is the ordinal distance between the first calculation unit and the fifth calculation unit.

14. The method according to any one of claims 1-13, characterized in that, The collective communication operation includes one or more of the following: reduction-distribution operation, global collection operation, global reduction operation, distribution operation, broadcast operation, one-step reduction operation, and one-step broadcast operation; Wherein, if the data in each of the N computing units is greater than the first threshold, the global reduction operation includes the reduction-distribution operation and the global collection operation, and the broadcast operation includes the distribution operation and the global collection operation; or... When the data in each of the N computing units is less than or equal to the first threshold, the global reduction operation includes the one-step reduction operation and the one-step broadcast operation, and the broadcast operation includes the one-step broadcast operation.

15. A computing device cluster, characterized in that, The computing device cluster includes at least one computing device, and each computing device includes a processor and a memory; the memory is used to store computer program instructions. The processor invokes computer program instructions stored in the memory to execute the method as described in any one of claims 1-14.

16. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computing device, cause the computing device to perform the method as described in any one of claims 1-14.

17. A computer program product containing instructions, characterized in that, When the instructions are executed by the computing device, the computing device performs the method as described in any one of claims 1-14.

Images