Collective communication method, collective communication system and related apparatus
By adopting a re-execution mechanism at the level of the set communication operator, the problem of training task interruption caused by AI cluster communication failure is solved, which improves cluster stability and recovery efficiency and reduces resource and time consumption.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- HUAWEI TECH CO LTD
- Filing Date
- 2025-11-05
- Publication Date
- 2026-05-21
AI Technical Summary
As AI clusters grow in size, hardware failure rates increase, leading to frequent communication failures. Existing technologies rely on checkpoint mechanisms for recovery, which are costly and have a high probability of interrupting training tasks.
In the event of a communication failure, a re-execution mechanism at the level of communication operators is used to allow the management node to negotiate and calculate the node status. Once it is confirmed that the original data has not been changed, the link is switched to re-execute the communication task, thereby reducing recovery time.
It improves the stability of the AI cluster, reduces the computing resources and time required for business recovery, and avoids a long checkpoint recovery process.
Smart Images
Figure CN2025132666_21052026_PF_FP_ABST
Abstract
Description
A collective communication method, a collective communication system, and related devices.
[0001] This application claims priority to Chinese Patent Application No. 202411617058.0, filed on November 12, 2024, entitled "A Collection Communication Method, Collection Communication System and Related Device", the entire contents of which are incorporated herein by reference. Technical Field
[0002] This application relates to the field of communications, and in particular to a collective communication method, a collective communication system, and related devices. Background Technology
[0003] To shorten the training cycle of AI training tasks and accommodate larger-scale models, large-scale AI clusters are needed for training. As the size of the AI cluster increases, the number of components involved (switching equipment, optical modules, etc.) increases significantly. The failure rates of individual components accumulate, leading to a continuous decrease in the overall stability of the cluster as the scale of use grows. Therefore, the probability of training tasks being interrupted due to communication failures caused by hardware issues is increasing.
[0004] AI training progresses through multiple rounds of iterative updates to intermediate variables. In the event of a communication failure, it primarily relies on a checkpoint mechanism to revert to a previously saved state and restart training. However, this recovery method requires pausing training, migrating data to computing nodes, and restarting, which is costly. Summary of the Invention
[0005] This application provides a collection communication method, collection communication system, and related apparatus to reduce service recovery time in the event of a communication failure.
[0006] In a first aspect, embodiments of this application provide a collective communication method applied to a collective communication system, the collective communication system including a management node and N computing nodes, the N computing nodes being used to process a first communication task in parallel, where N is an integer greater than 1, the method including: the management node receiving first information, the first information indicating that at least one first link has failed, each first link being a communication link between the N computing nodes for processing the first communication task; the management node receiving N second information from the N computing nodes, each second information indicating whether the original data of the first communication task stored locally on one computing node has been modified; if the original data of the N computing nodes has not been modified, the management node sending third information to the N computing nodes, the third information instructing the N computing nodes to reprocess the first communication task based on the original data.
[0007] The aggregated communication method in this application focuses on the recovery mechanism when encountering failures during communication. The compute nodes are GPU cards or NPU cards, and the management node is a host CPU. When a network failure occurs, data transmission between compute nodes fails, the first communication task is interrupted, and this failure information is returned to the compute nodes, indicating a failure in the first link. The compute nodes then report the communication failure information to the management node, which negotiates the status of each compute node, determining whether the original data stored locally on each compute node has been altered. Each compute node reports the detection results to the management node after detection. The management node then determines whether to re-execute the aggregated communication operator based on the detection results.
[0008] In one possible implementation, each GPU card or NPU card corresponds to a host CPU. After receiving a fault message, the host CPU needs to negotiate with other host CPUs to determine whether to re-execute the first communication task at the level of the aggregate communication operator.
[0009] By employing the above method, a re-execution mechanism for the aggregate communication operator is added, enhancing the stability of the AI cluster. In the event of a communication failure, the aggregate communication system only needs to roll back one aggregate communication operator, resulting in a loss of approximately one second, thereby reducing the computing resources and time required for service recovery.
[0010] In one possible implementation, the first link is a communication link between a first computing node and a second computing node, wherein the first computing node and the second computing node are two computing nodes among the N computing nodes, and the third information is also used to indicate a second link, which is a communication link between the first computing node and the second computing node for reprocessing the first communication task.
[0011] In this application, at least one backup second link exists between the first computing node and the second computing node. The first computing node uses the backup link through a bypass mechanism and re-executes the first communication task. The third information indicates the specific second link used by the first and second computing nodes to reprocess the first communication task.
[0012] In one possible implementation, the third information includes the identifier of the second link.
[0013] In one possible implementation, the first information is specifically used to indicate that an optical module failure has occurred in the first link.
[0014] In most cases, optical module failures occur as intermittent interruptions. Some optical module components have self-diagnosis and recovery functions, and the faulty link has a certain probability of self-recovery within a few seconds.
[0015] In one possible implementation, the second link indicated by the third information is the first link.
[0016] In one possible implementation, the first computing node includes a first network port and a second network port, the first link is a communication link based on the first network port, and the second link is a communication link based on the second network port.
[0017] Using the above method, for devices with multiple network ports, combined with aggregated communication re-execution technology, fault points can be bypassed, further improving cluster stability.
[0018] In one possible implementation, before the management node receives N pieces of second information from the N computing nodes, the method further includes:
[0019] The management node sends a fourth message to each of the N computing nodes, and each of the fourth messages is used to request the second message.
[0020] In one possible implementation, if any of the original data in the N computing nodes is changed, the management node decides to use the checkpoint mechanism to roll back to the previously saved state, migrate the original data stored on the network side to each computing node, and restart training.
[0021] Secondly, embodiments of this application provide a collective communication method applied to a collective communication system, which includes a management node and N computing nodes. The N computing nodes are used to process a first communication task in parallel, where N is an integer greater than 1. The method includes:
[0022] The first computing node sends a first message to the management node. The first message indicates that the first link has failed. The first link is a communication link between the first computing node and the second computing node used to process the first communication task. The first computing node and the second computing node are two computing nodes out of N computing nodes.
[0023] The first computing node sends a second message to the management node, which indicates whether the original data of the first communication task stored locally on the first computing node has been changed.
[0024] The first computing node receives third information from the management node. The third information is used to instruct the first computing node to reprocess the first communication task based on the original data. The third information is determined by the management node based on the fact that the original data of N computing nodes has not been changed.
[0025] The first computing node reprocesses the first communication task based on the third information and the original data.
[0026] In this application, the compute nodes are GPU cards or NPU cards, and the management node is a host CPU. When a network failure occurs, data transmission between compute nodes fails, the first communication task is interrupted, and this failure information is returned to the compute nodes, indicating a failure in the first link. The compute nodes then report the communication failure information to the management node, which negotiates the status of each compute node, i.e., determines whether the original data stored locally in each compute node has been modified. Each compute node reports the detection results to the management node after detection. The management node determines whether to re-execute the aggregated communication operator based on the detection results.
[0027] In one possible implementation, each GPU card or NPU card corresponds to a host CPU. After receiving a fault message, the host CPU needs to negotiate with other host CPUs to determine whether to re-execute the first communication task at the level of the aggregate communication operator.
[0028] By employing the above method, a re-execution mechanism for the aggregate communication operator is added, enhancing the stability of the AI cluster. In the event of a communication failure, the aggregate communication system only needs to roll back one aggregate communication operator, resulting in a loss of approximately one second, thereby reducing the computing resources and time required for service recovery.
[0029] In one possible implementation, before the first computing node sends the first information to the management node, the method further includes:
[0030] The first computing node receives a first indication message, which indicates that the data transmission of the first communication task has failed; the first computing node determines that the first link has failed based on the first indication message.
[0031] In one possible implementation, the third information is also used to indicate a second link, which is a data transmission link between the first computing node and the second computing node for reprocessing the first communication task.
[0032] In one possible implementation, the third information includes the identifier of the second link.
[0033] In one possible implementation, the first information is specifically used to indicate that an optical module failure has occurred in the first link.
[0034] In most cases, optical module failures occur as intermittent interruptions. Some optical module components have self-diagnosis and recovery functions, and the faulty link has a certain probability of self-recovery within a few seconds.
[0035] In one possible implementation, the second link indicated by the third information is the first link.
[0036] In one possible implementation, the first computing node includes a first network port and a second network port, the first link is a data transmission link based on the first network port, and the second link is a data transmission link based on the second network port.
[0037] Using the above method, for devices with multiple network ports, combined with aggregated communication re-execution technology, fault points can be bypassed, further improving cluster stability.
[0038] In one possible implementation, before the first computing node sends the second information to the management node, the method further includes:
[0039] The first computing node receives a fourth message from the management node, which is used to request the second message.
[0040] Thirdly, embodiments of this application provide a collective communication system, including: a management node and N computing nodes, wherein the N computing nodes are used to process a first communication task in parallel, and N is an integer greater than 1;
[0041] When at least one first link fails, the management node receives N first messages from N computing nodes. Each first message indicates whether the original data of the first communication task stored locally on a computing node has been modified. Each first link is a communication link between the N computing nodes for processing the first communication task. If the original data of the N computing nodes has not been modified, the management node sends second messages to the N computing nodes. The second messages instruct the N computing nodes to reprocess the first communication task based on the original data. The N computing nodes reprocess the first communication task based on the second messages and the original data.
[0042] In one possible implementation, the first link is a communication link between a first computing node and a second computing node, wherein the first computing node and the second computing node are two computing nodes among the N computing nodes, and the third information is also used to indicate a second link, which is a communication link between the first computing node and the second computing node for reprocessing the first communication task.
[0043] Fourthly, embodiments of this application provide a communication device applied in a collective communication system. The collective communication system includes a management node and N computing nodes, where the N computing nodes are used to process a first communication task in parallel, and N is an integer greater than 1. The communication device is the management node, and the device includes:
[0044] The transceiver unit is used to receive first information, which indicates that at least one first link has failed. Each first link is a communication link between N computing nodes used to process the first communication task.
[0045] The transceiver unit is also used to receive N second messages from N computing nodes, each second message indicating whether the original data of the first communication task stored locally on a computing node has been modified.
[0046] The processing unit is used to determine whether the original data of N computing nodes has been modified.
[0047] If the original data of the N computing nodes has not been changed, the transceiver unit is also used to send third information to the N computing nodes. The third information is used to instruct the N computing nodes to reprocess the first communication task based on the original data.
[0048] In one possible implementation, the first link is a communication link between a first computing node and a second computing node, where the first computing node and the second computing node are two computing nodes out of N computing nodes. The third information is also used to indicate the second link, which is a communication link between the first computing node and the second computing node for reprocessing the first communication task.
[0049] In one possible implementation, the third information includes the identifier of the second link.
[0050] In one possible implementation, the first information is specifically used to indicate that an optical module failure has occurred in the first link.
[0051] In one possible implementation, the second link indicated by the third information is the first link.
[0052] In one possible implementation, the first computing node includes a first network port and a second network port, the first link is a communication link based on the first network port, and the second link is a communication link based on the second network port.
[0053] In one possible implementation, before the management node receives N second messages from N computing nodes, the transceiver unit is further configured to:
[0054] Send a fourth message to each of the N computing nodes. Each fourth message is used to request the second message.
[0055] Fifthly, embodiments of this application provide a communication device applied to a collective communication system. The collective communication system includes a management node and N computing nodes, where the N computing nodes are used to process a first communication task in parallel, and N is an integer greater than 1. The communication device is the first computing node, which is one of the N computing nodes. The device includes:
[0056] The transceiver unit is used to send first information to the management node. The first information is used to indicate that the first link has failed. The first link is a communication link between the first computing node and the second computing node for processing the first communication task. The first computing node and the second computing node are two computing nodes out of N computing nodes.
[0057] The transceiver unit is also used to send second information to the management node, the second information being used to indicate whether the original data of the first communication task stored locally on the first computing node has been changed;
[0058] The transceiver unit is also used to receive third information from the management node. The third information is used to instruct the first computing node to reprocess the first communication task based on the original data. The third information is determined by the management node based on the fact that the original data of N computing nodes has not been changed.
[0059] The processing unit is used to reprocess the first communication task based on the third information and the original data.
[0060] In one possible implementation, before the first computing node sends the first information to the management node, the transceiver unit is further configured to: receive first indication information, the first indication information indicating that the data transmission of the first communication task has failed;
[0061] The processing unit is also used to determine that the first link has failed based on the first indication information.
[0062] In one possible implementation, the third information is also used to indicate a second link, which is a data transmission link between the first computing node and the second computing node for reprocessing the first communication task.
[0063] In one possible implementation, the third information includes the identifier of the second link.
[0064] In one possible implementation, the first information is specifically used to indicate that an optical module failure has occurred in the first link.
[0065] In one possible implementation, the second link indicated by the third information is the first link.
[0066] In one possible implementation, the first computing node includes a first network port and a second network port, the first link is a data transmission link based on the first network port, and the second link is a data transmission link based on the second network port.
[0067] In one possible implementation, before the first computing node sends the second information to the management node, the transceiver unit is further configured to: receive a fourth message from the management node, the fourth message being used to request the second message.
[0068] A sixth aspect of this application provides a communication device including at least one processor coupled to a memory; the memory is used to store a program or instructions; the at least one processor is used to execute the program or instructions to cause the device to implement the method described in any possible implementation of any of the first aspects. Optionally, the communication device may include the memory.
[0069] A seventh aspect of this application provides a communication device including at least one processor coupled to a memory; the memory is used to store a program or instructions; the at least one processor is used to execute the program or instructions to cause the device to implement the method described in any possible implementation of any of the second aspects described above. Optionally, the communication device may include the memory.
[0070] The eighth aspect of this application provides a communication device including at least one logic circuit and an input / output interface; the logic circuit is used to perform the method described in any possible implementation of any of the first aspects described above.
[0071] The ninth aspect of this application provides a communication device including at least one logic circuit and an input / output interface; the logic circuit is used to perform the method described in any possible implementation of any of the preceding second aspects.
[0072] The tenth aspect of this application provides a computer-readable storage medium including computer instructions that, when executed on a computer, cause the computer to perform an implementation as described in the first aspect or any of the first aspects.
[0073] The eleventh aspect of this application provides a computer-readable storage medium including computer instructions that, when executed on a computer, cause the computer to perform an implementation as described in the second aspect or any of the second aspects.
[0074] The twelfth aspect of this application provides a computer program product including instructions that, when run on a computer, cause the computer to perform an implementation as described in the first aspect or any of the first aspects.
[0075] The thirteenth aspect of this application provides a computer program product including instructions that, when run on a computer, cause the computer to perform an implementation as described in the second aspect or any of the second aspects.
[0076] The fourteenth aspect of this application provides a chip device including a processor for calling a program stored in a memory, such that the processor executes the first aspect or any implementation thereof.
[0077] Optionally, the memory may be located inside or outside the chip device.
[0078] The fifteenth aspect of this application provides a chip device including a processor for calling a program stored in a memory, such that the processor executes the second aspect or any implementation thereof described above.
[0079] Optionally, the memory may be located inside or outside the chip device.
[0080] The technical effects of the fourth aspect or any possible implementation of the fourth aspect, the sixth aspect, the eighth aspect, the tenth aspect, the twelfth aspect or the fourteenth aspect can be found in the first aspect or the technical effects of different possible implementations of the first aspect, and will not be repeated here.
[0081] The technical effects of the fifth aspect or any possible implementation of the fifth aspect, the seventh aspect, the ninth aspect, the eleventh aspect, the thirteenth aspect, or the fifteenth aspect can be found in the technical effects of the second aspect or different possible implementations of the second aspect, and will not be repeated here. Attached Figure Description
[0082] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0083] Figure 1 is a schematic diagram of the network architecture of a collective communication system provided in an embodiment of this application;
[0084] Figure 2A is a flowchart of a set communication operator;
[0085] Figure 2B is another flowchart of the set communication operator;
[0086] Figure 2C is another flowchart of the set communication operator;
[0087] Figure 2D is another flowchart of the set communication operator;
[0088] Figure 3 is a flowchart illustrating a collection communication method provided in an embodiment of this application;
[0089] Figure 4 is a schematic diagram of a communication link in a collective communication system;
[0090] Figure 5 is a schematic diagram of a communication link between network interface cards of computing nodes;
[0091] Figure 6 is another flowchart illustrating the collection communication method provided in an embodiment of this application;
[0092] Figure 7 is another flowchart illustrating the collection communication method provided in an embodiment of this application;
[0093] Figure 8 is a schematic diagram illustrating the technical effects of the collection communication method provided in the embodiments of this application;
[0094] Figure 9 is a structural schematic diagram of a communication device provided in an embodiment of this application;
[0095] Figure 10 is another structural schematic diagram of the communication device provided in an embodiment of this application;
[0096] Figure 11 is a structural schematic diagram of a computing device provided in an embodiment of this application;
[0097] Figure 12 is a schematic diagram of a computing device cluster provided in an embodiment of this application;
[0098] Figure 13 is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation
[0099] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0100] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0101] The terms “substantially,” “about,” and similar terms used herein are used as approximations, not as terms of degree, and are intended to take into account the inherent biases of measurements or calculations known to those skilled in the art. Furthermore, the term “may” used in describing embodiments of this application means “one or more possible embodiments.” The terms “use,” “using,” and “used” used herein are to be considered synonymous with the terms “utilize,” “utilizing,” and “utilized,” respectively. Additionally, the term “exemplary” is intended to refer to an instance or illustration.
[0102] First, some terms used in the embodiments of this application will be explained to facilitate understanding by those skilled in the art.
[0103] (1) Collective communication system
[0104] Aggregate communication systems are crucial for efficient communication between multiple computing nodes in an AI cluster. In distributed training, computing nodes frequently exchange data, including model parameters and gradient information, to ensure training consistency and accuracy. Aggregate communication systems provide the underlying support needed for these communication operations, including sending, receiving, copying, and synchronization. In parallel computing using graphics processing units (GPUs) or neural network processing units (NPUs), the computing nodes are GPUs or NPU cards.
[0105] Please refer to Figure 1, which is a schematic diagram of the network architecture of a collective communication system applied in an embodiment of this application. This collective communication system includes a management node and N computing nodes (including computing node 1, computing node 2, and computing node n in Figure 1). The N computing nodes are connected via data plane communication links, primarily based on communication links between the network interface cards (NICs) integrated into the GPU or NPU cards. These communication links typically employ remote direct memory access (RDMA) or bus protocols.
[0106] The management node includes a central processing unit (CPU). When a process (the aggregation communication process in Figure 1) assigns a computing task to a computing node, it needs to transfer relevant data from main memory to the computing node's memory, and read the result data from the computing node's memory after the communication task is completed. The computing node (GPU or NPU) is mounted as a coprocessor on the host CPU, and the host CPU assigns tasks to it.
[0107] In one possible implementation, the aggregated communication system includes multiple host CPUs, each of which is responsible for managing the communication tasks of at least one GPU (or NPU). The multiple host CPUs negotiate and synchronize with each other via TCP / IP or other more efficient communication protocols.
[0108] (2) Set communication operator
[0109] Set communication operators are specific algorithms or functions that implement set communication primitives. In distributed training and parallel computing frameworks, set communication operators are typically provided by the framework and can be re-executed. Each set communication operation involves multiple low-level network transmissions. Set communication operators provide various communication modes, such as Broadcast, Reduce, Scatter, and Gather, which can help multiple GPUs (or NPUs) efficiently synchronize data and communication results.
[0110] As shown in Figures 2A to 2D, the commonly used collection communication operators in the industry are as follows:
[0111] Figure 2A shows the Allreduce operator. It sums the data on each card, and after the Allreduce operation, all cards obtain the same summed result.
[0112] Figure 2B shows the Broadcast operator. It copies data from one card to all cards.
[0113] Figure 2C shows the Allgather operator. By concatenating the contents of each card, each card can obtain a large tensor from the combined data.
[0114] Figure 2D shows the ReduceScatter operator. After splitting the tensor on each card, each card receives its corresponding different part.
[0115] (3) Input memory (InputMem)
[0116] Input memory is the memory area used by set communication operators to receive input data during execution. It can generally be understood as the input buffer of the set communication operator. In set communication operations, each participating process or node needs to provide input data, which is stored in its respective input buffer. Then, through the action of set communication operators (such as the Allreduce operator and the Broadcast operator), this input data is exchanged and processed between processes or nodes, ultimately yielding the desired output result.
[0117] (4) Output memory (OutputMem)
[0118] Corresponding to InputMem, OutputMem is the memory area used by the set communication operator to store output data. After the set communication operator is executed, the processed data is written to OutputMem.
[0119] (5) Checkpoint mechanism
[0120] Checkpointing is a method used in AI frameworks for business-level recovery. A key characteristic of AI business training is multi-round iterative updates. Each iteration updates some stored intermediate variables, and the next iteration is based on these intermediate variables for calculation. Therefore, as long as the values of intermediate variables at the end of a certain round are available, training can continue from those values. This can be expressed as: W t+1 =W t +G t
[0121] Where W is the variable updated each time; G t The data generated in each iteration depends only on the variable W of the current iteration. t .
[0122] The variable is saved every n steps; this saved data is called a "checkpoint". For example, if a checkpoint is saved at time t-3, that is, W is saved... t-3 This state. When calculating G. t Or an error occurred during communication, causing G t Unable to complete calculation or W t If an update fails to be complete, a checkpoint can be used to restore the state. Specifically, this involves reloading the Wt-3 state and continuing subsequent calculations based on it.
[0123] In an AI cluster, each component (such as a switch or optical module) has its own failure rate. As the cluster grows and the number of components increases, the failure rates of individual components accumulate, leading to a higher overall cluster failure probability. This means that the likelihood of a failure in the cluster increases with its size, thus affecting the cluster's stability. Consequently, the probability of training tasks being interrupted due to hardware failures becomes increasingly high. For example, a Wanka AI cluster training a model together might experience a failure approximately every few hours, causing the training task to fail.
[0124] For communication failures caused by hardware issues, checkpointing is currently the most common recovery mechanism. However, checkpoints are saved at relatively long intervals, typically on an hourly basis. Furthermore, the saved checkpoint files are often enormous (up to terabytes in large models), requiring transmission over the network to network storage devices, where bandwidth is relatively limited. Therefore, both saving and loading checkpoints are time-consuming. In addition, the checkpoint recovery process requires pausing service operations, migrating the checkpoint data from network storage to the current device, and then restarting the service. This service-level restart operation also incurs significant overhead.
[0125] To address the aforementioned problems, embodiments of this application provide a collection communication method. The data processing method of this application embodiment will be described in detail below with reference to the accompanying drawings.
[0126] Figure 3 is a flowchart illustrating a collective communication method provided in an embodiment of this application. This collective communication method focuses on the recovery mechanism when a fault occurs during communication. As shown in Figure 3, this collective communication method is applied to a collective communication system, which includes a management node and N computing nodes. The N computing nodes are used to process the first communication task in parallel. The following describes the collective communication method using the interaction process between a computing node and the management node as an example. This method may include steps 301 to 306, which are described in detail below.
[0127] 301. The computing node detects a failure in the first link.
[0128] Specifically, the compute nodes are GPU cards or NPU cards, and the management node is a host CPU. The first and second compute nodes in the N compute nodes communicate via a first link. During parallel processing of the first communication task, the first and second compute nodes transmit the data of that task through this first link. When a network failure occurs, data transmission between the first and second compute nodes fails, the first communication task is interrupted, and this failure information is returned to the compute nodes, indicating a failure in the first link.
[0129] For example, N computing nodes are used to process the Allreduce operator in parallel. This requires summing the data on each computing node's InputMem and sending the summed result to each computing node's OutputMem, ultimately ensuring that all computing nodes obtain the same summed result. After a communication failure, the first computing node and / or the second computing node perform multiple retransmissions to ultimately determine that the first link has failed.
[0130] In one possible implementation, an optical module failure occurs in the first link. In most cases, this optical module failure is intermittent, and some optical module components have self-diagnostic and recovery capabilities, meaning the failed link has a certain probability of self-recovering within a few seconds.
[0131] 302. The compute node sends the first message to the management node; correspondingly, the management node receives the first message.
[0132] After determining that the first link has failed, the computing node sends a first message to the management node, indicating that the first link has failed and the training of the first communication task has been interrupted. Upon receiving the error message, the management node needs to check the computing nodes participating in the first communication task and initiate a negotiation process.
[0133] In one possible implementation, each GPU card or NPU card corresponds to a host CPU. After receiving a fault message, the host CPU needs to negotiate with other host CPUs to determine whether to re-execute the first communication task at the level of the aggregate communication operator.
[0134] 303. The management node sends a fourth message to the compute node; correspondingly, the compute node receives the fourth message.
[0135] Specifically, the management node sends a fourth message to each compute node, which requests whether the original data of the first communication task stored locally on each compute node has been modified.
[0136] Each compute node is responsible for detecting whether the raw data in InputMem has been contaminated and reporting the results to the management node. Specifically, the compute node verifies whether there is memory address overlap between InputMem and OutputMem. When executing a communication task, the compute node allocates a starting address to InputMem and OutputMem respectively, while referring to the size of the raw data block stored in InputMem and the size of the target data block stored in OutputMem. If these two memory regions overlap in address, it means that the raw data in InputMem has been contaminated, preventing the compute node from reprocessing the first communication task based on this raw data.
[0137] For example, the aggregated communication system includes NPU1, NPU2, and NPU3. NPU1 communicates with NPU2 via "Link 1," and NPU2 and NPU3 communicate via "Link 2." In a single communication task, for example, the aggregated communication operator used is the Allreduce operator. The data on each card is summed, and the same summation result is obtained on each card. The summation operation is performed within NPU1, and NPU1 transmits the result to NPU2, which then transmits the result to NPU3. However, during the transmission of the result, "Link 2" fails, while "Link 1" operates normally. At this point, NPU1 and NPU2 store the final summed data, i.e., the target data is stored in OutputMem. There is a possibility that NPU1 and NPU2 have overlapping memory addresses.
[0138] 304. The compute node sends a second message to the management node; correspondingly, the management node receives the second message.
[0139] Specifically, N compute nodes send N second messages to the management node, each second message indicating whether the original data (i.e., the original data in InputMem) stored locally on that compute node has been modified.
[0140] 305. The management node sends third information to the compute node; correspondingly, the compute node receives the third information.
[0141] Specifically, if the original data of the N computing nodes indicated by the N second messages has not been changed, the management node sends a third message to each computing node, which is used to instruct the N computing nodes to reprocess the first communication task based on the original data.
[0142] In one possible implementation, at least one backup second link exists between the first computing node and the second computing node, allowing the first computing node to reprocess the first communication task via a bypass mechanism. The third information also indicates the second link used by the first and second computing nodes to reprocess the first communication task.
[0143] In one possible implementation, the third information includes the identifier of the second link.
[0144] Figure 4 shows a schematic diagram of a communication link in a centralized communication system. This system includes multiple NPUs, multiple L1 switches, and multiple L2 switches. The NPUs are designated NPU1, NPU2, NPU3, NPU4, and NPU5. The NPUs are interconnected via the HCCS bus / RoCE protocol, using a CLOS networking approach.
[0145] For example, L1 switches include top-of-rack (TOR) switches, and L2 switches include spine switches.
[0146] For example, data transmission between NPU1 and NPU5 requires passing through L1 and L2 switches. If the L1-L2 connection includes a primary link and a backup link as shown in Figure 4, and the optical module on the primary link or the link itself fails, then after a data transmission failure in the communication task, the L1-L2 switches switch to the backup link (or wait for the primary link to recover) for data transmission. However, this communication failure may cause the loss of some data packets in the previous communication task. Using the aggregated communication method provided in this application, if the original data on multiple NPUs has not been changed, a re-execution can occur at the level of a single aggregated communication operator. This re-execution occurs after the L1-L2 link is automatically repaired, and the multiple NPUs restart processing the previously error-prone communication task based on the original data of InputMem, obtaining the final correct result. This avoids returning to a checkpoint for recovery.
[0147] In one possible implementation, the computing node includes multiple network interfaces. Figure 5 illustrates a communication link between network interface cards (NICs) of computing nodes. Based on chip-on-chip (ICO) technology, a single chip can contain multiple computing dies, each including a network interface controller (NIC).
[0148] As shown in Figure 5, communication is performed on a per-die basis. Taking communication between die0 of compute node 1 and die0 of compute node 2 as an example, the main link is link ①. When the main link fails, the link can be switched to avoid the failure point before resuming execution and restoring the service. For example, if the NIC of die0 of compute node 1 fails, communication can be switched to the backup link corresponding to the NIC of die1 of compute node 1, namely links ③ and ④ in Figure 5.
[0149] In one possible implementation, the first link is interrupted due to a fault in the optical module. Since the optical module component has self-diagnosis and recovery functions, the first link has been restored, and the second link indicated by the third information is the original first link.
[0150] 306. The compute node reprocesses the first communication task.
[0151] Specifically, if the original data in the N computing nodes has not been changed, the N computing nodes will reprocess the first communication task that was interrupted due to the previous failure based on the third information and the original data stored locally.
[0152] For example, if N computing nodes are used to process the Allreduce operator in parallel, the data on the InputMem of each computing node are summed again. The N computing nodes send the summed result to the OutputMem of each computing node through the communication link indicated in the third information, and each computing node gets the same summed result.
[0153] In another possible implementation, if any of the original data in the N computing nodes is changed, the management node decides to use the checkpoint mechanism to roll back to the previously saved state, migrate the original data stored on the network side to each computing node, and restart the training.
[0154] Please refer to Figure 6, which is a flowchart illustrating a collective communication method provided in an embodiment of this application. Compared with the service recovery mechanism of the prior art, this embodiment adds a recovery mechanism at the collective communication operator level. As shown in Figure 6, when the first link fails, the transport layer determines that the first link has failed based on multiple retransmissions, then performs fault judgment and negotiates the status of each computing node, i.e., whether the original data stored locally on the computing node has been modified. If the original data of the computing node has not been modified, the collective communication library selects to switch links, selects an available second link, and re-executes the previously failed communication task. The training task at the framework layer continues without stopping the service; the checkpoint data is migrated from network storage to the current computing node before re-execution.
[0155] In one possible implementation, the second link that was severed also fails. The transport layer determines the second link failure based on multiple reconnections. After the aggregate communication library determines that no redundant links are available, the training task of the framework layer is interrupted. A checkpoint mechanism is used to roll back to the previously saved state and rewrite the communication task.
[0156] Please refer to Figure 7, which is another flowchart illustrating the ensemble communication method provided in this embodiment. The failure of the first link is an optical module failure, which recovers automatically after a period of time. After determining that the first link has recovered, the ensemble communication library instructs that the first link continue to be used to process the first communication task. The training task at the framework layer continues.
[0157] Please refer to Figure 8, which is a schematic diagram of the technical effects of an embodiment of this application.
[0158] For example, checkpoints are saved every 30 minutes. If a communication failure occurs 20 minutes after a checkpoint is saved, recovery using the checkpoint requires reloading the checkpoint from network storage and restarting computation from that checkpoint. This results in the loss of the past 20 minutes of training progress, wasting time and resources. This application enhances the stability of the AI cluster by integrating a re-execution mechanism for ensemble communication operators into the communication library. After a communication failure, the ensemble communication system only needs to roll back one ensemble communication operator, with a loss of approximately one second, thereby reducing the computational resources and time required for service recovery.
[0159] The methods provided in the embodiments of this application have been described in detail above. Next, the device for performing the above methods provided in the embodiments of this application will be described.
[0160] Please refer to Figure 9, which is a schematic diagram of the structure of a communication device 900 provided in an embodiment of this application. This device is applied to a collective communication system, which includes a management node and N computing nodes. The N computing nodes are used to process a first communication task in parallel, where N is an integer greater than 1. The communication device is the management node. As shown in Figure 9, the device includes:
[0161] The transceiver unit 901 is used to receive first information, which indicates that at least one first link has failed. Each first link is a communication link between N computing nodes for processing a first communication task.
[0162] The transceiver unit 901 is also used to receive N second messages from N computing nodes, each second message indicating whether the original data of the first communication task stored locally on a computing node has been changed;
[0163] Processing unit 902 is used to determine whether the original data of N computing nodes has been changed;
[0164] If the original data of the N computing nodes has not been changed, the transceiver unit 901 is also used to send third information to the N computing nodes. The third information is used to instruct the N computing nodes to reprocess the first communication task based on the original data.
[0165] In one possible implementation, the first link is a communication link between a first computing node and a second computing node, where the first computing node and the second computing node are two computing nodes out of N computing nodes. The third information is also used to indicate the second link, which is a communication link between the first computing node and the second computing node for reprocessing the first communication task.
[0166] In one possible implementation, the third information includes the identifier of the second link.
[0167] In one possible implementation, the first information is specifically used to indicate that an optical module failure has occurred in the first link.
[0168] In one possible implementation, the second link indicated by the third information is the first link.
[0169] In one possible implementation, the first computing node includes a first network port and a second network port, the first link is a communication link based on the first network port, and the second link is a communication link based on the second network port.
[0170] In one possible implementation, before the management node receives N second messages from N computing nodes, the transceiver unit 901 is further configured to:
[0171] Send a fourth message to each of the N computing nodes. Each fourth message is used to request the second message.
[0172] Please refer to Figure 10, which is a schematic diagram of the structure of a communication device 1000 provided in an embodiment of this application. This device is applied to a collective communication system, which includes a management node and N computing nodes. The N computing nodes are used to process a first communication task in parallel, where N is an integer greater than 1. The communication device 1000 is the first computing node, and the first computing node is one of the N computing nodes. As shown in Figure 10, the device includes:
[0173] The transceiver unit 1001 is used to send first information to the management node. The first information is used to indicate that the first link has failed. The first link is a communication link between the first computing node and the second computing node for processing the first communication task. The first computing node and the second computing node are two computing nodes out of N computing nodes.
[0174] The transceiver unit 1001 is also used to send second information to the management node, the second information being used to indicate whether the original data of the first communication task stored locally on the first computing node has been changed;
[0175] The transceiver unit 1001 is also used to receive third information from the management node. The third information is used to instruct the first computing node to reprocess the first communication task based on the original data. The third information is determined by the management node based on the fact that the original data of N computing nodes has not been changed.
[0176] The processing unit 1002 is used to reprocess the first communication task based on the third information and the original data.
[0177] In one possible implementation, before the first computing node sends the first information to the management node, the transceiver unit 1001 is further configured to: receive first indication information, the first indication information indicating that the data transmission of the first communication task has failed;
[0178] The processing unit 1002 is also used to determine that a first link has failed based on the first indication information.
[0179] In one possible implementation, the third information is also used to indicate a second link, which is a data transmission link between the first computing node and the second computing node for reprocessing the first communication task.
[0180] In one possible implementation, the third information includes the identifier of the second link.
[0181] In one possible implementation, the first information is specifically used to indicate that an optical module failure has occurred in the first link.
[0182] In one possible implementation, the second link indicated by the third information is the first link.
[0183] In one possible implementation, the first computing node includes a first network port and a second network port, the first link is a data transmission link based on the first network port, and the second link is a data transmission link based on the second network port.
[0184] In one possible implementation, before the first computing node sends the second information to the management node, the transceiver unit 1001 is further configured to: receive a fourth message from the management node, the fourth message being used to request the second message.
[0185] This application also provides a computing device 100. As shown in FIG11, the computing device 100 includes: a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other via the bus 102. The computing device 100 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 100.
[0186] Bus 102 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 11, but this does not imply that there is only one bus or one type of bus. Bus 102 can include pathways for transmitting information between various components of computing device 100 (e.g., memory 106, processor 104, communication interface 108).
[0187] The processor 104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
[0188] Memory 106 may include volatile memory, such as random access memory (RAM). Processor 104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).
[0189] The memory 106 stores executable program code, and the processor 104 executes the executable program code to implement the functions of the aforementioned transceiver unit and processing unit, thereby realizing the collective communication method. That is, the memory 106 stores instructions for executing the collective communication method.
[0190] Alternatively, the memory 106 stores executable code, which the processor 104 executes to implement the functions of the aforementioned computing devices, thereby realizing the collective communication method. That is, the memory 106 stores instructions for executing the collective communication method.
[0191] The communication interface 103 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 100 and other devices or communication networks.
[0192] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.
[0193] As shown in Figure 12, the computing device cluster includes at least one computing device 100. The memory 106 of one or more computing devices 100 in the computing device cluster may store the same instructions for executing the aggregate communication method.
[0194] This application also relates to a computer storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.
[0195] This application also relates to a computer program product that stores instructions that, when executed by a computer, cause the computer to perform steps as performed by the aforementioned execution device, or to perform steps as performed by the aforementioned training device.
[0196] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the data processing method described in the above embodiments, or to cause the chip within the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).
[0197] Specifically, please refer to Figure 13, which is a schematic diagram of the chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1300. The NPU 1300 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1303, which is controlled by the controller 1304 to extract matrix data from the memory and perform multiplication operations.
[0198] In some implementations, the arithmetic circuit 1303 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1303 is a two-dimensional pulsating array. The arithmetic circuit 1303 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1303 is a general-purpose matrix processor.
[0199] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1302 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1301 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is stored in the accumulator 1308.
[0200] Unified memory 1306 is used to store input and output data. Weight data is directly transferred to weight memory 1302 via Direct Memory Access Controller (DMAC) 1305. Input data is also transferred to unified memory 1306 via DMAC.
[0201] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1309.
[0202] The Bus Interface Unit (BIU) 1313 is used by the instruction fetch memory 1309 to fetch instructions from external memory, and also by the memory access controller 1305 to fetch the original data of the input matrix A or the weight matrix B from external memory.
[0203] The DMAC is mainly used to move input data from external memory DDR to unified memory 1306, or to weight data to weight memory 1302, or to input data to input memory 1301.
[0204] The vector computation unit 1307 includes multiple processing units that, when needed, further process the output of the computation circuit 1303, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as Batch Normalization, pixel-level summation, and upsampling of the predicted label plane.
[0205] In some implementations, the vector computation unit 1307 can store the processed output vector in the unified memory 1306. For example, the vector computation unit 1307 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1303, such as linearly interpolating the predicted label plane extracted from the convolutional layer, or, for example, accumulating a vector of values to generate activation values. In some implementations, the vector computation unit 1307 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to the computation circuit 1303, for example, for use in subsequent layers of the neural network.
[0206] The instruction fetch buffer 1309 connected to the controller 1304 is used to store the instructions used by the controller 1304;
[0207] Unified memory 1306, input memory 1301, weighted memory 1302, and instruction fetch memory 1309 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.
[0208] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.
[0209] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.
[0210] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose GPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0211] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.
[0212] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).
Claims
1. An aggregation communication method characterized by, The method is applied to a collective communication system, which includes a management node and N computing nodes. The N computing nodes are used to process a first communication task in parallel, where N is an integer greater than 1. The method includes: The management node receives first information, which indicates that at least one first link has failed, and each first link is a communication link between the N computing nodes used to process the first communication task. The management node receives N second pieces of information from the N computing nodes, each second piece of information indicating whether the original data of the first communication task stored locally on one of the computing nodes has been changed; If the original data of the N computing nodes has not been changed, the management node sends a third message to the N computing nodes, the third message being used to instruct the N computing nodes to reprocess the first communication task based on the original data.
2. The method of claim 1, wherein, The first link is a communication link between a first computing node and a second computing node. The first computing node and the second computing node are two computing nodes among the N computing nodes. The third information is also used to indicate a second link, which is a communication link between the first computing node and the second computing node used to reprocess the first communication task.
3. The method of claim 2, wherein, The third piece of information includes the identifier of the second link.
4. The method according to claim 2 or 3, characterized in that, The first information is specifically used to indicate that an optical module failure has occurred in the first link.
5. The method of claim 4, wherein, The second link indicated by the third information is the first link.
6. The method according to any one of claims 2-5, characterized in that, The first computing node includes a first network port and a second network port, the first link is a communication link based on the first network port, and the second link is a communication link based on the second network port.
7. The method according to any one of claims 1 to 6, characterized in that, Before the management node receives N pieces of second information from the N computing nodes, the method further includes: The management node sends a fourth message to each of the N computing nodes, and each of the fourth messages is used to request the second message.
8. An aggregation communication method characterized by comprising: The method is applied to a collective communication system, which includes a management node and N computing nodes. The N computing nodes are used to process a first communication task in parallel, where N is an integer greater than 1. The method includes: The first computing node sends first information to the management node. The first information is used to indicate that the first link has failed. The first link is a communication link between the first computing node and the second computing node for processing the first communication task. The first computing node and the second computing node are two computing nodes among the N computing nodes. The first computing node sends a second message to the management node, the second message indicating whether the original data of the first communication task stored locally on the first computing node has been changed; The first computing node receives third information from the management node, the third information being used to instruct the first computing node to reprocess the first communication task based on the original data, wherein the third information is determined by the management node based on the fact that none of the original data of the N computing nodes has been changed; The first computing node reprocesses the first communication task based on the third information and the original data.
9. The method of claim 8, wherein, Before the first computing node sends the first information to the management node, the method further includes: The first computing node receives a first indication message, which indicates that the data transmission of the first communication task has failed. The first computing node determines that the first link has failed based on the first indication information.
10. The method according to claim 8 or 9, characterized in that, The third information is also used to indicate a second link, which is a data transmission link between the first computing node and the second computing node for reprocessing the first communication task.
11. The method of claim 10, wherein, The third piece of information includes the identifier of the second link.
12. The method according to claim 10 or 11, characterized in that, The first information is specifically used to indicate that an optical module failure has occurred in the first link.
13. The method of claim 12, wherein, The second link indicated by the third information is the first link.
14. The method according to any one of claims 10-12, characterized in that, The first computing node includes a first network port and a second network port, the first link is a data transmission link based on the first network port, and the second link is a data transmission link based on the second network port.
15. The method according to any one of claims 8-14, characterized in that, Before the first computing node sends the second information to the management node, the method further includes: The first computing node receives fourth information from the management node, the fourth information being used to request the second information.
16. A collective communication system, characterized by include: The system consists of a management node and N computing nodes, where the N computing nodes are used to process the first communication task in parallel, and N is an integer greater than 1. When at least one first link fails, the management node is used to receive N first messages from the N computing nodes. Each first message is used to indicate whether the original data of the first communication task stored locally by one of the computing nodes has been changed. Each first link is a communication link between the N computing nodes for processing the first communication task. If the original data of the N computing nodes has not been changed, the management node sends a second message to the N computing nodes, the second message being used to instruct the N computing nodes to reprocess the first communication task based on the original data; The N computing nodes reprocess the first communication task based on the second information and the original data.
17. A communications device, characterized by The communication device is applied to a collective communication system, which includes a management node and N computing nodes. The N computing nodes are used to process the first communication task in parallel, where N is an integer greater than 1. The communication device is the management node, and it includes a transceiver unit and a processing unit. The transceiver unit is used to perform the sending step or the receiving step in the method of any one of claims 1 to 7; The processing unit is used to perform steps in the method of any one of claims 1 to 7, excluding the sending step and the receiving step.
18. A communications device, characterized by The communication device is applied to a collective communication system, which includes a management node and N computing nodes. The N computing nodes are used to process a first communication task in parallel, where N is an integer greater than 1. The communication device is the first computing node, which is one of the N computing nodes. The communication device includes a transceiver unit and a processing unit. The transceiver unit is used to perform the sending step or the receiving step in the method of any one of claims 8 to 15; The processing unit is used to perform steps in the method of any one of claims 8 to 15, excluding the sending step and the receiving step.
19. A communications device, characterized by Includes at least one processor, said at least one processor being coupled to memory; The memory is used to store programs or instructions; The at least one processor is configured to execute the program or instructions to cause the apparatus to implement the method as claimed in any one of claims 1 to 7, or to cause the apparatus to implement the method as claimed in claim 8 or 15.
20. The communication apparatus according to claim 19, wherein The communication device is a chip or chip system.
21. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 7, or to implement the method as described in claims 8 to 15.
22. A computer program product comprising instructions, characterized in that, When it is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 7, or to implement the method as described in any one of claims 8 to 15.