Fault recovery method and system, and related product

Abstract

Embodiments of the present application relate to the technical field of big data. Disclosed are a fault recovery method and system, and a related product. In the embodiments of the present application, when an accelerator card among accelerator cards in computing nodes fails, the number of faulty accelerator cards is first obtained, the availability of a computing cluster corresponding to each replacement strategy among a plurality of replacement strategies is estimated on the basis of the number of the faulty accelerator cards, and then on the basis of the availability of the computing cluster corresponding to each replacement strategy among the plurality of replacement strategies, a target replacement strategy suitable for a current fault scenario is selected. In other words, in the embodiments of the present application, when there is a backup node in the computing cluster, no fixed replacement strategy is preconfigured, but the target replacement strategy suitable for the current fault scenario is selected from among the plurality of replacement strategies on the basis of the current fault scenario, thereby avoiding performance degradation of the computing cluster, after fault recovery, caused by preconfiguring the fixed replacement strategy.

Description

Fault recovery methods, systems and related products

[0001] This application claims priority to Chinese Patent Application No. 202510091045.2, filed on January 17, 2025, entitled "Fault Recovery Method, System and Related Products", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of big data technology, and in particular to a fault recovery method, system and related products. Background Technology

[0003] With the development of artificial intelligence (AI) technology, AI models are becoming increasingly large-scale, and correspondingly, the computing clusters used to train these models are also growing in size. Large-scale computing clusters consist of numerous computing nodes, each equipped with multiple accelerator cards such as neural processing unit (NPU) cards. During the training of AI models using these computing clusters, if any accelerator card malfunctions, fault recovery is necessary to restore the computing cluster to normal operation.

[0004] In related technologies, backup nodes are configured in computing clusters. When an accelerator card in a computing node fails, an accelerator card from the backup node is used to replace the failed one, allowing the computing cluster to continue training AI models. However, as the scale of AI models increases, tensor parallelism (TP) and other parallel strategies are often used to improve training efficiency. In this scenario, because the replacement strategy in related technologies directly uses an accelerator card from the backup node to replace the failed one, this can easily lead to cross-node communication problems among accelerator cards belonging to the same TP domain after the replacement strategy is adopted, resulting in performance degradation of the computing cluster. Summary of the Invention

[0005] This application provides a fault recovery method, system, and related products, which can select a suitable target replacement strategy from multiple replacement strategies based on the current fault scenario. The technical solution is as follows:

[0006] Firstly, a fault recovery method is provided, which is applied to the management node in a computing cluster. The computing cluster also includes multiple computing nodes and at least one backup node, and each of the multiple computing nodes and at least one backup node includes multiple accelerator cards.

[0007] In this method, the number of faulty accelerator cards in multiple computing nodes is obtained; based on the number of faulty accelerator cards, the availability of the computing cluster corresponding to each of the multiple replacement strategies for the faulty accelerator cards is estimated, wherein the replacement strategy is the strategy of replacing the faulty accelerator card with an accelerator card in at least one backup node, and the availability represents the degree of resource utilization in the computing cluster when the computing cluster executes tasks; based on the availability of the computing cluster corresponding to each of the multiple replacement strategies, a target replacement strategy for replacing the faulty accelerator card is determined; and based on the target replacement strategy, the accelerator card in the backup node is used to replace the faulty accelerator card.

[0008] In this embodiment, when an accelerator card in a computing node fails, the number of failed accelerator cards is first obtained. Based on the number of failed accelerator cards, the availability of the computing cluster corresponding to each of the various replacement strategies is estimated. Then, based on the availability of the computing cluster corresponding to each of the various replacement strategies, a target replacement strategy suitable for the current failure scenario is selected. In other words, in this embodiment, when a backup node exists in the computing cluster, a fixed replacement strategy is not pre-configured. Instead, a target replacement strategy suitable for the current failure scenario is selected from multiple replacement strategies, thereby avoiding a performance degradation of the computing cluster after fault recovery due to pre-configured fixed replacement strategies.

[0009] Based on the method provided in the first aspect, in one possible implementation, the strategy of replacing a faulty accelerator card with an accelerator card in at least one backup node includes: replacing the faulty accelerator card with M groups of accelerator cards in at least one backup node, wherein each group of accelerator cards in the M groups has N accelerator cards, and each group of accelerator cards is an accelerator card in the same backup node, and each group of accelerator cards in the M groups is used to replace a batch of accelerator cards to which a faulty accelerator card belongs; wherein, the number N of each group of accelerator cards is different for different replacement strategies.

[0010] In the M groups of accelerator cards, each group of accelerator cards is used to replace a faulty accelerator card. This can be understood as follows: for any faulty accelerator card, a group of accelerator cards from the backup node is used to replace a batch of accelerator cards in the computing node to which the faulty accelerator card belongs. This batch of accelerator cards includes the faulty accelerator card.

[0011] Based on the method provided in the first aspect, in one possible implementation, the task executed by the computing cluster is an artificial intelligence (AI) task, the AI ​​task corresponds to an AI model, and the AI ​​model corresponds to a tensor parallel domain and a sequence parallel domain; in this scenario, multiple replacement strategies include at least two of the following replacement strategies:

[0012] The first replacement strategy, in which a faulty accelerator card belongs to a batch of accelerator cards, includes all accelerator cards in the tensor parallel domain to which the faulty accelerator card belongs.

[0013] The second replacement strategy, in which a batch of accelerator cards to which a faulty accelerator card belongs includes all accelerator cards in the tensor parallel domain to which the faulty accelerator card belongs, as well as other accelerator cards in the sequence parallel domain to which each accelerator card in the tensor parallel domain belongs.

[0014] The third replacement strategy, in which a faulty accelerator card belongs to a batch of accelerator cards, includes all accelerator cards in the accelerator card board to which the faulty accelerator card belongs.

[0015] The fourth replacement strategy, in which a batch of accelerator cards to which a faulty accelerator card belongs includes all accelerator cards in the compute node to which the faulty accelerator card belongs.

[0016] In this embodiment, various replacement strategies can be configured according to the parallel domain of the AI ​​model, which improves the application flexibility of this embodiment.

[0017] Based on the method provided in the first aspect, in one possible implementation, the process of estimating the availability of the computing cluster corresponding to each of the multiple replacement strategies for faulty accelerator cards based on the number of faulty accelerator cards is as follows: Based on the number N of each group of accelerator cards corresponding to each replacement strategy and the total number of accelerator cards in at least one backup node, determine the number of faulty accelerator cards that can be replaced for each replacement strategy; for any replacement strategy, if the number of faulty accelerator cards is less than or equal to the number of faulty accelerator cards that can be replaced for the replacement strategy, estimate the degree of cluster performance degradation caused by cross-node communication within the parallel domain after replacing the faulty accelerator cards according to the replacement strategy; and determine the availability of the computing cluster corresponding to the replacement strategy based on the initial availability of the computing cluster when there are no faulty accelerator cards and the degree of cluster performance degradation caused by cross-node communication within the parallel domain.

[0018] Accordingly, when the number of faulty accelerator cards exceeds the replaceable number of faulty accelerator cards corresponding to the replacement strategy, the degree of cluster performance degradation caused by cross-node communication within the parallel domain is estimated after replacing the first part of the faulty accelerator cards according to the replacement strategy. The total number of the first part of faulty accelerator cards is equal to the replaceable number of faulty accelerator cards corresponding to the replacement strategy. The degree of cluster performance degradation caused by training degradation is estimated after fault recovery of the second part of faulty accelerator cards (excluding the first part) through training degradation. Training degradation refers to reducing the total number of data parallel domains corresponding to the AI ​​model. Based on the initial availability of the computing cluster when there are no faulty accelerator cards, the degree of cluster performance degradation caused by cross-node communication within the parallel domain, and the degree of cluster performance degradation caused by training degradation, the availability of the computing cluster corresponding to the replacement strategy is determined.

[0019] For any replacement strategy, if the number of currently faulty accelerator cards is less than or equal to the number of replaceable faulty accelerator cards corresponding to that strategy, fault recovery can be achieved by replacing all faulty accelerator cards with accelerator cards from backup nodes according to the strategy. However, if the number of currently faulty accelerator cards exceeds the number of replaceable faulty accelerator cards corresponding to that strategy, it is impossible to replace all faulty accelerator cards with accelerator cards from backup nodes according to the strategy. In this scenario, a portion of the faulty accelerator cards can be replaced with accelerator cards from backup nodes according to the strategy, and the remaining faulty accelerator cards need to be recovered through training degradation. Therefore, in this embodiment, the cluster availability corresponding to the replacement strategy needs to be calculated separately for different situations.

[0020] In this scenario, if the number of faulty accelerator cards is less than or equal to the number of replaceable faulty accelerator cards corresponding to the replacement strategy, fault recovery can be achieved by replacing all faulty accelerator cards with accelerator cards from backup nodes. Therefore, the current performance degradation of the computing cluster is mainly caused by cross-node communication within the parallel domain during the replacement of faulty accelerator cards. Thus, in this scenario, determining the cluster availability corresponding to this replacement strategy requires considering the degree of cluster performance degradation caused by cross-node communication within the parallel domain corresponding to the replacement strategy.

[0021] If the number of faulty accelerator cards exceeds the number of replaceable faulty accelerator cards under this replacement strategy, a portion of the faulty accelerator cards need to be replaced using accelerator cards from backup nodes, while the remaining faulty accelerator cards need to be recovered through training degradation. Therefore, in this scenario, determining the cluster availability corresponding to this replacement strategy requires considering not only the cluster performance degradation caused by cross-node communication within the parallel domain corresponding to the replacement strategy, but also the cluster performance degradation caused by training degradation.

[0022] Based on the method provided in the first aspect, in one possible implementation, the AI ​​model also corresponds to an expert parallel domain; in this scenario, when the replacement strategy is the first replacement strategy, the degree of cluster performance degradation caused by cross-node communication within the parallel domain includes: a first degree of performance degradation and a second degree of performance degradation. The first degree of performance degradation is the degree of cluster performance degradation caused by cross-node communication within the same sequence parallel domain, and the second degree of performance degradation is the degree of cluster performance degradation caused by cross-node communication within the same expert parallel domain.

[0023] After adopting the first replacement strategy, accelerator cards that originally belonged to the same TP domain remained within the same node, thus avoiding cross-node communication between accelerator cards within the same TP domain. However, cross-node communication might occur between accelerator cards within the same CP domain or the same EP domain. Therefore, it is necessary to determine the availability of the computing cluster corresponding to the first replacement strategy based on the degree of cluster performance degradation caused by cross-node communication within the same CP domain and the degree of cluster performance degradation caused by cross-node communication within the same EP domain.

[0024] Accordingly, when the replacement strategy is the second replacement strategy, the degree of cluster performance degradation caused by cross-node communication within the parallel domain includes: the second performance degradation degree.

[0025] When using the second replacement strategy for fault recovery, accelerator cards originally belonging to the same TP*CP domain remain within the same node, thus avoiding cross-node communication between accelerator cards within the same TP domain and the same CP domain. However, cross-node communication may occur between accelerator cards within the same EP domain. Therefore, the availability of the computing cluster corresponding to this replacement strategy needs to be determined based on the degree of cluster performance degradation caused by cross-node communication within the same EP domain.

[0026] Accordingly, when the replacement strategy is the third replacement strategy, the degree of cluster performance degradation caused by cross-node communication within the parallel domain includes: the first degree of performance degradation, the second degree of performance degradation, and the third degree of performance degradation. The third degree of performance degradation is the degree of cluster performance degradation caused by cross-node communication within the same tensor parallel domain.

[0027] When using the third replacement strategy for fault recovery, cross-node communication may occur between accelerator cards within the same TP domain, CP domain, and EP domain. Therefore, it is necessary to determine the availability of the computing cluster corresponding to the replacement strategy based on the degree of cluster performance degradation caused by cross-node communication within the same TP domain, the same CP domain, and the same EP domain.

[0028] Correspondingly, when the replacement strategy is the fourth replacement strategy, the degree of cluster performance degradation caused by cross-node communication within the parallel domain is 0.

[0029] Since replacing a single faulty accelerator card on a compute node basis does not cause cross-node communication between accelerator cards within the same TP domain, CP domain, or EP domain, the degree of cluster performance degradation caused by cross-node communication within the parallel domain is 0 in this scenario.

[0030] Secondly, a fault recovery system is provided, which is applied to the management node in a computing cluster. The computing cluster also includes multiple computing nodes and at least one backup node, and each of the multiple computing nodes and at least one backup node includes multiple accelerator cards.

[0031] The fault recovery system has the function of implementing the fault recovery method behavior described in the first aspect above. The fault recovery system includes at least one module, which is used to implement the fault recovery method provided in the first aspect above.

[0032] Thirdly, a computing cluster is provided, comprising at least one computing device, each computing device including a processor and a memory. The memory stores programs that support the computing cluster in executing the fault recovery method provided in the first aspect, and stores data related to implementing the fault recovery method provided in the first aspect. The processor is configured to execute the programs stored in the memory.

[0033] Fourthly, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when run on a computing cluster, cause the computing cluster to perform the fault recovery method described in the first aspect.

[0034] Fifthly, a computer program product containing instructions is provided that, when run on a computing cluster, causes the computing cluster to perform the fault recovery method described in the first aspect above.

[0035] The technical effects achieved by the corresponding technical means in the second to fifth aspects mentioned above are similar, and will not be repeated here. Attached Figure Description

[0036] Figure 1 is a schematic diagram illustrating the principle of replacement strategy 2;

[0037] Figure 2 is a schematic diagram of the architecture of a computing cluster provided in an embodiment of this application;

[0038] Figure 3 is a schematic diagram of the structure of a computer device provided in an embodiment of this application;

[0039] Figure 4 is a flowchart of a fault recovery method provided in an embodiment of this application;

[0040] Figure 5 is a schematic diagram of a scenario where a single faulty accelerator card is replaced on a TP domain basis, according to an embodiment of this application.

[0041] Figure 6 is a schematic diagram of a scenario in which a single faulty acceleration card is replaced on a TP*CP domain basis, according to an embodiment of this application.

[0042] Figure 7 is a schematic diagram of a scenario in which a single faulty accelerator card is replaced on a per-accelerator-card basis, according to an embodiment of this application.

[0043] Figure 8 is a schematic diagram of a scenario in which a single faulty accelerator card is replaced on a computing node basis, according to an embodiment of this application.

[0044] Figure 9 is a schematic diagram of a fault recovery system provided in an embodiment of this application. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0046] Before explaining the embodiments of this application, the technical terms used in the embodiments of this application will be explained.

[0047] CUDA Event: CUDA Event is a feature in the compute unified device architecture (CUDA) used for time measurement, time synchronization, and flow control in the graphics processing unit (GPU).

[0048] Profiling: A performance analysis tool used to understand and optimize the efficiency and resource usage of model training.

[0049] Tensor parallelism (TP) is a parallelization strategy for accelerating model training, suitable for training large-scale models. The core idea of ​​TP is to divide the parameters of a single layer of the model into multiple parts, train these parts in parallel, and finally aggregate the parallel training results to speed up the training of large-scale models. Each part of the parameters needs to be trained on a certain number of accelerator cards. The number of accelerator cards required for each part can be called a TP domain, and the number of accelerator cards in a TP domain can be called the size of the TP domain. For example, a TP domain with a size of 8 means that it includes 8 accelerator cards.

[0050] In this application, the accelerator card involved in the embodiments can be a GPU, or a neural processing unit (NPU), central processing unit (CPU), or other computing processing unit, which will not be described in detail here.

[0051] Data parallelism (DP) is a parallelization strategy used to accelerate model training, suitable for scenarios requiring large-scale training data. The core idea of ​​DP is to divide the training data into multiple mini-batches and process these mini-batches in parallel using the same model. Each mini-batch requires training on a certain number of accelerator cards. The number of accelerator cards required for each mini-batch can be called a DP domain, and the number of accelerator cards in a DP domain can be called the size of the DP domain. For example, a DP domain with a size of 72 means that it includes 72 accelerator cards.

[0052] Pipeline parallelism (PP): A parallelization strategy used to accelerate model training. The core idea of ​​PP is to divide the model training process into multiple stages and process these stages in parallel, thereby improving the training efficiency of the model.

[0053] Context parallelism (CP) is a parallelization strategy used to accelerate model training. The core idea of ​​CP is as follows: For large language models (LLMs), the input sequence to the LLM training process within any DP domain is still a long sequence. In this scenario, the long sequence is divided into multiple short sequences, and these short sequences are trained in parallel. Finally, the parallel training results are aggregated. Each short sequence needs to be trained on a certain number of accelerator cards. The number of accelerator cards required for each short sequence can be called a CP domain, and the number of accelerator cards in a CP domain can be called the size of the CP domain. For example, a CP domain with a size of 16 means that a CP domain includes 16 accelerator cards.

[0054] Expert parallelism (EP) is a parallelization strategy used to accelerate model training. The core idea of ​​EP is to execute the processing of different experts in the mixture-of-experts (MoE) layer of the model in parallel. Each expert needs to be trained on a certain number of accelerator cards. The number of accelerator cards required for each expert can be called an EP domain, and the number of accelerator cards in an EP domain can be called the size of the EP domain. For example, an EP domain with a size of 32 means that an EP domain includes 32 accelerator cards.

[0055] Field-replaceable unit (FRU): refers to a circuit board or component that can be quickly and easily removed and reinstalled by the user or technician, thus eliminating the need to send the entire product or system to a repair shop.

[0056] The application scenarios of the embodiments of this application will be described next.

[0057] With the rapid development of AI technology, the scale of computing clusters used to train AI models is also expanding rapidly, and the number of computing nodes in these clusters is increasing significantly. Computing clusters are typically deployed in data centers or cloud computing platforms, featuring highly complex network structures and a massive scale of hardware equipment.

[0058] The complexity of computing cluster network structures stems from the fact that they typically employ multi-layered network architectures, such as Top of Rack (TOR) architectures. In these multi-layered architectures, different computing nodes are interconnected via high-speed networks. The speed and stability of network communication between these nodes significantly impact the model training process; poor network communication quality can directly lead to a substantial decrease in cluster performance. This is particularly critical in training large-scale AI models, where synchronization between nodes is paramount. An anomaly in the network communication of any single node can cause delays or failures in the entire training task.

[0059] Furthermore, in large-scale computing clusters, the management and scheduling of compute nodes becomes a complex engineering problem. This is especially true when the compute nodes in the cluster contain diverse hardware architectures, such as different models of GPUs, NPUs, or CPUs on different compute nodes, which increases the overall complexity of cluster management. Simultaneously, various synchronization operations exist between the compute nodes in the cluster; if one compute node fails, the entire cluster's services become unusable.

[0060] Considering all these factors, even if the failure rate of an individual computing node in a computing cluster is low, the overall frequency of cluster failures increases dramatically with the total number of computing nodes. In some scenarios, a computing cluster with thousands of accelerator cards has a failure rate as high as 93% per day. For example, a computing cluster with 16K accelerator cards experienced 466 failures in 54 days of training an AI model, averaging 8 failures per day.

[0061] To improve the reliability and availability of computing clusters, the design of backup nodes has become an important strategy. By introducing backup nodes into a computing cluster, the cluster can tolerate the failure of some computing nodes without affecting the overall execution of tasks. After introducing backup nodes, there are two possible replacement strategies.

[0062] Replacement Strategy 1: When a computing node fails, the backup node can quickly take over the tasks of that computing node, thereby ensuring that the computing cluster can continue to execute tasks, that is, ensuring the stability of the computing cluster.

[0063] However, after restoring the computing cluster according to the replacement strategy 1 described above, the performance of the computing cluster may decrease. For example, if computing node A and computing node B are distributed under the same physical switch, the bandwidth between them is high, so computing node A and computing node B are chosen as a TP domain for model training. If computing node B fails, replacement strategy 1 will directly select the normal backup node C to replace the failed computing node B, thereby achieving rapid fault recovery. However, replacement strategy 1 does not consider the impact on cluster performance after fault recovery. Assuming that backup node C needs to communicate with computing node A across a Layer 3 switch, replacement strategy 1 will lead to a significant reduction in the bandwidth between computing node A and backup node C. Since computing node A and backup node C currently belong to the same TP domain, the amount of data that needs to be transmitted within the same TP domain is huge. Therefore, the low bandwidth between computing node A and backup node C will lead to a decrease in the performance of the entire computing cluster.

[0064] Replacement Strategy 2: Replacement is performed in units of minimum FRU, such as accelerator cards. Thus, if an accelerator card in a compute node fails, a working accelerator card is selected from the backup node to replace the failed one.

[0065] Figure 1 is a schematic diagram of the principle of replacement strategy 2. As shown in Figure 1, at time t0, the accelerator card 0 of computing node 1 in the computing cluster fails, causing the training task to be interrupted. At this time, user intervention is required to restart the training task. If there is no backup node, the user needs to readjust the parallel strategy, such as changing the number of DP domains from 3 to 2. Although this will reduce the data throughput during the execution of the training task, it can ensure the normal operation of the training task.

[0066] If a backup node is available, a working accelerator card from the backup node can be used to replace the faulty accelerator card, ensuring that the number of cards used for training remains consistent with the number before the failure, thus preventing a decrease in data throughput during the training task. As shown in Figure 1, at time t1, accelerator card 0 from the backup node is used to replace accelerator card 0 in computing node 1.

[0067] However, the replacement strategy 2 mentioned above only considers using the minimum FRU for resource replacement and does not take into account the impact on the performance of the computing cluster. For example, when there are no backup nodes, if the user restarts the training job by downgrading the training (reducing the number of DP domains), the performance of the computing cluster will be degraded by 1 / 3 compared to the performance of the computing cluster before the failure.

[0068] When a backup node is available, a faulty accelerator card on compute node 1 is replaced with a working accelerator card from the backup node. Assuming each compute node's four accelerator cards form a TP domain, after adopting replacement strategy 2, different accelerator cards within the same TP domain now need to communicate across nodes. This cross-node communication may require passing through multiple switches. If the execution efficiency of the same TP domain degrades by *a*% due to cross-node communication, then after adopting replacement strategy 2, because of the synchronization mechanism during training, if the execution efficiency of one TP domain degrades by *a*%, the overall performance of the compute cluster will also degrade by *a*%. When *a*% > 1 / 3, the addition of the backup node actually has a negative effect. This wastes a significant amount of computing resources on the backup node without yielding any benefits.

[0069] Based on this, in this embodiment, when an accelerator card in a computing node fails, the number of failed accelerator cards is first obtained. Then, based on the number of failed accelerator cards, the availability of the computing cluster corresponding to each of the various replacement strategies is estimated. Finally, a target replacement strategy suitable for the current failure scenario is selected based on the availability of the computing cluster corresponding to each of the various replacement strategies. In other words, in this embodiment, when a backup node exists in the computing cluster, a fixed replacement strategy is not pre-configured. Instead, a target replacement strategy suitable for the current failure scenario is selected from multiple replacement strategies based on the current failure scenario. This avoids the performance of the computing cluster decreasing after fault recovery due to pre-configuring a fixed replacement strategy.

[0070] The following describes the computing clusters, hardware devices, fault recovery methods, and systems involved in the embodiments of this application.

[0071] Figure 2 is a schematic diagram of the architecture of a computing cluster provided in an embodiment of this application. As shown in Figure 2, the computing cluster includes multiple computing nodes (workers) and one management node (master). Each computing node and the management node communicate with each other via wired or wireless means.

[0072] Referring again to Figure 2, the management node includes a scheduler, and each compute node includes multiple accelerator cards, which can be GPUs, NPUs, or CPUs, for example. The scheduler receives user-triggered tasks, such as model training tasks, and distributes corresponding compute tasks (compute tasks can be understood as subtasks of the task) to each accelerator card to execute the task through these multiple accelerator cards. This process can be called job scheduling.

[0073] Additionally, as shown in Figure 2, the computing cluster also includes at least one backup node (Figure 2 uses one backup node as an example). Each backup node communicates with the management node via wired or wireless means. Each backup node includes multiple accelerator cards. This allows for the replacement of a faulty accelerator card in a computing node with one from a backup node, ensuring that tasks can continue to execute.

[0074] Referring again to Figure 2, each compute node is equipped with a fault detection agent module. This module collects service metrics data from each accelerator card during task execution on the compute node, such as the time taken for each accelerator card to complete a computing task, and detects whether the accelerator card is faulty based on the collected service metrics data. When the fault detection agent module detects a fault in an accelerator card, it reports the fault information to the management node. The fault information indicates the location of the faulty accelerator card within the compute node; for example, it includes the identifier of the faulty accelerator card.

[0075] The management node is configured with a recovery decision-making module. This module selects a target replacement strategy from multiple options based on fault information reported by the compute nodes. The target replacement strategy indicates how to replace the faulty accelerator card with an accelerator card from a backup node, allowing the compute cluster to continue executing tasks. After determining the target replacement strategy, the recovery decision-making module transmits it to the scheduler. The scheduler then reissues tasks based on the target replacement strategy, thereby enabling the replacement of the faulty accelerator card with an accelerator card from a backup node according to the chosen strategy.

[0076] The aforementioned fault perception agent module, scheduler, and recovery decision determination module can be implemented in software, hardware, or a combination of both. This application embodiment does not limit this implementation.

[0077] Figure 3 is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The aforementioned computing node, management node, and backup node can all be implemented using the computer device shown in Figure 3.

[0078] As shown in Figure 3, the computer device includes at least one processor 301, a communication bus 302, a memory 303, and at least one communication interface 304.

[0079] The processor 301 may be a general-purpose central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application.

[0080] The communication bus 302 may include a path for transmitting information between the aforementioned components.

[0081] The memory 303 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or it may be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), a magnetic disk or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory 303 may exist independently and be connected to the processor 301 via a communication bus 302. The memory 303 may also be integrated with the processor 301.

[0082] The memory 303 stores program code for executing the scheme of this application, and its execution is controlled by the processor 301. The processor 301 executes the program code stored in the memory 303. The program code may include one or more software modules. Embodiments of this application can determine data for application development using the processor 301 and one or more software modules in the program code in the memory 303.

[0083] Communication interface 304 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.

[0084] In a specific implementation, as one embodiment, the computer device may include multiple processors, such as processor 301 and processor 305 shown in FIG. 3. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. Here, "processor" may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0085] In a specific implementation, as one embodiment, the computer device may further include an output device 306 and an input device 307. The output device 306 communicates with the processor 301 and can display information in various ways. For example, the output device 306 may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device 307 communicates with the processor 301 and can receive user input in various ways. For example, the input device 307 may be a mouse, a keyboard, a touchscreen device, or a sensing device, etc.

[0086] The aforementioned computer device can be a general-purpose computer device or a special-purpose computer device. In specific implementations, the computer device can be a desktop computer, a portable computer, a network server, a handheld computer (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. This application does not limit the type of computer device.

[0087] Figure 4 is a flowchart of a fault recovery method provided in an embodiment of this application. This method is exemplarily applied to the recovery decision determination module shown in Figure 2. As shown in Figure 4, the method includes the following steps.

[0088] Step 401: Obtain the number of faulty accelerator cards that have failed in multiple compute nodes in the compute cluster.

[0089] In some embodiments, in the scenario shown in Figure 2, the recovery decision determination module in the management node receives fault information reported by at least one computing node. The fault information reported by any computing node is used to indicate the information of the faulty accelerator card in the corresponding computing node. Based on the fault information reported by each computing node, the recovery decision determination module can determine the number of faulty accelerator cards that have failed in the current computing cluster.

[0090] For example, the fault information includes the identifier of each faulty accelerator card. Specifically, for any faulty accelerator card, its identifier indicates which accelerator card belongs to which compute node.

[0091] For example, a computing cluster includes compute node 1, compute node 2, and compute node 3. Each of these three compute nodes contains four accelerator cards, which are labeled as accelerator card 0, accelerator card 1, accelerator card 2, and accelerator card 3. In this scenario, the fault information reported by compute node 1 could be: Accelerator card 2 in compute node 1 has failed.

[0092] Additionally, in this embodiment, the faulty accelerator card includes an accelerator card that has malfunctioned and is unable to perform computing tasks. Optionally, the faulty accelerator card may further include an accelerator card whose computing efficiency is lower than a reference rate. The reference rate is a pre-configured rate. When the computing rate of an accelerator card is lower than the reference rate, it indicates that the computing rate of the accelerator card when performing computing tasks is very low, and therefore the accelerator card can be defined as a faulty accelerator card.

[0093] It should be noted that the definition of the fault acceleration card in this application embodiment is not limited. When applying this application embodiment, the fault acceleration card can be defined based on user needs. Examples will not be provided here.

[0094] Step 402: Estimate the availability of the computing cluster corresponding to each of the multiple replacement strategies for the faulty accelerator cards based on the number of faulty accelerator cards. The replacement strategy is the strategy of replacing the faulty accelerator card with an accelerator card from at least one backup node. Availability represents the degree of resource utilization in the computing cluster when the computing cluster executes tasks.

[0095] In this embodiment of the application, in order to avoid the performance degradation caused by a fixed replacement strategy, multiple replacement strategies for replacing faulty accelerator cards can be pre-set. Then, based on the number of faulty accelerator cards in the current fault scenario, the availability of the computing cluster corresponding to each of the multiple replacement strategies is estimated, thereby selecting the target replacement strategy suitable for the current fault scenario.

[0096] In some embodiments, the strategy of replacing a faulty accelerator card with an accelerator card from at least one backup node includes: replacing the faulty accelerator card with M groups of accelerator cards from at least one backup node, wherein each group of accelerator cards in the M groups contains N accelerator cards, and each group of accelerator cards is an accelerator card from the same backup node, and each group of accelerator cards in the M groups is used to replace a batch of accelerator cards to which a faulty accelerator card belongs.

[0097] Accordingly, the number N of each group of accelerator cards varies among different replacement strategies.

[0098] In the M groups of accelerator cards, each group of accelerator cards is used to replace a batch of accelerator cards to which a faulty accelerator card belongs. This can be understood as follows: for any faulty accelerator card, a group of accelerator cards from the backup node is used to replace a batch of accelerator cards in the computing node to which the faulty accelerator card belongs. This batch of accelerator cards includes the faulty accelerator card.

[0099] In this embodiment, the computing cluster is used to execute model-related tasks. These model-related tasks include, for example, model training tasks, real-time inference tasks based on deep learning, and model-based data processing tasks. For instance, the task executed by the computing cluster is an AI task. Subsequent embodiments will use an AI model corresponding to the AI ​​task as an example.

[0100] In addition, in this embodiment of the application, to improve the task execution efficiency of the computing cluster, a parallel strategy is configured for the AI ​​model. This parallel strategy includes, for example, TP, CP, EP, and DP. In scenarios where the parallel strategy includes TP, the accelerator cards on the computing nodes in the computing cluster are divided into multiple TP domains, and the accelerator cards in each TP domain are used to execute computing tasks belonging to the same TP domain. In scenarios where the parallel strategy includes CP, the accelerator cards on the computing nodes in the computing cluster are divided into multiple CP domains, and the accelerator cards in each CP domain are used to execute computing tasks belonging to the same CP domain. In scenarios where the parallel strategy includes EP, the accelerator cards on the computing nodes in the computing cluster are divided into multiple EP domains, and the accelerator cards in each EP domain are used to execute computing tasks belonging to the same EP domain. In scenarios where the parallel strategy includes DP, the accelerator cards on the computing nodes in the computing cluster are divided into multiple DP domains, and the accelerator cards in each DP domain are used to execute computing tasks belonging to the same DP domain.

[0101] Furthermore, during AI model training, different parallel domains have different network bandwidth requirements. Based on the required data volume and transmission frequency, the bandwidth requirements of parallel domains are ranked as follows: bandwidth requirement within the same TP domain > bandwidth requirement within the same CP domain > bandwidth requirement within the same EP domain > bandwidth requirement within the same DP domain > bandwidth requirement within the same PP domain. This is because different accelerator cards within the same TP domain need to perform frequent intermediate layer activation value communication, thus requiring high bandwidth; different accelerator cards within the same CP domain need to perform frequent key-value (KV) communication, thus requiring high bandwidth; and different accelerator cards within the same EP domain need to perform frequent activation value communication, thus requiring high bandwidth.

[0102] Considering that the communication bandwidth demand is greatest within the same TP domain, the same CP domain, and the same EP domain, accelerator cards within the same TP domain are generally distributed in the same computing node, accelerator cards within the same CP domain are distributed in the same computing node, and accelerator cards within the same EP domain are distributed in the same computing node.

[0103] Based on this, in the embodiments of this application, the multiple replacement strategies include at least two of the following: a first replacement strategy, a second replacement strategy, a third replacement strategy, and a fourth replacement strategy.

[0104] The first replacement strategy is as follows: N is the size of the tensor parallel domain of the AI ​​model corresponding to the AI ​​task. Correspondingly, a batch of accelerator cards belonging to a failed accelerator card includes all accelerator cards in the tensor parallel domain to which the failed accelerator card belongs. In other words, a single failed accelerator card is replaced by an accelerator card from a backup node, on a per-TP domain basis. In other words, when an accelerator card fails within a TP domain, an accelerator card from one of the backup nodes is used to replace all accelerator cards within that TP domain.

[0105] The second replacement strategy is as follows: N is the product of the size of the tensor parallel domain and the size of the sequence parallel domain of the AI ​​model corresponding to the AI ​​task. Accordingly, a batch of accelerator cards belonging to a failed accelerator card includes all accelerator cards in the tensor parallel domain to which the failed accelerator card belongs, as well as the other accelerator cards in the sequence parallel domain to which each accelerator card in the tensor parallel domain belongs. That is, a single failed accelerator card is replaced by an accelerator card from a backup node, with each TP*CP domain as a unit. Assuming the size of a TP domain is w, a TP*CP domain can be understood as a domain composed of the CP domains to which w accelerator cards belong within the TP domain to which the single failed accelerator card belongs. In other words, when an accelerator card fails in a TP*CP domain of a computing node, an accelerator card from one of the backup nodes is used to replace all accelerator cards in that TP*CP domain.

[0106] For example, a TP domain has a size of 4, and a CP domain has a size of 3. For a faulty accelerator card, suppose its TP domain includes accelerator card 1, accelerator card 2, accelerator card 3, and accelerator card 4, where accelerator card 4 is the faulty accelerator card. Then, determine the CP domain to which accelerator card 1 belongs, which includes accelerator card 1, accelerator card 5, and accelerator card 9; determine the CP domain to which accelerator card 2 belongs, which includes accelerator card 2, accelerator card 6, and accelerator card 10; determine the CP domain to which accelerator card 3 belongs, which includes accelerator card 3, accelerator card 7, and accelerator card 11; determine the CP domain to which accelerator card 4 belongs, which includes accelerator card 4, accelerator card 8, and accelerator card 12. Therefore, the TP*CP domain to which the faulty accelerator card belongs includes accelerator card 1 through accelerator card 12, a total of 12 accelerator cards.

[0107] The third replacement strategy is as follows: N is the number of accelerator cards included in an accelerator board within the computing cluster. Correspondingly, a batch of accelerator cards belonging to a failed accelerator card includes all accelerator cards in the accelerator board to which the failed accelerator card belongs. In other words, on a per-accelerator-board basis, an accelerator card is replaced by an accelerator card from at least one backup node.

[0108] The fourth replacement strategy is as follows: N is the number of accelerator cards in a compute node. Correspondingly, a batch of accelerator cards belonging to a failed accelerator card includes all accelerator cards in the compute node to which the failed accelerator card belongs. In other words, on a per-compute-node basis, an accelerator card from a backup node is used to replace a single failed accelerator card. Put simply, when an accelerator card fails in a compute node, one of the backup nodes is used to replace that compute node.

[0109] In this embodiment, various replacement strategies can be configured according to the parallel domain of the AI ​​model, which improves the application flexibility of this embodiment.

[0110] It should be noted that the above four replacement strategies are for illustrative purposes only. Other replacement strategies may also be used when applying the embodiments of this application, which will not be illustrated here.

[0111] Furthermore, since the replacement unit (N) used to replace a single faulty accelerator card differs across replacement strategies, the total number of times at least one backup node can be used to replace a single faulty accelerator card also varies depending on the replacement strategy. For ease of explanation, for any replacement strategy, the total number of times at least one backup node can be used to replace a single faulty accelerator card can be referred to as the number of faulty accelerator cards that can be replaced under that replacement strategy.

[0112] For any replacement strategy, if the number of currently faulty accelerator cards is less than or equal to the number of replaceable faulty accelerator cards corresponding to that strategy, fault recovery can be achieved by replacing all faulty accelerator cards with accelerator cards from backup nodes according to the strategy. However, if the number of currently faulty accelerator cards exceeds the number of replaceable faulty accelerator cards corresponding to that strategy, it is impossible to replace all faulty accelerator cards with accelerator cards from backup nodes according to the strategy. In this scenario, a portion of the faulty accelerator cards can be replaced with accelerator cards from backup nodes according to the strategy, and the remaining faulty accelerator cards need to be recovered through training degradation. Therefore, in this embodiment, the cluster availability corresponding to the replacement strategy needs to be calculated separately for different situations.

[0113] Based on this, in this embodiment of the application, the implementation method for estimating the availability of the computing cluster corresponding to each of the multiple replacement strategies for faulty accelerator cards based on the number of faulty accelerator cards can be as follows: determine the number of faulty accelerator cards that can be replaced for each replacement strategy based on the number N of each group of accelerator cards corresponding to each replacement strategy and the total number of accelerator cards in at least one backup node; for any replacement strategy, if the number of faulty accelerator cards is less than or equal to the number of faulty accelerator cards that can be replaced for that replacement strategy, estimate the degree of cluster performance degradation caused by cross-node communication within the parallel domain after replacing the faulty accelerator cards according to that replacement strategy, and determine the availability of the computing cluster corresponding to that replacement strategy based on the initial availability of the computing cluster when there are no faulty accelerator cards and the degree of cluster performance degradation caused by cross-node communication within the parallel domain.

[0114] It should be noted that in parallel configurations including DPs, the probability of multiple accelerator cards failing simultaneously within a single DP domain is typically very low. Therefore, in this embodiment, when a DP domain fails, it is assumed that only one accelerator card in that DP domain has failed. That is, only a single accelerator card failure will occur within a DP domain. In this scenario, the number of failed accelerator cards can also be referred to as the number of failed DP domains.

[0115] When the number of faulty accelerator cards is less than or equal to the number of replaceable faulty accelerator cards corresponding to this replacement strategy, fault recovery can be completed by replacing all faulty accelerator cards with accelerator cards from backup nodes according to this replacement strategy. Therefore, the current performance degradation of the computing cluster is mainly caused by cross-node communication within the parallel domain when replacing faulty accelerator cards. Therefore, in this scenario, determining the cluster availability corresponding to this replacement strategy needs to consider the degree of cluster performance degradation caused by cross-node communication within the parallel domain corresponding to the replacement strategy.

[0116] Accordingly, when the number of faulty accelerator cards exceeds the number of replaceable faulty accelerator cards corresponding to the replacement strategy, the degree of cluster performance degradation caused by cross-node communication within the parallel domain is estimated after replacing the first part of the faulty accelerator cards according to the replacement strategy. The total number of the first part of faulty accelerator cards is equal to the number of replaceable faulty accelerator cards corresponding to the replacement strategy. The degree of cluster performance degradation caused by training degradation is estimated after fault recovery of the second part of faulty accelerator cards (excluding the first part) through training degradation. Training degradation refers to reducing the total number of data parallel domains of the AI ​​model corresponding to the AI ​​task. The availability of the computing cluster corresponding to the replacement strategy is determined based on the initial availability of the computing cluster when there are no faulty accelerator cards, the degree of cluster performance degradation caused by cross-node communication within the parallel domain, and the degree of cluster performance degradation caused by training degradation.

[0117] If the number of faulty accelerator cards exceeds the number of replaceable faulty accelerator cards under this replacement strategy, a portion of the faulty accelerator cards need to be replaced using accelerator cards from backup nodes, while the remaining faulty accelerator cards need to be recovered through training degradation. Therefore, in this scenario, determining the cluster availability corresponding to this replacement strategy requires considering not only the cluster performance degradation caused by cross-node communication within the parallel domain corresponding to the replacement strategy, but also the cluster performance degradation caused by training degradation.

[0118] For example, the implementation of determining the cluster availability corresponding to the replacement strategy based on the degree of cluster performance degradation caused by cross-node communication and training degradation can be as follows: For the first part of the faulty accelerator cards, the first cluster availability is determined based on the initial availability of the computing cluster when no faulty accelerator cards appear and the degree of cluster performance degradation caused by cross-node communication within the parallel domain corresponding to the replacement strategy; for the second part of the faulty accelerator cards, the second cluster availability is determined based on the initial availability of the computing cluster when no faulty accelerator cards appear and the degree of cluster performance degradation caused by training degradation; and the availability of the computing cluster corresponding to the replacement strategy is determined based on the first cluster availability and the second cluster availability.

[0119] Using the methods described above, the cluster availability can be calculated separately for the replacement of faulty accelerator cards and the training degradation approach. These two cluster availability values ​​are then processed to obtain the cluster availability corresponding to the replacement strategy in this scenario.

[0120] The method for determining the availability of the computing cluster corresponding to the replacement strategy based on the availability of the first cluster and the availability of the second cluster can be as follows: The availability of the first cluster is added to the availability of the second cluster to obtain the cluster availability corresponding to the replacement strategy. Optionally, different weights can be pre-configured for the availability of the first cluster and the availability of the second cluster, and then the availability of these two clusters can be processed based on these two weights to obtain the cluster availability corresponding to the replacement strategy. Further examples will not be provided here.

[0121] Furthermore, the total number of faulty acceleration cards in the first part can also be less than the number of replaceable faulty acceleration cards corresponding to this replacement strategy. The total number of faulty acceleration cards in the first part can be pre-configured, for example, it can be set to 90% of the number of replaceable faulty acceleration cards corresponding to this replacement strategy.

[0122] The following explanation uses the four replacement strategies mentioned above as examples to illustrate step 402. In scenarios where the replacement strategy is another, step 402 can also be implemented using the following method.

[0123] (1) The replacement strategy is the first replacement strategy, in which a single faulty accelerator card is replaced by an accelerator card in a backup node on a TP domain basis.

[0124] The number of faulty accelerator cards that can be replaced under the first replacement strategy is the ratio between the total number of all accelerator cards in at least one backup node and the size of a TP domain (i.e., N corresponding to the first replacement strategy). The size of a TP domain is the number of accelerator cards included in a TP domain.

[0125] Accordingly, the degree of cluster performance degradation caused by cross-node communication within the parallel domain corresponding to the first replacement strategy includes: a first degree of performance degradation and a second degree of performance degradation. The first degree of performance degradation is the degree of cluster performance degradation caused by cross-node communication within the same CP domain, and the second degree of performance degradation is the degree of cluster performance degradation caused by cross-node communication within the same EP domain.

[0126] Figure 5 is a schematic diagram of a scenario where a single faulty accelerator card is replaced on a TP domain basis, according to an embodiment of this application. As shown in Figure 5, assuming a TP domain includes 4 accelerator cards, i.e., the size of a TP domain is 4, and currently 3 DP domains have failed, these 3 DP domains are labeled DP#1, DP#2 and DP#3 respectively. As shown in Figure 5, each of these 3 DP domains contains one faulty accelerator card. When adopting the first replacement strategy, the backup node needs to have 3 sets of accelerator cards, each set of faulty cards includes 4 accelerator cards, and then each set of accelerator cards is used to replace all accelerator cards in the TP domain to which the faulty accelerator card in a faulty DP domain belongs. In this way, after adopting the first replacement strategy, accelerator cards that originally belonged to the same TP domain are still within the same node, thus avoiding cross-node communication between accelerator cards in the same TP domain. However, accelerator cards within the same CP domain or the same EP domain may engage in cross-node communication. Therefore, it is necessary to determine the availability of the computing cluster corresponding to the first replacement strategy based on the degree of cluster performance degradation caused by cross-node communication within the same CP domain and the degree of cluster performance degradation caused by cross-node communication within the same EP domain.

[0127] In this scenario, when the number of faulty accelerator cards is less than or equal to the number of faulty accelerator cards that can be replaced under the first replacement strategy, the availability of the computing cluster corresponding to the first replacement strategy can be achieved by formula (1) as follows: θ1(1)=A[(1-b%)(1-c%)] Formula (1)

[0128] Where θ1(1) is: the availability of the computing cluster corresponding to the first replacement strategy when the number of faulty accelerator cards is less than or equal to the number of faulty accelerator cards that can be replaced by the first replacement strategy. A is the initial availability of the computing cluster when there are no faulty accelerator cards. b% is the first performance degradation level. c% is the second performance degradation level.

[0129] Accordingly, when the number of faulty accelerator cards exceeds the number of faulty accelerator cards that can be replaced under the first replacement strategy, the availability of the computing cluster corresponding to the first replacement strategy can be achieved by formula (2) as follows:

[0130] Where θ1(2) is: the availability of the computing cluster corresponding to the first replacement strategy when the number of faulty accelerator cards is greater than the number of faulty accelerator cards that can be replaced by the first replacement strategy. a1 is the number of faulty accelerator cards that can be replaced by the first replacement strategy, and dp is the total number of DP domains before training degradation. β1 is the availability of the first cluster, and β2 is the availability of the second cluster.

[0131] k represents the number of faulty accelerator cards. R represents the probability of an accelerator card malfunctioning. Accordingly, This represents the probability that i DP fields out of dp DP fields are simultaneously faulty. Additionally, This indicates the degree of cluster performance degradation caused by training downgrades.

[0132] For example, the total number of accelerator cards in each compute node is equal to the product of the size of the TP domain, the size of the CP domain, and the size of the EP domain. That is, the total number of accelerator cards in each backup node = TP * CP * EP. In this scenario, a1 = CP * EP * n, where n is the total number of backup nodes.

[0133] (2) The replacement strategy is the second replacement strategy, in which a single faulty accelerator card is replaced by an accelerator card in a backup node, with a TP*CP domain as the unit.

[0134] The number of faulty acceleration cards that can be replaced under the second replacement strategy is the ratio between the total number of all acceleration cards in at least one backup node and the first value (i.e., N corresponding to the second replacement strategy). The first value is the product of the size of a TP domain and the size of a CP domain. The degree of cluster performance degradation caused by cross-node communication within the parallel domain corresponding to the second replacement strategy includes the second performance degradation degree.

[0135] Figure 6 is a schematic diagram of a scenario where a single faulty accelerator card is replaced on a TP*CP domain basis, according to an embodiment of this application. As shown in Figure 6, assuming a TP*CP domain includes 8 accelerator cards, and 3 DP domains are currently faulty, these 3 DP domains are labeled DP#1, DP#2, and DP#3 respectively. As shown in Figure 6, each of these 3 DP domains contains one faulty accelerator card. When adopting the second replacement strategy, the backup node needs to have 3 sets of accelerator cards, each set of faulty cards including 8 accelerator cards. Then, each set of accelerator cards is used to replace all accelerator cards in the TP*CP domain to which the faulty accelerator card in a faulty DP domain belongs. In this way, after adopting the second replacement strategy, accelerator cards that originally belonged to the same TP*CP domain are still within the same node, thus avoiding cross-node communication between accelerator cards in the same TP domain and the same CP domain. However, cross-node communication may occur between accelerator cards in the same EP domain. Therefore, it is necessary to determine the cluster availability corresponding to the second replacement strategy based on the degree of cluster performance degradation caused by cross-node communication within the same EP domain.

[0136] In this scenario, when the number of faulty accelerator cards is less than or equal to the number of faulty accelerator cards that can be replaced under the second replacement strategy, the availability of the computing cluster corresponding to the second replacement strategy can be achieved by formula (3) as follows: θ2(1)=A[(1-c%)] Formula (3)

[0137] Wherein, θ2(1) is: the availability of the computing cluster corresponding to the second replacement strategy when the number of faulty accelerator cards is less than or equal to the number of faulty accelerator cards that can be replaced by the second replacement strategy.

[0138] Accordingly, when the number of faulty accelerator cards exceeds the number of faulty accelerator cards that can be replaced under the second replacement strategy, the availability of the computing cluster under the second replacement strategy can be achieved by formula (4) as follows:

[0139] Where θ2(2) is the availability of the computing cluster corresponding to the second replacement strategy when the number of faulty accelerator cards is greater than the number of faulty accelerator cards that can be replaced under the second replacement strategy. a2 is the number of faulty accelerator cards that can be replaced under the second replacement strategy.

[0140] For example, the total number of accelerator cards in each compute node is equal to the product of the size of the TP domain, the size of the CP domain, and the size of the EP domain. That is, the total number of accelerator cards in each backup node = TP * CP * EP. In this scenario, a2 = EP * n, where n is the total number of backup nodes.

[0141] (3) The replacement strategy is the third replacement strategy, which replaces a single faulty accelerator card with an accelerator card in a backup node on a unit basis.

[0142] The number of replaceable faulty accelerator cards corresponding to the third replacement strategy is the ratio between the total number of accelerator cards in at least one backup node and the number of accelerator cards on an accelerator card board (i.e., N corresponding to the third replacement strategy). The degree of cluster performance degradation caused by cross-node communication within the parallel domain corresponding to the third replacement strategy includes: a first degree of performance degradation, a second degree of performance degradation, and a third degree of performance degradation. The third degree of performance degradation is the degree of cluster performance degradation caused by cross-node communication within the same TP domain.

[0143] Figure 7 is a schematic diagram of a scenario where a single faulty accelerator card is replaced on a per-accelerator-board basis, according to an embodiment of this application. As shown in Figure 7, assuming an accelerator-board includes two accelerator cards, and a TP domain includes four accelerator cards, three DP domains are currently experiencing failures, labeled DP#1, DP#2, and DP#3 respectively. As shown in Figure 7, each of these three DP domains contains one faulty accelerator card. When adopting the third replacement strategy, the backup node needs to have three accelerator-boards, each containing two accelerator cards. Then, each accelerator-board replaces all accelerator cards in the accelerator-board to which the faulty accelerator card in a faulty DP domain belongs. Thus, after adopting the third replacement strategy, cross-node communication may occur between accelerator cards within the same TP domain, the same CP domain, and the same EP domain. Therefore, the cluster availability corresponding to the third replacement strategy needs to be determined based on the degree of cluster performance degradation caused by cross-node communication within the same TP domain, the same CP domain, and the same EP domain.

[0144] In this scenario, when the number of faulty accelerator cards is less than or equal to the number of faulty accelerator cards that can be replaced under the third replacement strategy, the availability of the computing cluster under the second replacement strategy can be achieved by formula (5) as follows: θ3(1)=A[(1-a%)(1-b%)(1-c%)] Formula (5)

[0145] Where θ3(1) is: the availability of the computing cluster corresponding to the third replacement strategy when the number of faulty accelerator cards is less than or equal to the number of faulty accelerator cards that can be replaced by the third replacement strategy. a% is the third performance degradation level.

[0146] Accordingly, when the number of faulty accelerator cards exceeds the number of faulty accelerator cards that can be replaced under the third replacement strategy, the availability of the computing cluster under the third replacement strategy can be achieved by formula (6) as follows:

[0147] Where θ3(2) is: the availability of the computing cluster corresponding to the third replacement strategy when the number of faulty accelerator cards is greater than the number of faulty accelerator cards that can be replaced by the third replacement strategy. a3 is the number of faulty accelerator cards that can be replaced by the third replacement strategy.

[0148] For example, the total number of accelerator cards in each compute node is equal to the product of the size of the TP domain, the size of the CP domain, and the size of the EP domain. That is, the total number of accelerator cards in each backup node = TP * CP * EP. In this scenario, assuming the number of accelerator cards in a TP domain is m times the number of accelerator cards on an accelerator card board, then a3 = TP * CP * EP * n / m, where n is the total number of backup nodes.

[0149] (4) The replacement strategy is the fourth replacement strategy. In the fourth replacement strategy, a single faulty accelerator card is replaced by an accelerator card in a backup node on a unit of computing node.

[0150] The number of faulty accelerator cards that can be replaced under the fourth replacement strategy is the ratio between the total number of all accelerator cards in at least one backup node and the number of accelerator cards on a computing node (i.e., N under the fourth replacement strategy). The degree of cluster performance degradation caused by cross-node communication within the parallel domain under the fourth replacement strategy is 0.

[0151] Figure 8 is a schematic diagram of a scenario where a single faulty accelerator card is replaced on a compute node basis, according to an embodiment of this application. As shown in Figure 8, three DP domains have failed, labeled DP#1, DP#2, and DP#3. Each of these three DP domains contains one faulty accelerator card. When using the fourth replacement strategy, if the size of a single backup node is the same as the size of a single compute node, three backup nodes are required. Each backup node then replaces all accelerator cards in the compute node containing the faulty accelerator card in one faulty DP domain. Thus, after adopting the fourth replacement strategy, cross-node communication between accelerator cards within the same TP domain, CP domain, and EP domain will not occur. That is, the cluster performance degradation caused by cross-node communication within the parallel domain corresponding to the fourth replacement strategy is 0.

[0152] In this scenario, when the number of faulty accelerator cards is less than or equal to the number of faulty accelerator cards that can be replaced under the third replacement strategy, the cluster availability under the second replacement strategy can be achieved by formula (7) as follows: θ4(1)=A Formula (7)

[0153] Wherein, θ4(1) is: the cluster availability corresponding to the fourth replacement strategy when the number of faulty accelerator cards is less than or equal to the number of faulty accelerator cards that can be replaced by the fourth replacement strategy. Since replacing a single faulty accelerator card on a compute node basis does not cause cross-node communication between accelerator cards in the same TP domain, the same CP domain, or the same EP domain, the degree of cluster performance degradation caused by cross-node communication in this scenario is 0.

[0154] Accordingly, when the number of faulty accelerator cards exceeds the number of faulty accelerator cards that can be replaced under the fourth replacement strategy, the cluster availability under the fourth replacement strategy can be achieved by formula (8) as follows:

[0155] Where θ4(2) is the cluster availability corresponding to the third replacement strategy when the number of faulty acceleration cards is greater than the number of faulty acceleration cards that can be replaced by the fourth replacement strategy. a4 is the number of faulty acceleration cards that can be replaced by the fourth replacement strategy.

[0156] For example, the total number of accelerator cards in each compute node is equal to the product of the size of the TP domain, the size of the CP domain, and the size of the EP domain. That is, the total number of accelerator cards in each backup node = TP * CP * EP. In this scenario, a4 = n, where n is the total number of backup nodes.

[0157] In addition, all parameters in the above formula, except for the number of fault acceleration cards k, can be pre-configured.

[0158] For example, the initial cluster availability of the computing cluster when no performance degradation has occurred can be pre-calculated using the following formula (9):

[0159] Where MTBF is the fault interval time, t ch For training pause time, t ckpt To save the overhead for CKPT (a file format used to store model parameters), t re Fault detection and resource rescheduling time, t rm Training recovery time. These parameters can be obtained from the model configuration or through CUDA Event testing. In the above formula (9), A can represent the proportion of time that the AI ​​model actually uses to perform training tasks during the training period.

[0160] Alternatively, in this embodiment of the application, A can also be calculated using the following formula (10):

[0161] In the above formula (10), A1 is the actual size of the data processed by the computing cluster, and A2 is the maximum data that the underlying hardware of the computing cluster can support.

[0162] Formulas (9) and (10) above are used to illustrate how A is calculated. Optionally, in the embodiments of this application, A can also be determined in other ways, which will not be elaborated here. For example, an A can be set by a technician based on experience.

[0163] Furthermore, the aforementioned first, second, and third performance degradation levels can be calculated based on the configuration of the computing cluster. For example, regarding the third performance degradation level, which is the cluster performance degradation caused by cross-node communication between different accelerator cards in the same TP domain, the minimum communication bandwidth between the computing node and the backup node, as well as the minimum communication bandwidth between different accelerator cards within the computing node, can be determined, and the ratio of these two can be used as the third performance degradation level.

[0164] Optionally, the first, second, and third performance degradation levels can also be set by technical personnel based on experience, and will not be elaborated on here.

[0165] Step 403: Determine the target replacement strategy for replacing the faulty accelerator card based on the availability of the computing cluster corresponding to each of the multiple replacement strategies.

[0166] After obtaining the cluster availability corresponding to each of the multiple replacement strategies in step 402, the replacement strategy with the highest availability of the corresponding computing cluster can be selected as the target replacement strategy.

[0167] Step 404: Replace the faulty accelerator card with the accelerator card from the backup node according to the target replacement strategy.

[0168] For example, if the number of replaceable faulty accelerator cards corresponding to the target replacement strategy is greater than or equal to the current number of faulty accelerator cards, then all current faulty accelerator cards can be replaced by accelerator cards in the backup node. In this scenario, replacing faulty accelerator cards with accelerator cards in the backup node according to the target replacement strategy can be understood as: the scheduler shown in Figure 2 re-issues the tasks originally issued to the replaced accelerator cards (including all faulty accelerator cards) to the accelerator cards in the backup node.

[0169] For example, if the number of replaceable faulty accelerator cards corresponding to the target replacement strategy is less than the current number of faulty accelerator cards, then some of the faulty accelerator cards can be replaced by accelerator cards in the backup nodes, while other faulty accelerator cards cannot be replaced by accelerator cards in the backup nodes. In this scenario, the scheduler not only needs to re-divide the tasks to reduce the number of DP domains and achieve training degradation, but also needs to re-distribute the tasks originally assigned to the replaced accelerator cards (including some faulty accelerator cards) to the accelerator cards in the backup nodes.

[0170] In summary, in this embodiment, when an accelerator card in a computing node fails, the number of failed accelerator cards is first obtained. Based on the number of failed accelerator cards, the availability of the computing cluster corresponding to each of the various replacement strategies is estimated. Then, based on the availability of the computing cluster corresponding to each of the various replacement strategies, a target replacement strategy suitable for the current failure scenario is selected. In other words, in this embodiment, when a backup node exists in the computing cluster, a fixed replacement strategy is not pre-configured. Instead, a target replacement strategy suitable for the current failure scenario is selected from multiple replacement strategies, thereby avoiding a performance degradation of the computing cluster after fault recovery due to pre-configured fixed replacement strategies.

[0171] Figure 9 is a schematic diagram of a fault recovery system provided in an embodiment of this application. The fault recovery system 900 is applied to the management node in a computing cluster, which also includes multiple computing nodes and at least one backup node. Each of the multiple computing nodes and at least one backup node includes multiple accelerator cards. The fault recovery system 900 includes the following modules.

[0172] The acquisition module 901 is used to acquire the number of faulty accelerator cards that have failed in multiple computing nodes; the specific implementation method can be referred to step 401 in the embodiment of Figure 4.

[0173] The estimation module 902 is used to estimate the availability of the computing cluster corresponding to each of the multiple replacement strategies for the faulty accelerator cards based on the number of faulty accelerator cards. The replacement strategy is to replace the faulty accelerator card with an accelerator card from at least one backup node. The availability represents the utilization of resources in the computing cluster when the computing cluster executes a task. For a specific implementation, please refer to step 402 in the embodiment of Figure 4.

[0174] The determination module 903 is used to determine the target replacement strategy for replacing the faulty accelerator card based on the availability of the computing cluster corresponding to each replacement strategy among multiple replacement strategies; the specific implementation method can be referred to step 403 in the embodiment of Figure 4.

[0175] The replacement module 904 is used to replace the faulty acceleration card with an acceleration card from the backup node according to the target replacement strategy. For a specific implementation, please refer to step 404 in the embodiment shown in Figure 4.

[0176] In some embodiments, the strategy of replacing a faulty accelerator card with an accelerator card in at least one backup node includes: replacing the faulty accelerator card with M groups of accelerator cards in at least one backup node, wherein each group of accelerator cards in the M groups has N accelerator cards and each group of accelerator cards is an accelerator card in the same backup node, and each group of accelerator cards in the M groups is used to replace a batch of accelerator cards to which a faulty accelerator card belongs.

[0177] Among these various replacement strategies, the number N of each group of accelerator cards differs depending on the specific replacement strategy.

[0178] In some embodiments, the task executed by the computing cluster is an artificial intelligence (AI) task, which corresponds to an AI model, and the AI ​​model corresponds to a tensor parallel domain and a sequence parallel domain.

[0179] Multiple replacement strategies include at least two of the following replacement strategies:

[0180] The first replacement strategy, in which a faulty accelerator card belongs to a batch of accelerator cards, includes all accelerator cards in the tensor parallel domain to which the faulty accelerator card belongs.

[0181] The second replacement strategy, in which a batch of accelerator cards to which a faulty accelerator card belongs includes all accelerator cards in the tensor parallel domain to which the faulty accelerator card belongs, as well as other accelerator cards in the sequence parallel domain to which each accelerator card in the tensor parallel domain belongs.

[0182] The third replacement strategy, in which a faulty accelerator card belongs to a batch of accelerator cards, includes all accelerator cards in the accelerator card board to which the faulty accelerator card belongs.

[0183] The fourth replacement strategy, in which a batch of accelerator cards to which a faulty accelerator card belongs includes all accelerator cards in the compute node to which the faulty accelerator card belongs.

[0184] In some embodiments, the estimation module is used for:

[0185] Based on the number N of each group of accelerator cards corresponding to each replacement strategy and the total number of accelerator cards in at least one backup node, determine the number of faulty accelerator cards that can be replaced for each replacement strategy.

[0186] For any replacement strategy, if the number of faulty accelerator cards is less than or equal to the number of faulty accelerator cards that can be replaced according to the replacement strategy, estimate the degree of cluster performance degradation caused by cross-node communication within the parallel domain after replacing the faulty accelerator cards according to the replacement strategy. Based on the initial availability of the computing cluster when there are no faulty accelerator cards and the degree of cluster performance degradation caused by cross-node communication within the parallel domain, determine the availability of the computing cluster corresponding to the replacement strategy.

[0187] In some embodiments, the estimation module is further configured to:

[0188] If the number of faulty accelerator cards is greater than the number of faulty accelerator cards that can be replaced according to the replacement strategy, the degree of cluster performance degradation caused by cross-node communication in the parallel domain is estimated after replacing the first part of the faulty accelerator cards according to the replacement strategy. The total number of the first part of faulty accelerator cards is equal to the number of faulty accelerator cards that can be replaced according to the replacement strategy.

[0189] The extent of cluster performance degradation caused by training degradation is estimated when the second part of the faulty accelerator cards (excluding the first part of the accelerator cards) is restored through training degradation. Training degradation refers to reducing the total number of data parallel domains of the AI ​​model corresponding to the AI ​​task.

[0190] The availability of the computing cluster corresponding to this replacement strategy is determined based on the initial availability of the computing cluster when there is no faulty accelerator card, the degree of cluster performance degradation caused by cross-node communication within the parallel domain, and the degree of cluster performance degradation caused by training degradation.

[0191] In some embodiments, when the replacement strategy is a first replacement strategy, the degree of cluster performance degradation caused by cross-node communication within the parallel domain includes: a first degree of performance degradation and a second degree of performance degradation. The first degree of performance degradation is the degree of cluster performance degradation caused by cross-node communication within the same sequence parallel domain, and the second degree of performance degradation is the degree of cluster performance degradation caused by cross-node communication within the same expert parallel domain.

[0192] When the replacement strategy is the second replacement strategy, the degree of cluster performance degradation caused by cross-node communication within the parallel domain includes: the second performance degradation degree;

[0193] When the replacement strategy is the third replacement strategy, the degree of cluster performance degradation caused by cross-node communication within the parallel domain includes: the first degree of performance degradation, the second degree of performance degradation, and the third degree of performance degradation. The third degree of performance degradation is: the degree of cluster performance degradation caused by cross-node communication within the same tensor parallel domain.

[0194] When the replacement strategy is the fourth replacement strategy, the degree of cluster performance degradation caused by cross-node communication within the parallel domain is 0.

[0195] In summary, in this embodiment, when an accelerator card in a computing node fails, the number of failed accelerator cards is first obtained. Based on the number of failed accelerator cards, the availability of the computing cluster corresponding to each of the various replacement strategies is estimated. Then, based on the availability of the computing cluster corresponding to each of the various replacement strategies, a target replacement strategy suitable for the current failure scenario is selected. In other words, in this embodiment, when a backup node exists in the computing cluster, a fixed replacement strategy is not pre-configured. Instead, a target replacement strategy suitable for the current failure scenario is selected from multiple replacement strategies, thereby avoiding a performance degradation of the computing cluster after fault recovery due to pre-configured fixed replacement strategies.

[0196] It should be noted that the fault recovery system provided in the above embodiments is only illustrated by the division of the above functional modules when performing tasks. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the fault recovery system and fault recovery method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0197] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0198] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects, and should not be construed as indicating or implying relative importance.

[0199] In the description of the embodiments in this application, unless otherwise stated, "at least one" means one or more. "More than one" means two or more.

[0200] A references B, which means that A is the same as B or A is a simple variation of B.

[0201] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that there are three possible relationships. For example, A and / or B means: A exists alone, A and B exist simultaneously, or B exists alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0202] Optionally, in the above embodiments, all or part of the implementation is carried out by software, hardware, firmware, or any combination thereof. Optionally, when implemented using software, it is implemented in the form of a computer program product, which is implemented in whole or in part. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. Optionally, the computer is a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Optionally, the computer instructions are stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. Optionally, the computer-readable storage medium is any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. Alternatively, the available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video disks (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0203] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A fault recovery method, characterized in that, The method is applied to a management node in a computing cluster, the computing cluster further comprising multiple computing nodes and at least one backup node, each of the multiple computing nodes and the at least one backup node comprising multiple accelerator cards; the method includes: Obtain the number of faulty accelerator cards that have failed among the multiple computing nodes; Based on the number of faulty accelerator cards, the availability of the computing cluster corresponding to each of the multiple replacement strategies for the faulty accelerator cards is estimated. The replacement strategy is to replace the faulty accelerator card with an accelerator card from at least one backup node. The availability represents the degree of resource utilization in the computing cluster when the computing cluster executes a task. The target replacement strategy for replacing the faulty accelerator card is determined based on the availability of the computing cluster corresponding to each of the multiple replacement strategies. The faulty accelerator card is replaced with an accelerator card from the backup node according to the target replacement strategy.

2. The method as described in claim 1, characterized in that, The strategy of replacing the faulty accelerator card with accelerator cards in the at least one backup node includes: replacing the faulty accelerator card with M groups of accelerator cards in the at least one backup node, wherein each group of accelerator cards in the M groups has N accelerator cards and each group of accelerator cards is an accelerator card in the same backup node, and each group of accelerator cards in the M groups of accelerator cards is used to replace a batch of accelerator cards to which a faulty accelerator card belongs. Among these multiple replacement strategies, the number N of each group of accelerator cards is different for each replacement strategy.

3. The method as described in claim 2, characterized in that, The computing cluster executes artificial intelligence (AI) tasks, each AI task corresponds to an AI model, and each AI model corresponds to a tensor parallel domain and a sequence parallel domain. The multiple replacement strategies include at least two of the following replacement strategies: The first replacement strategy, wherein the batch of accelerator cards to which the faulty accelerator card belongs includes all accelerator cards in the tensor parallel domain to which the faulty accelerator card belongs. The second replacement strategy, wherein the batch of accelerator cards to which a faulty accelerator card belongs includes all accelerator cards in the tensor parallel domain to which the faulty accelerator card belongs, and other accelerator cards in the sequence parallel domain to which each accelerator card in the tensor parallel domain belongs. The third replacement strategy, wherein the batch of accelerator cards to which the faulty accelerator card belongs includes all accelerator cards in the accelerator card board to which the faulty accelerator card belongs; The fourth replacement strategy, wherein the batch of accelerator cards to which the faulty accelerator card belongs includes all accelerator cards in the computing node to which the faulty accelerator card belongs.

4. The method as described in claim 3, characterized in that, The step of estimating the availability of the computing cluster corresponding to each of the multiple replacement strategies for the faulty accelerator cards based on the number of faulty accelerator cards includes: Based on the number N of each group of accelerator cards corresponding to each replacement strategy and the total number of accelerator cards in the at least one backup node, determine the number of faulty accelerator cards that can be replaced for each replacement strategy. For any replacement strategy, if the number of faulty accelerator cards is less than or equal to the number of faulty accelerator cards that can be replaced according to the replacement strategy, the degree of cluster performance degradation caused by cross-node communication within the parallel domain after replacing the faulty accelerator cards according to the replacement strategy is estimated. Based on the initial availability of the computing cluster when there are no faulty accelerator cards and the degree of cluster performance degradation caused by cross-node communication within the parallel domain, the availability of the computing cluster corresponding to the replacement strategy is determined.

5. The method as described in claim 4, characterized in that, The method further includes: When the number of faulty accelerator cards is greater than the number of faulty accelerator cards that can be replaced according to the replacement strategy, the degree of cluster performance degradation caused by cross-node communication in the parallel domain is estimated after replacing the first part of the faulty accelerator cards according to the replacement strategy. The total number of the first part of faulty accelerator cards is equal to the number of faulty accelerator cards that can be replaced according to the replacement strategy. The degree of cluster performance degradation caused by training degradation is estimated after the second part of the faulty accelerator cards (excluding the first part of the accelerator cards) in the faulty accelerator cards are recovered through training degradation. The training degradation refers to reducing the total number of data parallel domains corresponding to the AI ​​model. The availability of the computing cluster corresponding to the replacement strategy is determined based on the initial availability of the computing cluster when there is no faulty accelerator card, the degree of cluster performance degradation caused by cross-node communication within the parallel domain, and the degree of cluster performance degradation caused by the training degradation.

6. The method as described in any one of claims 4-5, characterized in that, The AI ​​model also corresponds to an expert parallel domain; When the replacement strategy is the first replacement strategy, the degree of cluster performance degradation caused by cross-node communication within the parallel domain includes: a first degree of performance degradation and a second degree of performance degradation. The first degree of performance degradation is the degree of cluster performance degradation caused by cross-node communication within the same sequence parallel domain, and the second degree of performance degradation is the degree of cluster performance degradation caused by cross-node communication within the same expert parallel domain. When the replacement strategy is the second replacement strategy, the degree of cluster performance degradation caused by cross-node communication within the parallel domain includes: the second performance degradation degree; When the replacement strategy is the third replacement strategy, the degree of cluster performance degradation caused by cross-node communication within the parallel domain includes: the first degree of performance degradation, the second degree of performance degradation, and the third degree of performance degradation. The third degree of performance degradation is: the degree of cluster performance degradation caused by cross-node communication within the same tensor parallel domain. When the replacement strategy is the fourth replacement strategy, the degree of cluster performance degradation caused by cross-node communication within the parallel domain is 0.

7. A fault recovery system, characterized in that, The system is applied to a management node in a computing cluster, which further includes multiple computing nodes and at least one backup node, each of the multiple computing nodes and the at least one backup node including multiple accelerator cards; the system includes: The acquisition module is used to acquire the number of faulty accelerator cards that have failed among the multiple computing nodes; The estimation module is used to estimate the availability of the computing cluster corresponding to each of the multiple replacement strategies for the faulty accelerator cards based on the number of faulty accelerator cards. The replacement strategy is to replace the faulty accelerator card with an accelerator card from at least one backup node. The availability represents the degree of resource utilization in the computing cluster when the computing cluster performs a task. The determination module is used to determine the target replacement strategy for replacing the faulty accelerator card based on the availability of the computing cluster corresponding to each of the multiple replacement strategies; The replacement module is used to replace the faulty accelerator card with an accelerator card from the backup node according to the target replacement strategy.

8. The system as described in claim 7, characterized in that, The strategy of replacing the faulty accelerator card with accelerator cards in the at least one backup node includes: replacing the faulty accelerator card with M groups of accelerator cards in the at least one backup node, wherein each group of accelerator cards in the M groups has N accelerator cards and each group of accelerator cards is an accelerator card in the same backup node, and each group of accelerator cards in the M groups of accelerator cards is used to replace a batch of accelerator cards to which a faulty accelerator card belongs. Among these multiple replacement strategies, the number N of each group of accelerator cards is different for each replacement strategy.

9. The system as described in claim 8, characterized in that, The computing cluster executes artificial intelligence (AI) tasks, each AI task corresponds to an AI model, and each AI model corresponds to a tensor parallel domain and a sequence parallel domain. The multiple replacement strategies include at least two of the following replacement strategies: The first replacement strategy, wherein the batch of accelerator cards to which the faulty accelerator card belongs includes all accelerator cards in the tensor parallel domain to which the faulty accelerator card belongs. The second replacement strategy, wherein the batch of accelerator cards to which a faulty accelerator card belongs includes all accelerator cards in the tensor parallel domain to which the faulty accelerator card belongs, and other accelerator cards in the sequence parallel domain to which each accelerator card in the tensor parallel domain belongs. The third replacement strategy, wherein the batch of accelerator cards to which the faulty accelerator card belongs includes all accelerator cards in the accelerator card board to which the faulty accelerator card belongs; The fourth replacement strategy, wherein the batch of accelerator cards to which the faulty accelerator card belongs includes all accelerator cards in the computing node to which the faulty accelerator card belongs.

10. The system as described in claim 9, characterized in that, The prediction module is used for: Based on the number N of each group of accelerator cards corresponding to each replacement strategy and the total number of accelerator cards in the at least one backup node, determine the number of faulty accelerator cards that can be replaced for each replacement strategy. For any replacement strategy, if the number of faulty accelerator cards is less than or equal to the number of faulty accelerator cards that can be replaced according to the replacement strategy, the degree of cluster performance degradation caused by cross-node communication within the parallel domain after replacing the faulty accelerator cards according to the replacement strategy is estimated. Based on the initial availability of the computing cluster when there are no faulty accelerator cards and the degree of cluster performance degradation caused by cross-node communication within the parallel domain, the availability of the computing cluster corresponding to the replacement strategy is determined.

11. The system as claimed in claim 10, characterized in that, The prediction module is also used for: When the number of faulty accelerator cards is greater than the number of faulty accelerator cards that can be replaced according to the replacement strategy, the degree of cluster performance degradation caused by cross-node communication in the parallel domain is estimated after replacing the first part of the faulty accelerator cards according to the replacement strategy. The total number of the first part of faulty accelerator cards is equal to the number of faulty accelerator cards that can be replaced according to the replacement strategy. The degree of cluster performance degradation caused by training degradation is estimated after the second part of the faulty accelerator cards (excluding the first part of the accelerator cards) is recovered through training degradation. The training degradation refers to reducing the total number of data parallel domains of the AI ​​model corresponding to the AI ​​task. The availability of the computing cluster corresponding to the replacement strategy is determined based on the initial availability of the computing cluster when there is no faulty accelerator card, the degree of cluster performance degradation caused by cross-node communication within the parallel domain, and the degree of cluster performance degradation caused by the training degradation.

12. The system as described in any one of claims 10-11, characterized in that, When the replacement strategy is the first replacement strategy, the degree of cluster performance degradation caused by cross-node communication within the parallel domain includes: a first degree of performance degradation and a second degree of performance degradation. The first degree of performance degradation is the degree of cluster performance degradation caused by cross-node communication within the same sequence parallel domain, and the second degree of performance degradation is the degree of cluster performance degradation caused by cross-node communication within the same expert parallel domain. When the replacement strategy is the second replacement strategy, the degree of cluster performance degradation caused by cross-node communication within the parallel domain includes: the second performance degradation degree; When the replacement strategy is the third replacement strategy, the degree of cluster performance degradation caused by cross-node communication within the parallel domain includes: the first degree of performance degradation, the second degree of performance degradation, and the third degree of performance degradation. The third degree of performance degradation is: the degree of cluster performance degradation caused by cross-node communication within the same tensor parallel domain. When the replacement strategy is the fourth replacement strategy, the degree of cluster performance degradation caused by cross-node communication within the parallel domain is 0.

13. A computer program product containing instructions, characterized in that, When the instruction is run by the computing cluster, the computing cluster performs the method as described in any one of claims 1-6.

14. A computer-readable storage medium, characterized in that, It includes computer program instructions that, when executed by a computing cluster, cause the computing cluster to perform the method as described in any one of claims 1-6.

Images