Fault handling method and system, and related product

Abstract

Embodiments of the present application relate to the technical field of AI, and disclose a fault handling method and system, and a related product. The method comprises: when a fault occurs in a first accelerator card in a computing cluster, determining a first training state on the basis of a current training state in which at least one accelerator card in the computing cluster is training an AI model; on the basis of the first training state, controlling a plurality of second training processes corresponding to a plurality of second accelerator cards in the computing cluster to lock the training process of the AI model by the plurality of second accelerator cards in a second training state; and when the fault occurring in the first accelerator card has been recovered, restarting the training process corresponding to the first accelerator card that has recovered from the fault, such that said first accelerator card is restored to the second training state, and the first accelerator card and the second accelerator cards continue AI model training from the second training state. In this way, the time required between the occurrence of a fault and the retraining of the AI model can be shortened.

Description

Troubleshooting methods, systems and related products

[0001] This application claims priority to Chinese Patent Application No. 202510214112.5, filed on February 24, 2025, entitled "Fault Handling Method, System and Related Products", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of AI technology, and in particular to a fault handling method, system and related products. Background Technology

[0003] With the development of artificial intelligence (AI) technology, AI models are becoming increasingly larger, and correspondingly, the computing clusters used to train these models are also growing in size. Large-scale computing clusters consist of numerous computing nodes, each equipped with multiple accelerator cards such as neural processing unit (NPU) cards. During the training of AI models using these computing clusters, if any accelerator card malfunctions, the fault must be addressed to restore the computing cluster to normal operation.

[0004] In related technologies, each computing node also runs a training process corresponding to each accelerator card. When an accelerator card malfunctions, the training processes corresponding to all other working accelerator cards are first terminated. Then, after the malfunctioning accelerator card is restored, the training processes corresponding to all accelerator cards are restarted to continue training the AI ​​model. This method is time-consuming, taking approximately ten minutes to half an hour from the occurrence of the malfunction to the resumption of AI model training. Summary of the Invention

[0005] This application provides a fault handling method, system, and related products, which can shorten the time required from the occurrence of a fault to the continuation of AI model training. The technical solution is as follows:

[0006] Firstly, a fault handling method is provided. In this method, when a first accelerator card in a computing cluster malfunctions, a first training state is determined based on the current training state of at least one accelerator card in the computing cluster training an AI model. Based on the first training state, multiple second training processes corresponding to multiple second accelerator cards in the computing cluster are controlled to lock the training process of the AI ​​model by the multiple second accelerator cards in the second training state. After the fault of the first accelerator card is recovered, the training process corresponding to the recovered first accelerator card is restarted, so that the recovered first accelerator card returns to the second training state, and the first and second accelerator cards continue to train the AI ​​model from the second training state.

[0007] In the fault handling method provided in this application embodiment, when the first accelerator card in the computing cluster fails, it is not necessary to exit the training process corresponding to other normal accelerator cards. Instead, the training process corresponding to other normal accelerator cards is locked in the second training state. After the fault of the first accelerator card is recovered, it is only necessary to restart the training process corresponding to the first accelerator card after the fault is recovered. There is no need to restart the training processes corresponding to all other normal accelerator cards. Therefore, the time required from the occurrence of the fault to the retraining of the AI ​​model can be shortened.

[0008] Furthermore, after the failure of the first accelerator card is resolved, the control system resumes AI model training from the second training state locked after the failure, instead of retraining the AI ​​model from the most recently saved checkpoint. This avoids repeatedly performing certain iterative training, thus saving computational resources.

[0009] Based on the method provided in the first aspect, in one possible implementation, the method of determining the first training state according to the current training state of the AI ​​model being trained by at least one accelerator card in the computing cluster can be as follows: while the first training process corresponding to the first accelerator card is still running, the current training state of the AI ​​model being trained by the first accelerator card is obtained from the first training process, and the current training state of the AI ​​model being trained by the first accelerator card is taken as the first training state, and the first training process is exited.

[0010] The above implementation method is applicable to the following scenario: when the first accelerator card fails, the first training process corresponding to the first accelerator card can still run. At this time, the current training state of the first accelerator card training the AI ​​model can be obtained from the first training process, and the current training state can be determined as the first training state.

[0011] Based on the method provided in the first aspect, in one possible implementation, the method of determining the first training state according to the current training state of at least one accelerator card in the computing cluster training the AI ​​model can be as follows: when the first training process corresponding to the first accelerator card is not running, obtain the current training states of multiple second accelerator cards training the AI ​​model from multiple second training processes; select one current training state from the multiple current training states of the AI ​​model training the second accelerator cards as the first training state.

[0012] The above implementation method is applicable to the following scenario: when the first accelerator card malfunctions, the first training process corresponding to the first accelerator card has stopped running. At this time, it is necessary to obtain the current training status of the AI ​​model trained by the multiple second accelerator cards from multiple second training processes, so as to determine the first training status based on the current training status of the AI ​​model trained by these multiple second accelerator cards.

[0013] Based on the method provided in the first aspect, in one possible implementation, the first training state is the first iteration number of the AI ​​model training, and the second training state is the second iteration number of the AI ​​model training.

[0014] The implementation method for locking the training process of multiple second accelerator cards on the AI ​​model in the second training state according to the first training state control computing cluster can be as follows: A first iteration number is sent to multiple second training processes to instruct them to lock the training process of multiple second accelerator cards on the AI ​​model in the first training state; wherein, when a third training process among the multiple second training processes receives the first iteration number, and the third accelerator card corresponding to the third training process has completed the optimizer update for the first iteration number, the third training process locks the training process of the third accelerator card on the AI ​​model in the second iteration number and feeds back the second iteration number, which is different from the first iteration number; the second iteration number is then sent to multiple second training processes to instruct them to lock the training process of multiple second accelerator cards on the AI ​​model in the second iteration number.

[0015] In the above implementation, there may be scenarios where the execution of "locking to the first iteration number" in some second training processes fails. In such scenarios, all second training processes can be controlled to relock the training process of multiple second accelerator cards on the AI ​​model to a new iteration number, that is, the second iteration number.

[0016] Based on the method provided in the first aspect, in one possible implementation, the first training state is the first iteration number of the AI ​​model training, and the second training state is the second iteration number of the AI ​​model training.

[0017] The implementation method for locking the training process of the AI ​​model by the multiple second accelerator cards in the computing cluster according to the first training state can be as follows: the first iteration number is sent to the multiple second training processes to instruct them to lock the training process of the AI ​​model by the multiple second accelerator cards in the first training state; wherein, when the multiple second training processes receive the first iteration number, and none of the multiple second training processes have completed the optimizer update in the first iteration number, the multiple second training processes lock the first iteration number, and the second iteration number is the same as the first iteration number.

[0018] In the above implementation, if none of the second training processes have completed the optimizer update in the first iteration after receiving the first iteration number, it indicates that all second training processes have successfully locked the training process of the AI ​​model on the second accelerator card to the first iteration number. In this scenario, it is not necessary to lock multiple times to ensure the following result: all second training processes lock the training process of the AI ​​model on the corresponding second accelerator card to the same iteration number.

[0019] Based on the method provided in the first aspect, in one possible implementation, for the fourth accelerator card in the computing cluster, when the fourth accelerator card completes the optimizer update for the current iteration number, the fourth training process corresponding to the fourth accelerator card takes the next iteration number as the corresponding current training state; when the fourth accelerator card has not completed the optimizer update for the current iteration number, the fourth training process takes the current iteration number as the corresponding current training state.

[0020] When the fourth accelerator card has completed the optimizer update for the current iteration, it indicates that the card has completed the relevant calculations for the sample data corresponding to the current iteration and updated the local optimizer parameters based on the calculation results. Therefore, the next iteration can be used as the current training state, indicating that the fourth accelerator card can directly start training from the next iteration when it continues training. Conversely, when the fourth accelerator card has not completed the optimizer update for the current iteration, it indicates that the card has not yet completed the relevant calculations for the sample data corresponding to the current iteration. Therefore, the current iteration can be used as the current training state, indicating that the fourth accelerator card needs to retrain from the current iteration when it continues training.

[0021] Based on the method provided in the first aspect, in one possible implementation, after the fault of the first accelerator card is recovered, the training process corresponding to the recovered first accelerator card is restarted, so that the recovered first accelerator card returns to the second training state. The implementation method is as follows: after restarting the training process corresponding to the recovered first accelerator card, the state information of the first accelerator card in the second training state is obtained, and the state information includes optimizer parameters; the state information of the first accelerator card in the second training state is sent to the training process corresponding to the recovered first accelerator card.

[0022] In the scenario where the second training state is the second iteration of training the AI ​​model, the optimizer parameters of the first accelerator card in the second training state can be understood as the optimizer parameters obtained by the first accelerator card after completing the optimizer update operation of the previous iteration before the second iteration. This allows the model parameters to be updated based on these optimizer parameters during subsequent training, and the training process for the second iteration can continue based on the updated model parameters.

[0023] Based on the method provided in the first aspect, in one possible implementation, fault recovery of the first accelerator card includes replacing all accelerator cards in the parallel domain to which the first accelerator card belongs with a batch of backup accelerator cards located in the same backup node in the computing cluster.

[0024] After restarting the training process corresponding to the first accelerator card after the fault recovery, the implementation method for obtaining the status information of the first accelerator card in the second training state can be: after restarting the training process corresponding to each backup accelerator card in a batch of backup accelerator cards, obtain the status information of each accelerator card in the parallel domain in the second training state.

[0025] Accordingly, the implementation method for sending the status information of the first accelerator card in the second training state to the training process corresponding to the first accelerator card after the fault is recovered can be: sending the status information of each accelerator card in the parallel domain in the second training state to the training process corresponding to the backup accelerator card.

[0026] In a scenario where the fault repair strategy is to replace all accelerator cards in the parallel domain to which the first accelerator card belongs with a batch of backup accelerator cards located in the same backup node of the computing cluster, it is necessary to restore all the replaced batch of backup accelerator cards to the second training state.

[0027] Secondly, a fault handling system is provided, which has the function of implementing the fault handling method described in the first aspect. The fault handling system includes at least one module for implementing the fault handling method provided in the first aspect.

[0028] Thirdly, a computing cluster is provided, comprising at least one computing device. The computing device includes a processor and a memory. The memory stores programs that support the computing device in performing some or all of the steps in the fault handling method provided in the first aspect, and stores data involved in implementing some or all of the steps in the fault handling method provided in the first aspect. The processor is configured to execute the programs stored in the memory.

[0029] Fourthly, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when run on a computing cluster, cause the computing cluster to perform the fault handling method described in the first aspect.

[0030] Fifthly, a computer program product containing instructions is provided that, when run on a computing cluster, causes the computing cluster to execute the fault handling method described in the first aspect.

[0031] The technical effects achieved by the technical means corresponding to the second to fifth aspects mentioned above are similar to the technical effects achieved by the technical means corresponding to the first aspect, and will not be repeated here. Attached Figure Description

[0032] Figure 1 is a schematic diagram of a distributed training scenario involving multiple machines and multiple GPUs;

[0033] Figure 2 is a schematic diagram of the architecture of a computing cluster provided in an embodiment of this application;

[0034] Figure 3 is a schematic diagram of another computing cluster architecture provided in an embodiment of this application;

[0035] Figure 4 is a schematic diagram of a deployment method of a control center module and a processing module provided in an embodiment of this application;

[0036] Figure 5 is a schematic diagram of another deployment method of the control center module and processing module provided in the embodiment of this application;

[0037] Figure 6 is a schematic diagram of another computing cluster architecture provided in an embodiment of this application;

[0038] Figure 7 is a flowchart of a fault handling method provided in an embodiment of this application;

[0039] Figure 8 is a schematic diagram of a scenario showing the scope of fault repair under a first fault type or a second fault type, provided in an embodiment of this application.

[0040] Figure 9 is a schematic diagram of a scenario illustrating the fault repair range under a third fault type, as provided in an embodiment of this application.

[0041] Figure 10 is a schematic diagram of an iterative computation process in the training of an AI model according to an embodiment of this application;

[0042] Figure 11 is a schematic diagram of an optimizer parallelism scenario provided by an embodiment of this application;

[0043] Figure 12 is a flowchart of another fault handling method provided in an embodiment of this application;

[0044] Figure 13 is a schematic diagram comparing the training process before and after fault recovery according to an embodiment of this application;

[0045] Figure 14 is a schematic diagram of the architecture of a fault handling system provided in an embodiment of this application;

[0046] Figure 15 is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0048] Before describing the embodiments of this application, let's first introduce the application scenarios involved in the embodiments of this application.

[0049] With the development of technologies such as computing power, storage, and networks, AI technology based on deep learning has entered a period of rapid development. At the same time, AI models such as machine vision models and language models have been implemented in production and have had a certain impact on people's lives, bringing many conveniences to daily life. Among them, large-scale language models (LLM) have become a hot technology due to their outstanding performance.

[0050] As the scale of AI models increases, it becomes difficult to meet the training needs of a single accelerator card. Therefore, distributed training with multiple machines and multiple accelerator cards is the main method for training large-scale AI models. Figure 1 is a schematic diagram of a distributed training scenario with multiple machines and multiple accelerator cards. As shown in Figure 1, the computing cluster includes multiple computing nodes, each computing node includes multiple accelerator cards, and each computing node also includes a training process corresponding to each accelerator card. Each training process is used to distribute the operators to be calculated when training the AI ​​model to the corresponding accelerator card, and each accelerator card is used to calculate the operators distributed by the corresponding training process.

[0051] Meanwhile, the training time for large-scale AI models can reach several months. To avoid the need to restart the AI ​​model due to failures during training, checkpoints are periodically saved, that is, checkpoints are saved at regular intervals (hereinafter referred to as CKPT periods). Therefore, checkpoints are also called periodic checkpoints. A checkpoint includes the currently trained model parameters (such as model weights), optimizer parameters, and the current iteration number. When any accelerator card fails, the training processes corresponding to all normal accelerator cards are terminated to interrupt the training of the AI ​​model. After the failed accelerator card recovers, the training processes corresponding to all accelerator cards are restarted, and all training processes continue to train the AI ​​model based on the most recently saved checkpoint.

[0052] In the aforementioned technology, when an accelerator card malfunctions, the training processes corresponding to all working accelerator cards need to be terminated. After the faulty accelerator card recovers, the training processes for all accelerator cards must be restarted. This process takes at least ten to thirty minutes, which is highly detrimental to training AI models. Furthermore, after the faulty accelerator card recovers, all training processes continue training the AI ​​model based on the most recently saved checkpoint. Currently, within a single CKPT cycle, the computing cluster may perform many iterative training iterations on the AI ​​model. This leads to the following problem: all iterative training performed between the time of the most recent checkpoint saving and the time of the failure needs to be re-executed after the failure is resolved, resulting in a waste of computing resources.

[0053] Based on this, this application provides a fault handling method. In the fault handling method provided in this application, when the first accelerator card in the computing cluster fails, it is not necessary to exit the training processes corresponding to other normal accelerator cards. Instead, the training processes corresponding to other normal accelerator cards are locked in a second training state. Subsequently, after the fault of the first accelerator card is recovered, it is only necessary to restart the training process corresponding to the recovered first accelerator card. There is no need to restart the training processes corresponding to all other normal accelerator cards. Therefore, the time required from the occurrence of the fault to the retraining of the AI ​​model can be shortened.

[0054] Furthermore, after the failure of the first accelerator card is resolved, the control system resumes AI model training from the second training state locked after the failure, instead of retraining the AI ​​model from the most recently saved checkpoint. This avoids repeatedly performing certain iterative training, thus saving computational resources.

[0055] The computing cluster, fault handling method, and related products provided in the embodiments of this application will be described in detail below.

[0056] Figure 2 is a schematic diagram of a computing cluster architecture provided in an embodiment of this application. As shown in Figure 2, the computing cluster includes a management node 10 and multiple computing nodes 20. Each computing node 20 communicates with the management node 10 via a wired or wireless connection.

[0057] As shown in Figure 2, each computing node 20 includes a CPU 21 and multiple accelerator cards 22. For any computing node 20, the CPU 21 of that computing node 20 runs training processes 221 corresponding to the multiple accelerator cards 22. For example, if a computing node 20 includes 10 accelerator cards 22, then the CPU of the computing node 20 runs 10 training processes corresponding to these 10 accelerator cards.

[0058] When training an AI model via a computing cluster, the management node 10 distributes tasks to the training processes 221 in the CPUs of each computing node 20; this process can be called job scheduling. The training process 221 in each computing node 20 distributes operators to the corresponding accelerator cards 22 based on the received tasks, and the corresponding accelerator cards 22 compute the received operators.

[0059] In some embodiments, to improve the efficiency of training AI models using a computing cluster, parallel strategies are also configured for the AI ​​models. These parallel strategies typically include tensor parallelism (TP), data parallelism (DP), pipeline parallelism (PP), and optimizer parallelism (OP). In scenarios where the parallel strategy includes TP, the accelerator cards in the computing cluster are divided into multiple TP domains, and the accelerator cards within each TP domain are used to execute training tasks belonging to the same TP domain. In scenarios where the parallel strategy includes DP, the accelerator cards in the computing cluster are divided into multiple DP domains, and the accelerator cards within each DP domain are used to execute computation tasks belonging to the same DP domain. In scenarios where the parallel strategy includes PP, the accelerator cards in the computing cluster are divided into multiple PP domains, and the accelerator cards within each PP domain are used to execute computation tasks belonging to the same PP domain. In scenarios where the parallel strategy includes OP, the accelerator cards in the computing cluster are divided into multiple OP domains, and the accelerator cards within each OP domain are used to execute computation tasks belonging to the same DP domain.

[0060] The TP, DP, PP, and OP domains mentioned above can be collectively referred to as parallel domains. Training processes corresponding to different accelerator cards within the same parallel domain need to communicate to exchange data. Therefore, training processes corresponding to different accelerator cards within the same parallel domain need to establish communication. Thus, a group of training processes corresponding to a set of accelerator cards within a parallel domain can also be called a set communication group.

[0061] In addition, in some embodiments, in the scenario where the management node 10 manages each compute node through Kubernetes (an open-source container orchestration platform, hereinafter referred to as K8s), the management node 10 deploys one or more pods (containers) in each compute node. Each pod includes an agent process and some training processes in the corresponding compute node. In this way, the management node 10 schedules the training processes in the compute node 20 by scheduling the agent process in the pod.

[0062] Figure 3 is a schematic diagram of another computing cluster architecture provided in an embodiment of this application. As shown in Figure 3, each computing node 20 deploys a pod, which includes all training processes 221 and an agent process in the corresponding computing node 20. When the computing cluster trains AI models using PyTorch (an open-source machine learning library), the agent process can also be called a PyTorch Agent.

[0063] In addition, in order to implement the fault handling scheme provided in the embodiments of this application, as shown in Figure 2, the computing cluster is also configured with a control center module (controller) for all training processes 221 globally, and a processing module (processor) corresponding to each training process 221.

[0064] For any training process 221, the corresponding processing module is used to communicate with the control center module. For example, the processing module reports information to the control center module, receives commands from the control center module, and triggers the corresponding training process to execute commands. For instance, if the accelerator card 22 corresponding to training process 221 malfunctions, the processing module reports the malfunction to the control center module.

[0065] The control center module is responsible for the entire fault handling process, executing the fault handling method provided in this application embodiment. This process includes receiving information reported by the processing module, analyzing the scope of fault repair, formulating fault repair strategies, and issuing fault repair instructions to the processing module. Details will be explained in subsequent embodiments.

[0066] In addition, in the scenario shown in Figure 3, the control center module and processing module in Figure 2 can be deployed in various ways.

[0067] Figure 4 is a schematic diagram of a deployment method for a control center module and a processing module provided in an embodiment of this application. As shown in Figure 4, the pods deployed on the management node 10 include pod0, pod1 to podn. Pod0 includes an agent process and a process corresponding to the control center module. Each pod from pod1 to podn includes an agent process and multiple training processes, as well as the processes corresponding to the processing modules of these multiple training processes. The relationship between pod1 to podn and the compute nodes can be referred to Figure 3. Pod0 can be a pod deployed on the management node; optionally, pod0 can also be a pod deployed on the compute node. This embodiment of the application does not limit this.

[0068] As shown in Figure 4, both the control center module and the processing module run as independent processes in the pod.

[0069] Figure 5 is a schematic diagram of another deployment method of the control center module and processing module provided in an embodiment of this application. As shown in Figure 5, the pods deployed on the management node 10 include pod0, pod1 to podn. pod0 includes an agent process, and the agent process includes a thread corresponding to the control center module. Each pod from pod1 to podn includes one agent process and multiple training processes. The agent process of each pod includes a thread corresponding to the processing module corresponding to each of these training processes. The relationship between pod1 to podn and the compute node can be referred to Figure 3. pod0 can be a pod deployed on the management node; optionally, pod0 can also be a pod deployed on the compute node, and this embodiment of the application does not limit this.

[0070] As shown in Figure 5, both the control center module and the processing module run as independent threads in the pod's agent process.

[0071] It should be noted that Figures 4 and 5 illustrate two example deployment methods for the control center module and the processing module. In this embodiment, the control center module and the processing module can also be deployed in the computing cluster in other ways, which will not be illustrated here.

[0072] In addition, based on the architecture of the computing cluster shown in Figure 2, as shown in Figure 6, each training process 221 in the computing cluster also includes a proxy module, an interruption module, and a state repair module.

[0073] For any training process 221, the agent module is used to: capture faults occurring in the corresponding accelerator card, report the faults to the processing module, clean up the set communication group of the training process corresponding to the faulty accelerator card, rebuild the set communication group of the training process corresponding to the repaired accelerator card, and perform data rollback. Data rollback can be understood as: after the faulty accelerator card recovers, controlling which iteration all training processes should roll back to for continued training. The interrupt module is used to: interrupt the main thread of the running training process. The state repair module is used to: after the faulty accelerator card recovers, obtain the data required for the training process to continue training the AI ​​model, such as optimizer parameters.

[0074] The implementation methods of the functions of each module shown in Figure 6 can be referred to in the subsequent embodiments, and will not be described in detail here.

[0075] It should be noted that Figures 2-6 are used to illustrate the architecture of the computing cluster provided in the embodiments of this application. Optionally, the computing cluster provided in the embodiments of this application may also adopt other architectures, which will not be illustrated here.

[0076] The fault handling method provided in the embodiments of this application will be described below.

[0077] Figure 7 is a flowchart of a fault handling method provided in an embodiment of this application. The method shown in Figure 7 exemplifies the application of the control center module shown in Figure 2. As shown in Figure 7, the method includes the following steps.

[0078] Step 701: When the first accelerator card in the computing cluster fails, determine the first training state based on the current training state of the AI ​​model being trained by at least one accelerator card in the computing cluster.

[0079] In some embodiments, the control center module can detect a fault in the first accelerator card in the computing cluster by receiving a fault notification indicating that the first accelerator card has malfunctioned. For example, in the scenario shown in Figure 6, when the first accelerator card malfunctions, the first training process corresponding to the first accelerator card will experience an anomaly, such as a missing heartbeat. At this time, the agent module in the first training process will detect the anomaly and report it to the processing module corresponding to the first training process. Upon receiving the anomaly, the processing module generates a fault notification and reports it to the control center module.

[0080] In addition, in this embodiment, the fault notification may also carry a fault type, which indicates the type of fault occurring in the first accelerator card. This allows the control center module to determine the fault repair scope based on the fault type. The fault repair scope can be understood as: which accelerator cards in the computing cluster need to be repaired. For ease of understanding, the fault types are explained below.

[0081] The fault types provided in this application embodiment may include the following three examples:

[0082] First type of fault: Faults that can be recovered from without restarting the first accelerator card. For example, if a physical memory block in the first accelerator card fails, there is no need to restart the first accelerator card. Simply isolating the faulty physical memory block, i.e., disabling the faulty physical memory block, will restore the fault. Another example is when the first accelerator card experiences intermittent disconnection, i.e., the first accelerator card loses connection with the CPU. In this case, simply reseating the first accelerator card will restore the fault without restarting it.

[0083] The second type of failure: failures that can only be recovered by restarting (i.e., resetting) the first accelerator card. For example, if the operating system (OS) inside the first accelerator card crashes, restarting the first accelerator card can restore the normal operation of the OS in the first accelerator card.

[0084] In cases where the fault notification carries a fault type of either the first fault type or the second fault type, since fault recovery can be achieved without replacing the first accelerator card, the scope of fault repair can be the first accelerator card itself in these two fault scenarios.

[0085] Figure 8 is a schematic diagram of a scenario where the fault repair scope is under a first fault type or a second fault type according to an embodiment of this application. As shown in Figure 8, the first accelerator card is the accelerator card 1 corresponding to the training process 1 in pod1. When the fault of the first accelerator card belongs to the first fault type or the second fault type, since the first accelerator card and the first training process can still be restored, the fault repair scope is the first accelerator card itself.

[0086] Furthermore, if the fault notification carries a fault type of either the first fault type or the second fault type, the control center module, upon receiving the fault notification, can also control the first training process to exit if it detects that the first training process is still running. For example, the control center module can send a process exit command to the processing module corresponding to the first training process. Upon receiving the process exit command, the processing module forwards it to a proxy module within the first training process. This proxy module, in response to the process exit command, exits the main thread within the first training process, thereby exiting the first training process.

[0087] The third type of failure: Failures that can only be recovered by replacing the first accelerator card. For example, if some hardware in the first accelerator card experiences an unrecoverable failure, then the first accelerator card needs to be replaced.

[0088] When the fault notification indicates a third fault type, the fault repair scope can be all accelerator cards within the parallel domain to which the first accelerator card belongs. In this case, during fault recovery, a batch of accelerator cards from the same backup node in the computing cluster replaces all accelerator cards within the parallel domain to which the first accelerator card belongs. That is, in addition to replacing the first accelerator card, all other accelerator cards in that parallel domain also need to be replaced. An example parallel domain could be a TP domain or all accelerator cards corresponding to training processes within the same pod. The purpose of this is to ensure that accelerator cards within the same parallel domain still belong to the same node after fault recovery, thereby avoiding training performance degradation caused by cross-node communication within the parallel domain.

[0089] Figure 9 is a schematic diagram illustrating the fault repair scope under a third fault type according to an embodiment of this application. As shown in Figure 9, the first accelerator card is accelerator card 1 corresponding to training process 1 in pod2. When the fault of the first accelerator card belongs to the third fault type, since the first accelerator card itself cannot recover, it needs to be replaced. In this scenario, the fault repair scope is the accelerator cards corresponding to all training processes within pod2. As shown in Figure 9, during fault repair, the accelerator card corresponding to the training process of another spare pod3 needs to be used to replace the accelerator cards corresponding to all training processes within pod2.

[0090] Furthermore, if the fault notification carries a fault type of the third type, the control center module, upon receiving the fault notification, can also control the first training process to exit if it detects that the first training process is still running. Optionally, since all accelerator cards within the fault repair scope will be replaced, the control center module can also control the training processes corresponding to other accelerator cards within the fault repair scope, excluding the first accelerator card, to also exit. The implementation method for how the control center module controls the exit of a specific training process can be referred to the aforementioned embodiments, and will not be repeated here.

[0091] For any of the above fault types, when the control center module detects a fault in the first accelerator card, it determines the first training state based on the current training state of the AI ​​model being trained by at least one accelerator card in the computing cluster, so that step 702 can be executed subsequently based on the first training state.

[0092] To facilitate subsequent understanding, the current training state will be explained here first.

[0093] In some embodiments, for any accelerator card in the computing cluster, such as the fourth accelerator card, the fourth accelerator card can be either the first accelerator card that has failed or the second accelerator card that is functioning normally. The fourth training process corresponding to the fourth accelerator card can determine the current training state of the fourth accelerator card in training the AI ​​model, and return the current training state of the fourth accelerator card in training the AI ​​model to the control center module through the processing module corresponding to the fourth training process.

[0094] The fourth training process determines the current training state of the AI ​​model by the fourth accelerator card in the following ways: when the fourth accelerator card completes the optimizer update for the current iteration number, the fourth training process takes the next iteration number as the current training state; when the fourth accelerator card has not completed the optimizer update for the current iteration number, the fourth training process takes the current iteration number as the current training state.

[0095] For example, if the current iteration number is 900, and the fourth accelerator card has already completed the 900th optimizer update, the fourth training process will determine the current training state as the 901st iteration. Conversely, if the fourth accelerator card has not yet completed the 900th optimizer update, the fourth training process will determine the current training state as the 900th iteration.

[0096] Figure 10 is a schematic diagram of the computational flow of one iteration in the training of an AI model according to an embodiment of this application. As shown in Figure 10, after obtaining the sample data corresponding to the current iteration number, the value of the loss function is first determined by forward computation, and then backward computation is performed based on the value of the loss function to determine the parameter gradient values ​​of the AI ​​model. Then, the averaged parameter gradient value is determined by the ReduceScatter (RS) communication operator, and the local optimizer parameter values ​​are updated based on the averaged parameter gradient value. Subsequently, all accelerator cards can aggregate the total optimizer parameter values ​​through the AllGather (AG) communication operator, and the model parameters (such as model weights) are updated based on the total optimizer parameter values. At this point, one round of training for the current iteration number is completed. Then, the sample data (nextbatch) corresponding to the next iteration number is obtained, and the next round of training is performed according to the above process.

[0097] The RS function collects and sums the data from all accelerator cards, then divides the summed data according to the total number of accelerator cards and distributes the divided data to the corresponding accelerator cards. The AG communication operator collects data from all accelerator cards and distributes it to all accelerator cards.

[0098] As shown in Figure 10, when the fourth accelerator card has completed the optimizer update for the current iteration, it indicates that the card has completed the relevant calculations for the sample data corresponding to the current iteration and updated the local optimizer parameters based on the calculation results. Therefore, the next iteration can be taken as the current training state, indicating that the fourth accelerator card can start training directly from the next iteration when it continues training. Conversely, when the fourth accelerator card has not completed the optimizer update for the current iteration, it indicates that the card has not yet completed the relevant calculations for the sample data corresponding to the current iteration. Therefore, the current iteration can be taken as the current training state, indicating that the fourth accelerator card needs to retrain from the current iteration when it continues training.

[0099] Optionally, the current training state corresponding to the fourth accelerator card determined by the fourth training process can also be directly expressed as: whether the fourth accelerator card has completed the optimizer update for the current iteration number.

[0100] In this embodiment of the application, the control center module determines the first training state based on the current training state of the AI ​​model being trained by at least one accelerator card in the computing cluster. The implementation methods for this determination can be exemplified by the following two:

[0101] Implementation method a: Obtain the current training state of the AI ​​model trained by the first accelerator card from the first training process corresponding to the first accelerator card, and use the current training state of the AI ​​model trained by the first accelerator card as the first training state.

[0102] The applicable scenario for implementation method a is: when the first accelerator card fails, the first training process corresponding to the first accelerator card can still run. At this time, the control center module can obtain the current training state of the first accelerator card training the AI ​​model from the first training process and determine the current training state as the first training state.

[0103] In the scenario shown in Figure 6, the implementation method for the control center module to obtain the current training status of the AI ​​model trained by the first accelerator card from the first training process can be as follows: the agent module in the first training process reports the current training status of the AI ​​model trained by the first accelerator card to the processing module corresponding to the first training process. After receiving the current training status, the processing module continues to report the current training status to the control center module.

[0104] Implementation method b: Obtain the current training state of multiple second accelerator cards for training the AI ​​model from multiple second training processes; select one current training state from the multiple current training states of the AI ​​model for training the AI ​​model as the first training state.

[0105] The scenario applicable to implementation method b is as follows: when the first accelerator card malfunctions, the first training process corresponding to the first accelerator card has stopped running. At this time, the control center module needs to obtain the current training status of the AI ​​model training by multiple second accelerator cards from multiple second training processes, so as to determine the first training status based on the current training status of the AI ​​model training by these multiple second accelerator cards.

[0106] Optionally, in a scenario where the first accelerator card malfunctions but the first training process corresponding to the first accelerator card can still run, the first training state can also be determined through implementation method b.

[0107] For example, in a scenario where the current training state is represented by the number of iterations, the control center module can select a current training state as the first training state from the current training states of multiple second accelerator cards training the AI ​​model. This can be achieved by selecting the largest current training state from the current training states of multiple second accelerator cards training the AI ​​model as the first training state.

[0108] For example, there are multiple second accelerator cards, including accelerator cards 1-10. The current training states of the AI ​​model trained by the multiple second accelerator cards are as follows: the current training state of the AI ​​model trained by accelerator cards 1-5 is the 900th iteration, and the current training state of the AI ​​model trained by accelerator cards 6-10 is the 901st iteration. Therefore, the first training state is the 901st iteration.

[0109] It should be noted that in implementation b, the multiple second accelerator cards can be understood as all accelerator cards except the first accelerator card among the accelerator cards training the AI ​​model. Optionally, in a scenario where the fault occurring in the first accelerator card belongs to the third type of fault, the multiple second accelerator cards in implementation b can also be understood as all accelerator cards except those within the fault repair scope among the accelerator cards training the AI ​​model. This application embodiment does not limit this.

[0110] Alternatively, in some other embodiments, when the control center module detects a malfunction in the first accelerator card, it can also directly obtain the current training status of the corresponding accelerator card training the AI ​​model from all running training processes, and thus determine the first training status. This will not be described in detail here.

[0111] Step 702: According to the first training state, control the multiple second training processes corresponding to the multiple second accelerator cards in the computing cluster to lock the training process of the AI ​​model by the multiple second accelerator cards in the second training state.

[0112] In step 702, the multiple second accelerator cards can be understood as all accelerator cards used for training the AI ​​model, excluding the first accelerator card. Optionally, in a scenario where the fault in the first accelerator card belongs to the third type of fault, the multiple second accelerator cards in step 702 can also be understood as all accelerator cards used for training the AI ​​model, excluding those within the fault repair scope. This application embodiment does not limit this.

[0113] In addition, for any second accelerator card, locking the training process of the AI ​​model on the second accelerator card to the second training state can be understood as: after detecting that the current training state has reached the second training state, the second training process corresponding to the second accelerator card will no longer continue to carry out the next round of iterative training.

[0114] For example, in the scenario shown in Figure 6, for any second training process, after the proxy module in the second training process detects that the current training state has reached the second training state, the proxy module sends a training interruption command to the interrupt module. After receiving the training interruption command, the interrupt module controls the main thread in the second training process to stop running, thereby realizing that the next round of iterative training will not be performed.

[0115] For example, in a scenario where the training state is represented by the number of iterations, the first training state is the first iteration of the AI ​​model training, and the second training state is the second iteration of the AI ​​model training. For any second accelerator card, locking the training process of the AI ​​model on the second accelerator card to the second training state (i.e., locking it to the second iteration number) can be understood as follows: after the second training process corresponding to the second accelerator card completes the optimizer update for the previous iteration number of the second iteration number, it will not enter the next round of optimizer update process.

[0116] In this scenario, step 702 can be implemented in the following two ways, for example.

[0117] Implementation Method 1: The first iteration count is sent to multiple second training processes to instruct them to lock the training process of the AI ​​model using multiple second accelerator cards to the first training state. Specifically, when a third training process receives the first iteration count, and the third accelerator card corresponding to the third training process has completed the optimizer update for the first iteration count, the third training process locks the training process of the AI ​​model using the third accelerator card to the second iteration count and provides feedback on the second iteration count, which is different from the first iteration count. The second iteration count is then sent to the multiple second training processes to instruct them to lock the training process of the AI ​​model using multiple second accelerator cards to the second iteration count.

[0118] In implementation method one, there may be scenarios where the execution of "locking to the first iteration number" in some second training processes fails. In such scenarios, all second training processes can be controlled to relock the training process of multiple second accelerator cards on the AI ​​model to a new iteration number, i.e., the second iteration number.

[0119] For example, in the scenario shown in Figure 6, the control center module sends the first iteration count to the processing module corresponding to each second training process. For any second training process, such as the third training process, the processing module corresponding to the third training process then forwards the first iteration count to the proxy module within the third training process. After receiving the first iteration count, the proxy module first checks the iteration count corresponding to the most recent optimizer update of the third accelerator card corresponding to the third training process, and increments that iteration count by 1 to obtain the second iteration count. If the second iteration count is inconsistent with the first iteration count, it indicates that the current training process cannot lock onto the first iteration count. At this time, the proxy module returns the second iteration count to the processing module, which then returns the second iteration count to the control center module. After receiving the second iteration count, the control center module determines that one of the multiple second training processes has failed to lock onto the first iteration count. Therefore, it resends the second iteration count to the processing modules corresponding to the multiple training processes to instruct the multiple second training processes to relock the training process onto the second iteration count.

[0120] For example, if the first iteration count is 900, the agent module in the third training process, after receiving the first iteration count as 900, detects that the optimizer update operation in the 900th iteration process has been completed, which means the second iteration count is 901. Obviously, it is not possible to lock the training process at the 900th iteration. Therefore, the second iteration count needs to be returned to the control center module, which will then control all second training processes to relock to the 901st iteration.

[0121] Through the control functions of the aforementioned control center module, it is possible to lock the training process of the AI ​​model by the corresponding second accelerator card in all the second training processes to the same second iteration number, that is, to the same second training state.

[0122] Implementation Method 2: Send the first iteration count to multiple second training processes to instruct them to lock the training process of the AI ​​model on multiple second accelerator cards to the first training state; wherein, when multiple second training processes receive the first iteration count, and none of the multiple second training processes have completed the optimizer update in the first iteration count, the multiple second training processes lock the first iteration count, and the second iteration count is the same as the first iteration count.

[0123] In implementation method two, if none of the second training processes have completed the optimizer update in the first iteration after receiving the first iteration number, it indicates that all second training processes have successfully locked the training process of the AI ​​model on the second accelerator card to the first iteration number. In this scenario, it is not necessary to lock multiple times as in implementation method one to ensure the following result: all second training processes lock the training process of the AI ​​model on the corresponding second accelerator card to the same iteration number.

[0124] For example, in the scenario shown in Figure 6, the first iteration number is 901. When any agent module in the second training process receives the first iteration number as 901, it detects that the optimizer update operation in the 901st iteration process has not yet been completed. In this scenario, the training process of the AI ​​model by the second accelerator card can be locked at the 901st iteration after the optimizer update operation in the 900th iteration process is completed.

[0125] Step 703: After the fault of the first accelerator card is recovered, restart the training process corresponding to the first accelerator card after the fault is recovered, so that the first accelerator card after the fault is recovered returns to the second training state, and the first accelerator card and the second accelerator card continue to train the AI ​​model from the second training state.

[0126] In some embodiments, after the fault of the first accelerator card is recovered, the training process corresponding to the recovered first accelerator card is restarted to restore the recovered first accelerator card to the second training state. The implementation method is as follows: after restarting the training process corresponding to the recovered first accelerator card, the state information of the first accelerator card in the second training state is obtained, and the state information includes optimizer parameters; the state information of the first accelerator card in the second training state is sent to the training process corresponding to the recovered first accelerator card.

[0127] In the scenario where the second training state is the second iteration of training the AI ​​model, the optimizer parameters of the first accelerator card in the second training state can be understood as the optimizer parameters obtained by the first accelerator card after completing the optimizer update operation of the previous iteration before the second iteration. This allows the model parameters to be updated based on these optimizer parameters during subsequent training, and the training process for the second iteration can continue based on the updated model parameters.

[0128] For example, if the second iteration number is 901, then the optimizer parameters of the first accelerator card in the second training state can be understood as: the optimizer parameters obtained by the first accelerator card after completing the optimizer update operation of the 900th iteration.

[0129] In addition, in scenarios where the fault type of the first accelerator card is either the first fault type or the second fault type, since the fault can be recovered without replacing the first accelerator card, the training process corresponding to the first accelerator card after the fault recovery is restarted, that is, the first training process corresponding to the first accelerator card is restarted.

[0130] In this scenario, the method to restore the first accelerator card to the second training state after the fault is recovered can be as follows: after restarting the first training process, obtain the status information of the first accelerator card in the second training state, which includes optimizer parameters; and send the status information of the first accelerator card in the second training state to the first training process.

[0131] Furthermore, in scenarios where the first accelerator card experiences a third type of failure, since replacing the first accelerator card is necessary for recovery, the recovery process includes replacing all accelerator cards in the parallel domain to which the first accelerator card belongs with a batch of backup accelerator cards located on the same backup node in the computing cluster. In other words, the fault repair strategy is to replace all accelerator cards in the parallel domain to which the first accelerator card belongs with a batch of backup accelerator cards located on the same backup node in the computing cluster. At this point, all the replaced backup accelerator cards need to be restored to the second training state.

[0132] Accordingly, restarting the training process corresponding to the first accelerator card after the fault is recovered can be understood as restarting the training process corresponding to each backup accelerator card in this batch of backup accelerator cards.

[0133] In this scenario, the way to restore the first accelerator card to the second training state after the fault is recovered can be as follows: obtain the state information of each accelerator card in the parallel domain to which the first accelerator card belongs when it is in the second training state, and send the state information of each accelerator card in the parallel domain when it is in the second training state to the training process corresponding to the backup accelerator card.

[0134] Additionally, an example of restarting the training process corresponding to each backup accelerator card in a batch of backup accelerator cards includes: registering the training process corresponding to each backup accelerator card and initializing the training process corresponding to each backup accelerator card. If the restarting of the training process corresponding to each backup accelerator card in a batch of backup accelerator cards occurs after replacing the first faulty accelerator card, this process can be called cold backup.

[0135] Optionally, in this embodiment of the application, to further shorten the fault recovery time, the training process corresponding to each backup accelerator card can be pre-started. In this case, after replacing the faulty first accelerator card, the status information of each accelerator card in the parallel domain during the second training state can be directly sent to the training process corresponding to the backup accelerator card. This process can be called warm backup.

[0136] The following explains how to obtain the state information of the first accelerator card in the second training state. The implementation method for obtaining the state information of each accelerator card in the parallel domain to which the first accelerator card belongs during the second training state can also be found in the following content.

[0137] In this embodiment, the parallel strategy corresponding to the AI ​​model also includes optimizer parallelism. The following explanation uses two optimizer parallelism as an example. In this scenario, the accelerator cards in the computing cluster can be divided into two optimizer parallel domains. Each optimizer parallel domain maintains a complete optimizer. For any optimizer parallel domain, each accelerator card in that domain stores a portion of the optimizer's parameters. The accelerator cards in the two optimizer parallel domains correspond one-to-one, and the optimizer parameters stored by a certain accelerator card in one optimizer parallel domain are exactly the same as the optimizer parameters stored by the corresponding accelerator card in the other optimizer parallel domain.

[0138] In this scenario, considering that the probability of a pair of accelerator cards in the parallel domains of the two optimizers failing simultaneously is small, the implementation method for obtaining the state information of the first accelerator card in the second training state can be as follows: determine the first and second parallel domains of the optimizer, with the first accelerator card being one of the first parallel domains, and determine the target accelerator card in the second parallel domain that corresponds to the first accelerator card. The first accelerator card and the target accelerator card are used to store the same optimizer parameters, and the state information of the target accelerator card in the second training state is used as the state information of the first accelerator card in the second training state.

[0139] Figure 11 is a schematic diagram of an optimizer parallelism scenario provided by an embodiment of this application. As shown in Figure 11, the first optimizer parallel domain includes accelerator cards numbered 0 to 500, and the second optimizer parallel domain also includes accelerator cards numbered 0 to 500. Accelerator cards with the same serial number are used to store the same optimizer parameters. Assuming the first accelerator card is accelerator card number 0 in the first optimizer parallel domain, the state information of accelerator card number 0 in the second optimizer parallel domain during the second training state is used as the state information of the first accelerator card during the second training state.

[0140] For example, in the scenario shown in Figure 2, the control center module sends a data transmission command to the training process corresponding to accelerator card 0 in the parallel domain of the second optimizer. The training process corresponding to accelerator card 0 in the parallel domain of the second optimizer sends the status information of the second training state to the training process corresponding to accelerator card 0 in the parallel domain of the first optimizer after the fault is recovered, according to the data transmission command.

[0141] Optionally, in the embodiments of this application, the state information of the first accelerator card in the second training state can also be obtained by other means, and the embodiments of this application do not limit this.

[0142] In addition, in scenarios where the fault type is the first fault type or the second fault type, the way to enable the first accelerator card and the second accelerator card to continue AI model training from the second training state can be as follows: re-establish the communication connection in the set communication group to which the first training process corresponding to the first accelerator card belongs, control all accelerator cards (including the first accelerator card and all normal accelerator cards other than the first accelerator card) to obtain the sample data that needs to be trained according to the second training state, and continue training according to the obtained sample data.

[0143] In a scenario where the fault type is the third type, the implementation method for enabling the first and second accelerator cards to continue AI model training from the second training state can be as follows: For each backup accelerator card after replacement, re-establish the communication connection in the set communication group to which the training process corresponding to the backup accelerator card belongs, control all accelerator cards (including the replaced backup accelerator cards and the normal accelerator cards that have not been replaced) to obtain the sample data that needs to be trained according to the second training state, and continue training according to the obtained sample data.

[0144] The following example, using the architecture shown in Figure 2, illustrates steps 701 to 703.

[0145] Figure 12 is a schematic diagram of a fault handling process provided in an embodiment of this application. It should be noted that Figure 12 is for illustrative purposes only and does not constitute a limitation on steps 701 to 703 described above. As shown in Figure 12, the fault handling process includes the following steps.

[0146] Step 12.1: When the first accelerator card malfunctions, the first training process corresponding to the first accelerator card will experience an anomaly, such as a missing heartbeat. After the processing module corresponding to the first training process detects the anomaly, it will report a fault notification to the control center module. The control center module will detect the malfunction of the first accelerator card by receiving the fault notification.

[0147] Step 12.2: The control center module collects the current training status of all normal second accelerator cards training the AI ​​model through the processing module corresponding to the second training process. The interpretation of the current training status of any second accelerator card training the AI ​​model can be found in step 701 above, and will not be repeated here.

[0148] Step 12.3: The control center module issues a training lock command to the processing modules corresponding to all second training processes to control all second training processes to lock the training process of the AI ​​model on the corresponding second accelerator card to the second training state. For detailed implementation, please refer to step 702 above, which will not be repeated here.

[0149] Step 12.4: Consistency Check. The consistency check includes two aspects. First, it checks whether all second accelerator cards are locked in the same second training state during the training process of the AI ​​model. Second, it checks whether there is a target accelerator card in the computing cluster that stores the same optimizer parameters as the first accelerator card. If it exists, proceed to the next step; if it does not exist, pause the fault recovery process and report a notification that the fault cannot be handled.

[0150] Step 12.5: Send a training termination command to the processing modules corresponding to all second training processes to control all second training processes to stop sending operators to the corresponding accelerator cards. During this process, the proxy module in the second training process will throw a force stop exception, and the processing module will respond to the exception by pausing the main thread in the second training process.

[0151] It should be noted that in steps 12.5 and 12.3 above, after the second training process locks the training process of the AI ​​model on the corresponding second accelerator card to the second training state, it can directly stop sending operators to the corresponding accelerator card without relying on the control center module to send a training termination command.

[0152] Step 12.6: The control center module sends a data cleanup command to the processing modules corresponding to all second training processes to control all second training processes to clean up the negative residual operators on the corresponding accelerator cards.

[0153] Step 12.7: When the fault type is the third fault type, that is, the backup accelerator card is used to replace the first accelerator card to realize fault recovery. In this case, the control center module sends a training process registration instruction to the processing module corresponding to the replaced backup accelerator card to register and start the training process corresponding to the backup accelerator card.

[0154] Step 12.8: The control center module sends a repair command to the processing module corresponding to the replaced backup accelerator card. The processing module sends the repair command to the agent module in the corresponding training process. The agent module responds to the repair command by rebuilding the set communication group and receiving the status information of the backup accelerator card in the second training state.

[0155] At the same time, the control center module also sends a repair instruction to the processing module corresponding to the second accelerator card that has not been replaced. The processing module sends the repair instruction to the agent module in the corresponding training process. The agent module responds to the repair instruction by rebuilding the set communication group and obtaining the status information of the second accelerator card in the second training state from the local cache.

[0156] Step 12.9: Data Rollback. The control center module sends a data rollback command to the processing modules corresponding to all training processes after the fault recovery. This data rollback command carries the second training state, for example, if the second training state is the second iteration number, then the data rollback command carries the second iteration number. After receiving the data rollback command, any processing module continues to send the data rollback command to the agent module in the corresponding training process, and the agent module responds to the data rollback command to obtain the sample data corresponding to the second training state.

[0157] For example, if the second iteration number carried by the data rollback instruction is the 901st iteration, then the agent module obtains the sample data corresponding to the 901st iteration, that is, the 901st batch of sample data, and then continues training based on the 901st batch of sample data.

[0158] Step 12.10: Continue training. The control center module sends a continue training instruction to the processing modules corresponding to all training processes after the fault is recovered. After receiving the continue training instruction, any processing module will send the instruction to the proxy module in the corresponding training process. The proxy module will then start the main thread in the second training process so that the main thread can continue training based on the previously acquired status information and sample data of the second training state.

[0159] It should be noted that the flow shown by the dashed line in Figure 12 represents the response information returned to the control center module after the relevant module performs the operation in response to the instruction issued by the control center module. It will not be described in detail here.

[0160] In summary, in this embodiment of the application, when the first accelerator card in the computing cluster fails, it is not necessary to exit the training process corresponding to other normal accelerator cards. Instead, the training process corresponding to other normal accelerator cards is locked in the second training state. After the failure of the first accelerator card is recovered, it is not necessary to restart the training process corresponding to all other normal accelerator cards. Therefore, the time required from the occurrence of the failure to retraining the AI ​​model can be shortened.

[0161] Furthermore, after the failure of the first accelerator card is resolved, the control system resumes AI model training from the second training state locked after the failure, along with other normal second accelerator cards. Instead of retraining the AI ​​model from the most recently saved checkpoint, this avoids repeatedly executing certain training iterations, thus saving computational resources.

[0162] Figure 13 is a comparative diagram of the training process before and after fault recovery provided in an embodiment of this application. As shown in Figure 13, assuming that the accelerator card corresponding to a certain training process fails, and assuming that the parallel domain to which the failed accelerator card belongs is all the accelerator cards in the computing node where the failed accelerator card is located, after replacing all the accelerator cards in the computing node to which the failed accelerator card belongs with a batch of backup accelerator cards from the backup node, the training process corresponding to this batch of backup accelerator cards can be restarted. The normal accelerator cards on other computing nodes do not exit before and after the fault recovery, thus shortening the time required from the occurrence of the fault to the retraining of the AI ​​model.

[0163] Figure 14 is a schematic diagram of the architecture of a fault handling system provided in an embodiment of this application. As shown in Figure 14, the fault handling system 1400 includes the following modules.

[0164] The first determining module 1401 is used to determine the first training state based on the current training state of the AI ​​model being trained by at least one accelerator card in the computing cluster when the first accelerator card in the computing cluster fails; the implementation method can refer to step 701 in the embodiment of Figure 7.

[0165] The locking module 1402 is used to control the training process of the AI ​​model by the multiple second accelerator cards in the computing cluster to lock the training process of the AI ​​model by the multiple second accelerator cards to the second training state according to the first training state; the implementation method can refer to step 702 in the embodiment of Figure 7.

[0166] The recovery module 1403 is used to restart the training process corresponding to the recovered first accelerator card after the fault in the first accelerator card is resolved, so that the recovered first accelerator card returns to the second training state, and the first and second accelerator cards continue to train the AI ​​model from the second training state. The implementation method can refer to step 703 in the embodiment of Figure 7.

[0167] In some embodiments, the first determining module is configured to:

[0168] Obtain the current training state of the AI ​​model trained by the first accelerator card from the first training process corresponding to the first accelerator card, and use the current training state of the AI ​​model trained by the first accelerator card as the first training state.

[0169] In some embodiments, the first determining module is configured to:

[0170] Obtain the current training status of the AI ​​model using multiple second accelerator cards from multiple second training processes;

[0171] Select one current training state from the current training states of multiple second accelerator cards training the AI ​​model as the first training state.

[0172] In some embodiments, the first training state is the first iteration number of AI model training, and the second training state is the second iteration number of AI model training;

[0173] The locking module is used for:

[0174] The first iteration number is sent to multiple second training processes to instruct them to lock the training process of the AI ​​model on the multiple second accelerator cards to the first training state; wherein, when the third training process among the multiple second training processes receives the first iteration number, and the third accelerator card corresponding to the third training process has completed the optimizer update for the first iteration number, the third training process locks the training process of the AI ​​model on the third accelerator card to the second iteration number and feeds back the second iteration number, which is different from the first iteration number;

[0175] Send the second iteration number to multiple second training processes to instruct the multiple second training processes to lock the training process of the AI ​​model on multiple second accelerator cards at the second iteration number.

[0176] In some embodiments, the first training state is the first iteration number of AI model training, and the second training state is the second iteration number of AI model training;

[0177] The locking module is used for:

[0178] The first iteration number is sent to multiple second training processes to instruct the multiple second training processes to lock the training process of the AI ​​model on multiple second accelerator cards in the first training state;

[0179] Specifically, when multiple second training processes receive the first iteration count, and none of the multiple second training processes have completed the optimizer update in the first iteration count, the multiple second training processes lock the first iteration count, and the second iteration count is the same as the first iteration count.

[0180] In some embodiments, the system further includes:

[0181] The second determining module is used for the fourth accelerator card in the computing cluster. When the fourth accelerator card completes the optimizer update for the current iteration number, the fourth training process corresponding to the fourth accelerator card will take the next iteration number as the corresponding current training state.

[0182] The third determination module is used when the fourth accelerator card has not completed the optimizer update for the current iteration number. In this case, the fourth training process will use the current iteration number as the corresponding current training state.

[0183] In some embodiments, the recovery module is used for:

[0184] After restarting the training process corresponding to the first accelerator card after the fault recovery, obtain the status information of the first accelerator card in the second training state. The status information includes the optimizer parameters.

[0185] The status information of the first accelerator card in the second training state is sent to the training process corresponding to the first accelerator card after the fault is recovered.

[0186] In some embodiments, fault recovery of the first accelerator card includes: replacing all accelerator cards in the parallel domain to which the first accelerator card belongs with a batch of backup accelerator cards located in the same backup node in the computing cluster;

[0187] The recovery module is used for:

[0188] After restarting the training process corresponding to each backup accelerator card in a batch of backup accelerator cards, obtain the state information of each accelerator card in the second training state in the parallel domain;

[0189] The state information of each accelerator card in the parallel domain during the second training state is sent to the training process corresponding to the backup accelerator card.

[0190] It should be noted that the fault handling system provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the equipment can be divided into different functional modules to complete all or part of the functions described above. In addition, the fault handling system and fault handling method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0191] Figure 15 is a schematic diagram of a computing device provided in an embodiment of this application. The aforementioned computing node, management node, and backup node can all be implemented using the computing device shown in Figure 15.

[0192] As shown in Figure 15, the computing device includes at least one processor 1501, a communication bus 1502, a memory 1503, and at least one communication interface 1504.

[0193] The processor 1501 may be a general-purpose central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application.

[0194] The communication bus 1502 may include a path for transmitting information between the aforementioned components.

[0195] Memory 1503 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), magnetic disks or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory 1503 may exist independently and be connected to processor 1501 via communication bus 1502. Memory 1503 may also be integrated with processor 1501.

[0196] The memory 1503 stores program code for executing the scheme of this application, and its execution is controlled by the processor 1501. The processor 1501 executes the program code stored in the memory 1503. The program code may include one or more software modules. Embodiments of this application can determine data for application development using the processor 1501 and one or more software modules in the program code in the memory 1503.

[0197] Communication interface 1504 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.

[0198] In a specific implementation, as one embodiment, the computing device may include multiple processors, such as processors 1501 and 1505 shown in FIG. 15. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. Here, a processor may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0199] In a specific implementation, as one embodiment, the computing device may further include an output device 1506 and an input device 1507. The output device 1506 communicates with the processor 1501 and can display information in various ways. For example, the output device 1506 may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device 1507 communicates with the processor 1501 and can receive user input in various ways. For example, the input device 1507 may be a mouse, keyboard, touchscreen device, or sensing device, etc.

[0200] The aforementioned computing device can be a general-purpose computing device or a dedicated computing device. In specific implementations, the computing device can be a desktop computer, a portable computer, a network server, a handheld digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. This application does not limit the type of computing device.

[0201] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0202] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0203] The above content is not intended to limit the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of the embodiments of this application.

Claims

1. A fault handling method, characterized in that, The method includes: When the first accelerator card in the computing cluster fails, the first training state is determined based on the current training state of the AI ​​model being trained by at least one accelerator card in the computing cluster. Based on the first training state, the multiple second training processes corresponding to the multiple second accelerator cards in the computing cluster are controlled to lock the training process of the AI ​​model by the multiple second accelerator cards in the second training state. After the fault in the first accelerator card is recovered, the training process corresponding to the recovered first accelerator card is restarted, so that the recovered first accelerator card is restored to the second training state, and the first accelerator card and the second accelerator card continue to train the AI ​​model from the second training state.

2. The method as described in claim 1, characterized in that, Determining the first training state based on the current training state of the AI ​​model being trained by at least one accelerator card in the computing cluster includes: While the first training process corresponding to the first accelerator card is still running, the current training state of the AI ​​model trained by the first accelerator card is obtained from the first training process, and the current training state of the AI ​​model trained by the first accelerator card is used as the first training state, and the first training process is exited.

3. The method as described in claim 1, characterized in that, Determining the first training state based on the current training state of the AI ​​model being trained by at least one accelerator card in the computing cluster includes: When the first training process corresponding to the first accelerator card is not running, the current training status of the AI ​​model being trained by the multiple second accelerator cards is obtained from the multiple second training processes. From the current training states of the AI ​​model trained by the plurality of second accelerator cards, select one current training state as the first training state.

4. The method according to any one of claims 1-3, characterized in that, The first training state is the first iteration number of the AI ​​model training, and the second training state is the second iteration number of the AI ​​model training; The step of controlling multiple second training processes corresponding to multiple second accelerator cards in the computing cluster to lock the training process of the AI ​​model by the multiple second accelerator cards in the second training state according to the first training state includes: The first iteration number is sent to the plurality of second training processes to instruct the plurality of second training processes to lock the training process of the AI ​​model by the plurality of second accelerator cards in the first training state; wherein, when the third training process among the plurality of second training processes receives the first iteration number, and the third accelerator card corresponding to the third training process has completed the optimizer update for the first iteration number, the third training process locks the training process of the AI ​​model by the third accelerator card in the second iteration number, and feeds back the second iteration number, which is different from the first iteration number; The second iteration number is sent to the plurality of second training processes to instruct the plurality of second training processes to lock the training process of the AI ​​model on the plurality of second accelerator cards at the second iteration number.

5. The method according to any one of claims 1-3, characterized in that, The first training state is the first iteration number of the AI ​​model training, and the second training state is the second iteration number of the AI ​​model training; The step of controlling multiple second training processes corresponding to multiple second accelerator cards in the computing cluster to lock the training process of the AI ​​model by the multiple second accelerator cards in the second training state according to the first training state includes: The first iteration number is sent to the plurality of second training processes to instruct the plurality of second training processes to lock the training process of the AI ​​model by the plurality of second accelerator cards in the first training state; When the plurality of second training processes receive the first iteration number, and none of the plurality of second training processes have completed the optimizer update in the first iteration number, the plurality of second training processes lock the first iteration number, and the second iteration number is the same as the first iteration number.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: For the fourth accelerator card in the computing cluster, when the fourth accelerator card completes the optimizer update for the current iteration number, the fourth training process corresponding to the fourth accelerator card will take the next iteration number as the corresponding current training state. If the fourth accelerator card has not completed the optimizer update for the current iteration number, the fourth training process will use the current iteration number as the corresponding current training state.

7. The method according to any one of claims 1-6, characterized in that, The step of restarting the training process corresponding to the recovered first accelerator card after the fault in the first accelerator card is resolved, so that the recovered first accelerator card returns to the second training state, includes: After restarting the training process corresponding to the first accelerator card after the fault recovery, obtain the status information of the first accelerator card in the second training state, the status information including optimizer parameters; The status information of the first accelerator card in the second training state is sent to the training process corresponding to the first accelerator card after the fault is recovered.

8. The method as described in claim 7, characterized in that, The fault recovery of the first accelerator card includes: replacing all accelerator cards in the parallel domain to which the first accelerator card belongs with a batch of backup accelerator cards located in the same backup node in the computing cluster; After restarting the training process corresponding to the first accelerator card after the fault recovery, the step of obtaining the status information of the first accelerator card in the second training state includes: After restarting the training process corresponding to each backup accelerator card in the batch of backup accelerator cards, obtain the state information of each accelerator card in the parallel domain in the second training state; The step of sending the status information of the first accelerator card in the second training state to the training process corresponding to the first accelerator card after the fault is recovered includes: The state information of each accelerator card in the parallel domain during the second training state is sent to the training process corresponding to the backup accelerator card.

9. A fault handling system, characterized in that, The system includes: The first determining module is used to determine the first training state based on the current training state of at least one accelerator card in the computing cluster training an artificial intelligence AI model when the first accelerator card in the computing cluster fails. The locking module is used to control multiple second training processes corresponding to multiple second accelerator cards in the computing cluster to lock the training process of the AI ​​model by the multiple second accelerator cards to the second training state according to the first training state. The recovery module is used to restart the training process corresponding to the first accelerator card after the fault of the first accelerator card is recovered, so that the first accelerator card after the fault is recovered is restored to the second training state, and the first accelerator card and the second accelerator card continue to perform AI model training from the second training state.

10. The system as described in claim 9, characterized in that, The first determining module is used for: While the first training process corresponding to the first accelerator card is still running, the current training state of the AI ​​model trained by the first accelerator card is obtained from the first training process, and the current training state of the AI ​​model trained by the first accelerator card is used as the first training state, and the first training process is exited.

11. The system as described in claim 9, characterized in that, The first determining module is used for: When the first training process corresponding to the first accelerator card is not running, the current training status of the AI ​​model being trained by the multiple second accelerator cards is obtained from the multiple second training processes. From the current training states of the AI ​​model trained by the plurality of second accelerator cards, select one current training state as the first training state.

12. The system of any of claims 9-11, wherein, The first training state is the first iteration number of the AI ​​model training, and the second training state is the second iteration number of the AI ​​model training; The locking module is used for: The first iteration number is sent to the plurality of second training processes to instruct the plurality of second training processes to lock the training process of the AI ​​model by the plurality of second accelerator cards in the first training state; wherein, when the third training process among the plurality of second training processes receives the first iteration number, and the third accelerator card corresponding to the third training process has completed the optimizer update for the first iteration number, the third training process locks the training process of the AI ​​model by the third accelerator card in the second iteration number, and feeds back the second iteration number, which is different from the first iteration number; The second iteration number is sent to the plurality of second training processes to instruct the plurality of second training processes to lock the training process of the AI ​​model on the plurality of second accelerator cards at the second iteration number.

13. The system as described in any one of claims 9-11, characterized in that, The first training state is the first iteration number of the AI ​​model training, and the second training state is the second iteration number of the AI ​​model training; The locking module is used for: The first iteration number is sent to the plurality of second training processes to instruct the plurality of second training processes to lock the training process of the AI ​​model by the plurality of second accelerator cards in the first training state; When the plurality of second training processes receive the first iteration number, and none of the plurality of second training processes have completed the optimizer update in the first iteration number, the plurality of second training processes lock the first iteration number, and the second iteration number is the same as the first iteration number.

14. The system as described in any one of claims 9-13, characterized in that, The system also includes: The second determining module is used for the fourth accelerator card in the computing cluster. When the fourth accelerator card completes the optimizer update for the current iteration number, the fourth training process corresponding to the fourth accelerator card takes the next iteration number as the corresponding current training state. The third determining module is used to determine the current training state of the fourth training process when the fourth accelerator card has not completed the optimizer update for the current iteration number.

15. The system as described in any one of claims 9-14, characterized in that, The recovery module is used for: After restarting the training process corresponding to the first accelerator card after the fault recovery, obtain the status information of the first accelerator card in the second training state, the status information including optimizer parameters; The status information of the first accelerator card in the second training state is sent to the training process corresponding to the first accelerator card after the fault is recovered.

16. The system as described in claim 15, characterized in that, The fault recovery of the first accelerator card includes: replacing all accelerator cards in the parallel domain to which the first accelerator card belongs with a batch of backup accelerator cards located in the same backup node in the computing cluster; The recovery module is used for: After restarting the training process corresponding to each backup accelerator card in the batch of backup accelerator cards, obtain the state information of each accelerator card in the parallel domain in the second training state; The state information of each accelerator card in the parallel domain during the second training state is sent to the training process corresponding to the backup accelerator card.

17. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing cluster, the computing cluster performs the method as described in any one of claims 1-8.

18. A computer-readable storage medium, characterized in that, Includes computer program instructions that, when executed by a computing cluster, cause the computing cluster to perform the method as described in any one of claims 1-8.

Images