Model training failure identification method and electronic device

Abstract

The present application relates to the technical field of communications, and provides a model training failure identification method and an electronic device, capable of improving the efficiency and accuracy of model training stuck failure identification. A processor from at least one server is used for model training. The method comprises: acquiring first information of each of a plurality of processors; for each processor, when N pieces of first information acquired for N consecutive times are consistent, acquiring second information of each processor; and when each of a plurality of pieces of second information meets a preset condition, determining that a stuck failure occurs in a model training process. The second information comprises at least one indicator used for indicating the performance of the processor, N is a positive integer greater than 1, and the first information is used for indicating the communication state of the processor.

Description

A model training method for fault identification and an electronic device

[0001] This application claims priority to Chinese Patent Application No. 202510008093.0, filed on January 2, 2025, entitled "A Model Training Fault Identification Method and Electronic Equipment", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of communication technology, and in particular to a model training fault identification method and electronic device. Background Technology

[0003] Artificial intelligence has now entered the model era, and the demand for computing power for models is increasing day by day. Computing power has moved from single machines to clusters. As the scale of the cluster expands, the number of cases where failures cause model training interruptions also increases. It is necessary to detect deadlocks in a timely manner during the training process so as to quickly find the cause of the deadlock in order to resume training.

[0004] Currently, timeouts are typically set for deadlock detection. For example, if the timeout is set to 30 minutes, during training, each process is usually considered dead only after 30 minutes of timeout. This means that when a deadlock occurs, another 30 minutes must be waited before the cause of the deadlock can be determined, increasing the ineffective training time of the model. Summary of the Invention

[0005] This application provides a model training fault identification method and electronic device that can quickly identify model training freeze faults without waiting for timeout, thereby improving the efficiency of identifying model training freeze faults.

[0006] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0007] Firstly, a method for identifying model training faults is provided. The model training uses processors from at least one server. In this method, firstly, first information of each processor among multiple processors is obtained. Then, for each processor, if the N first information obtained in N consecutive times are consistent, second information of each processor is obtained. Finally, if each of the multiple second information satisfies a preset condition, it is determined that a deadlock fault exists in the model training process.

[0008] The first piece of information is used to indicate the communication status of the processor.

[0009] The second information includes at least one metric used to indicate processor performance; N is a positive integer greater than 1.

[0010] In the above method, by comparing the first information of the processor obtained multiple times, it is determined that the model training may be stuck. Then, by using the second information, it is further determined that the model training has a dead fault. There is no need to wait for the timeout period, which improves the efficiency and accuracy of determining the model training dead fault.

[0011] In one possible implementation of the first aspect, a first prompt message is displayed to indicate a deadlock failure, so as to alert the user that a deadlock failure has occurred during model training.

[0012] In one possible implementation of the first aspect, when obtaining first information for each of the plurality of processors, a first request is sent to each of at least one server, and first information is received from the server. The first request instructs the server to obtain first information for each of the at least one processor provided by the server. Therefore, by interacting with the server, more accurate first information can be obtained.

[0013] In one possible implementation of the first aspect, a first page is displayed, comprising multiple controls, each indicating a corresponding processor. In response to a user's selection of a control on the first page, multiple processors used in the model training process are determined. These multiple processors may originate from the same server or from different servers. Alternatively, the processors can be configured by the user to meet the needs of different model training methods.

[0014] In one possible implementation of the first aspect, the second information satisfying the preset conditions includes any one of the following: at least one indicator included in the second information is less than or equal to a preset threshold corresponding to the indicator; or, at least one indicator included in the second information is less than or equal to a preset threshold corresponding to the indicator within a preset time range. It is evident that appropriate preset conditions can be selected based on the requirements for identifying stuck faults during model training, thereby improving the applicability of identifying stuck obstacles.

[0015] In one possible implementation of the first aspect, the metric is processor utilization or the amount of data input / output to the processor, so as to further determine whether the model training is stuck based on the metric and improve the accuracy of identifying the stuck fault.

[0016] In one possible implementation of the first aspect, after determining that a deadlock occurs during model training, the type of the deadlock is obtained and displayed to inform the user of the specific reason for the model training deadlock, which helps to quickly resolve the deadlock problem and resume model training.

[0017] In one possible implementation of the first aspect, when obtaining the fault type of a stuck fault, at least one abnormal processor among multiple processors is identified, and the fault type of the stuck fault is determined based on the abnormal processor. It is evident that the stuck fault type can be further determined based on the initially identified abnormal processor, thereby improving the accuracy of determining the stuck fault type.

[0018] In one possible implementation of the first aspect, each of the multiple processors trains the model using an operator, and the abnormal processor is the processor whose operator timeout occurs among the multiple processors. Therefore, the abnormal processor can be identified by the operator timeout.

[0019] In one possible implementation of the first aspect, when determining at least one abnormal processor among multiple processors, multiple communication domains are determined, each of the multiple communication domains representing a communication relationship composed of multiple processors, wherein the multiple processors originate from the same server or different servers. For each communication domain, first information of each of the multiple processors in the communication domain is obtained, and the abnormal processor among the multiple processors is determined based on the first information of each of the multiple processors. Alternatively, the abnormal processor can be determined using the first information of each processor in the communication domain, thus determining the abnormal processor from multiple perspectives and improving the accuracy of abnormal processor determination.

[0020] In one possible implementation of the first aspect, when there is only one exception handler, to determine the type of the stuck fault based on the exception handler, firstly, at least one communication domain where the exception handler is located is obtained. Then, multi-layer detection is performed on the at least one communication domain to obtain the detection results at each layer. The multi-layer detection includes network layer detection, storage layer detection, and processor layer detection. Finally, if the detection result for any layer indicates that there is a processor with an exception in at least one communication domain, the fault type of the processor with the exception is determined as the type of the stuck fault. Therefore, by performing multi-layer detection on at least one communication domain where the exception handler is located, the accuracy of determining the cause of the stuck fault can be improved.

[0021] In one possible implementation of the first aspect, when there are multiple exception handlers, to determine the type of the stuck fault based on the exception handlers, firstly, processor-level detection is performed on each of the multiple exception handlers. Then, if no exception is found in the processor-level detection of each exception handler, network-level detection is performed on each exception handler. If a processor with an exception is detected in multiple network-level detections, the fault type of the stuck fault is determined to be a network fault. Therefore, when multiple exception handlers are identified, performing processor-level and network-level detection on each exception handler further improves the efficiency of determining the cause of the stuck fault.

[0022] In one possible implementation of the first aspect, network-level detection includes detection of switching devices connected to the processor; the switching devices are used for communication between processors; storage-level detection includes detection of storage devices corresponding to model training; the storage devices are used to store data used for model training and data generated during model training; processor-level detection includes detection of at least one of the following: processor port, processor hardware, and the server where the processor resides. It is evident that determining the cause of a freeze failure considers multiple factors, including switching devices, storage devices, processor ports, processor hardware, and the server where the processor resides, further improving the accuracy of determining the cause of the freeze failure.

[0023] In one possible implementation of the first aspect, the type of training freeze includes at least one of the following: a fault in the switching device connected to the processor, a fault in the storage device corresponding to model training, a processor port fault, a processor hardware fault, or a fault in the server where the processor resides. It is evident that by considering multiple causes of training freezes, the accuracy of determining the cause of the freeze is further improved.

[0024] In a second aspect, an electronic device is provided, the electronic device including a memory and one or more processors; the memory is coupled to the processors; wherein the memory stores computer program code, the computer program code including computer instructions, and when the computer instructions are executed by the processor, the electronic device performs the model training fault identification method as described in the first aspect and any implementation thereof.

[0025] Thirdly, a computer-readable storage medium is provided, including computer instructions that, when executed on an electronic device, cause the electronic device to perform the model training fault identification method as described in the first aspect and any implementation thereof.

[0026] Fourthly, a computer program product is provided that, when the computer program product is run on an electronic device, causes the electronic device to execute the model training fault identification method as described in the first aspect and any of its implementations.

[0027] The beneficial effects that the electronic device provided in the second aspect, the computer-readable storage medium provided in the third aspect, and the computer program product provided in the fourth aspect can achieve are similar to the beneficial effects that can be achieved in the first aspect and any of its implementations, and will not be repeated here. Attached Figure Description

[0028] Figure 1 shows one of the schematic diagrams of a model training fault identification system provided in an embodiment of this application;

[0029] Figure 2 shows a schematic diagram of a server used for model training according to an embodiment of this application;

[0030] Figure 3 shows a schematic diagram of determining multiple NPUs used in the model training process according to an embodiment of this application;

[0031] Figure 4 shows a flowchart of a model training fault identification method provided in an embodiment of this application;

[0032] Figure 5 shows a schematic diagram of a switch control provided in an embodiment of this application;

[0033] Figure 6 shows one of the schematic diagrams of a prompt message provided in an embodiment of this application;

[0034] Figure 7 shows a schematic diagram of a model training fault identification method provided in an embodiment of this application;

[0035] Figure 8 shows a second schematic diagram of a prompt message provided in an embodiment of this application;

[0036] Figure 9 shows a schematic diagram illustrating a method for determining the type of model training deadlock provided in an embodiment of this application;

[0037] Figure 10 shows a second schematic diagram of a model training fault identification system provided in an embodiment of this application;

[0038] Figure 11 shows a schematic diagram of the hardware structure of another electronic device provided in an embodiment of this application. Detailed Implementation

[0039] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of this application, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can represent A or B. "And / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple. Furthermore, to facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" are not necessarily different. Meanwhile, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is being used as an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present related concepts in a concrete manner for ease of understanding.

[0040] Furthermore, the business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the emergence of new business scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0041] Various faults can occur during model training, causing training jobs to be interrupted and terminated (i.e., training job freeze). Training job freeze detection is needed to identify these faults, and training job freeze delimitation is needed to determine the cause of the freeze. Training job freeze detection and training job freeze delimitation are key technologies for model training fault diagnosis. Training job freeze detection involves real-time monitoring of the training job status and timely detection of situations where the training job cannot continue. Training job freeze delimitation involves quickly identifying the faulty node causing the freeze, isolating the faulty node, and resuming subsequent training.

[0042] For example, training job deadlock detection currently detects whether a training job is dead by monitoring its process status and resource utilization. A process can be started to periodically monitor changes in process status and resource utilization. Process status: Whenever there is a change in the input / output (I / O) data volume of any process in the training job, the next detection cycle begins. If the I / O data volume of all processes in the training job remains unchanged across multiple detection cycles, the resource utilization detection phase begins. Resource utilization: If I / O remains unchanged, the graphics processing unit (GPU) utilization rate is collected over a certain period. The variance and median of the GPU utilization rate during this period are used to determine if resource utilization has changed. If there is no change, the training job is determined to be dead. However, this method of detecting I / O data volume cannot determine whether the training job is slow or dead, which is prone to misjudgment and affects the accuracy of identifying training job deadlock faults.

[0043] For example, training job dead detection can also detect whether a training job is dead by setting a timeout. For instance, if the preset timeout is 30 minutes, it is usually only after the processors corresponding to the server used for model training have timed out for 30 minutes that a training job dead fault can be determined, which increases the model training time.

[0044] For example, in training job deadlock identification, after a training job gets stuck, fault diagnosis can usually be performed within the communication domain where the timed-out GPU is located to find the faulty GPU, isolate it, restart the training job, and restore it from the previous save point checkpoint. However, due to the propagation of faults, this method means that the identified faulty GPU may not be the actual cause of the fault.

[0045] In view of this, embodiments of this application provide a model training fault identification method, which compares the communication data on each processor used for model training obtained multiple times to sense whether the training may be stuck. If the training may be stuck, the metrics of each processor are queried, and the training is further determined based on the metrics of each processor.

[0046] Based on the above, this application provides a model training fault identification method. The model training uses processors from at least one server. First, the electronic device obtains first information from each of the multiple processors. Then, for each processor, if the N first information obtained by the electronic device in N consecutive transactions are consistent, the electronic device obtains second information from each processor. Finally, if each of the multiple second information satisfies a preset condition, it is determined that a deadlock fault exists during the model training process.

[0047] For example, the processor used for model training can be a GPU or a neural processing unit (NPU).

[0048] For example, the first information is used to indicate the communication status of the processor. For instance, the first information may be information about a communication operator executed on the processor, which is used for data transmission and synchronization during model training.

[0049] For example, the second information includes at least one metric for indicating processor performance; N is a positive integer greater than 1. The metric can be processor utilization or the amount of processor I / O data.

[0050] In the above method, by acquiring the first information multiple times, it is determined whether the communication status of the processor has changed, thereby determining whether the training may be stuck. If a stuck situation is possible, the processor indicators included in the second information are used to further determine whether the training is actually stuck. The stuck fault can be determined without waiting for the processor's timeout, which not only improves the efficiency of determining the stuck fault, but also improves the accuracy of determining the stuck fault.

[0051] The model training fault identification method provided in this application embodiment can be applied to a model training fault identification system. In some embodiments, the system includes an electronic device and at least one server (as shown in Figure 1, server 1 to server n). Each server includes multiple processors (as shown in Figure 1, server 1 includes processors 1 to m, server n includes processors 1 to q, where n, m, and q are all positive integers, and n, m, and q can be equal or unequal). Each processor is used to acquire first information and second information, and send the first information and second information to the electronic device. The electronic device acquires and stores the first information and second information. For each processor, it is determined whether the first information of the processor in the last N times is consistent. If the first information of each processor in the last N times is consistent, it is determined whether the second information of each processor meets the preset conditions. If the second information of each processor meets the preset conditions, it is determined that there is a deadlock fault in the model training process.

[0052] The aforementioned electronic devices can be mobile phones, tablets, handheld computers, laptops, personal computers (PCs), ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and other terminal devices.

[0053] The following uses a PC as an example and an NPU as a processor to illustrate the model training fault identification method provided in the embodiments of this application.

[0054] Model training can utilize NPUs from at least one server, each server typically containing multiple NPUs. Alternatively, model training can use containers from at least one server, each container configured with multiple NPUs on its respective server. For example, as shown in Figure 2, there are 8 servers: Server 1, Server 2, ..., Server 8, each server containing 8 NPUs: NPU1, NPU2, ..., NPU8, for a total of 64 NPUs. During model training, after each NPU completes its computational task, they communicate according to communication domains. Each communication domain consists of multiple NPUs, which can originate from the same server or different servers. Communication domains typically include tensor parallelism (TP) communication domains, pipeline parallelism (PP) communication domains, etc. Users can configure the NPUs included in each communication domain through a second device, which can be a PC or other devices. When configuring the NPUs included in each communication domain through other devices, the user also needs to synchronize the information of the NPUs included in each communication domain on the PC so that the PC understands the NPUs included in each communication domain. For example, the TP communication domain can be the communication between NPU1-NPU8 within the server; the PP communication domain can be the communication between NPU1 in server 1-server 8, etc.

[0055] Before executing the model training fault identification method on a PC, it is necessary to obtain the NPU of at least one server used for model training.

[0056] In some embodiments, the NPU of at least one server used for model training can be either the default or a user-configured one.

[0057] For example, if the NPU of at least one server used for model training is the default, the PC can directly obtain the NPU of at least one server used for model training from pre-configured information.

[0058] For example, when the NPU of at least one server used for model training is user-configured, as shown in Figure 3, the user can select a control to indicate the corresponding NPU from among the multiple controls included in the first page displayed on the PC. For example, the user selects NPU2 in server 1 and NPU1 in server 2 in Figure 3. In response to the user's selection operation on the control on the first page, the PC determines the multiple NPUs used in the model training process; wherein, the multiple NPUs come from the same server or from different servers.

[0059] After the PC obtains the NPU of at least one server used for model training, as shown in Figure 4, the above-mentioned model training fault identification method may include the following steps S401-S403.

[0060] S401, PC obtains the first information of each NPU among multiple NPUs.

[0061] The first piece of information is used to indicate the communication status of the NPU.

[0062] For example, the first information can be NPU communication data. For instance, the NPU's compute architecture for neural networks (CANN) is used for neural network computation during model training. The operation of CANN on the NPU involves multiple events, which together constitute the complete process of CANN executing model training on the NPU. The NPU's communication data (i.e., the first information) can be cannevent data. Cannevent data includes the sequence number of the communication operator used for data transmission in event processing, the communication operator type, the communication domain number, the NPUs included in the communication domain, and the communication operator status. For example, the sequence number can be the value corresponding to the `sequence` field, the communication operator type can be the value corresponding to the `opType` field, the communication domain number can be the value corresponding to the `pgId` field, the NPUs included in the communication domain can be the value corresponding to the `comm` field, and the communication operator status can be the value corresponding to the `status` field. The PC can obtain the corresponding values ​​based on these fields, thereby obtaining the first information.

[0063] For example, the first information could also be a unique identifier generated based on the NPU's communication data. For instance, the first information could be a hash value generated by using a hash algorithm on the NPU's cannevent data.

[0064] In some embodiments, when the PC obtains the first information of each NPU among multiple NPUs, the PC first sends a first request to each of at least one server. Then, the server obtains the first information of each NPU among at least one NPU provided by the server according to the first request. Finally, each server sends the first information to the PC.

[0065] The first request includes the location for obtaining the first information of the NPU used for model training among the multiple NPUs on the corresponding server.

[0066] For example, the location can be the file path of the file used to store the first information. The server can obtain the first information of the corresponding NPU based on the file path so as to send the first information to the PC.

[0067] For example, a PC can interact with a server to obtain initial information for each NPU. This initial information acquisition function can be configured on the server by a user or technician. For instance, as shown in Figure 5, a user can enable the initial information generation function on a server configuration page (such as the "Server Configuration" page) so that the server can obtain the initial information for each NPU. This server configuration page can be provided by the server itself, allowing the user to directly configure the server on that page; alternatively, it can be provided by other electronic devices used by the user to configure the server.

[0068] For example, each NPU on the server runs a main process / thread. This main process / thread can obtain the first information of the communication operator currently executing on the NPU and write it to the corresponding file. Technical personnel can know the file path where this file is stored and configure this file path to the PC, allowing the PC to save either the file path where the NPU's first information is stored on each server or the location where the first information is retrieved from each server. Later, during model training fault identification, the PC determines the multiple NPUs used for model training and the file path for retrieving the first information from each NPU. Based on the file path, it generates a first request and sends it to the server where each NPU resides. The server can retrieve the first information from the corresponding file based on the file path in the first request and send the retrieved information to the PC. The PC stores the retrieved first information for subsequent fault identification.

[0069] For example, the method of acquiring the first information and sending it to the PC can be periodic to ensure the performance of the server or PC, or it can be real-time to promptly identify model training freezes. The appropriate acquisition method can be selected according to actual needs.

[0070] S402. For each NPU, if the PC obtains the same N first pieces of information in N consecutive N times, it obtains the second information for each NPU.

[0071] Where N is a positive integer greater than 1.

[0072] For example, for each NPU, if the first information of the NPU obtained in N consecutive times is consistent, or if multiple first information of the NPU obtained within a first preset time range is consistent, it indicates that the NPU may stop communicating. In the case that each NPU may stop communicating, the second information of each NPU is obtained.

[0073] The second information includes at least one metric used to indicate processor performance.

[0074] For example, the metrics are NPU utilization or NPU I / O data volume.

[0075] For example, for each NPU, if the N first pieces of information obtained by the PC in N consecutive times are not consistent, or if the multiple first pieces of information obtained by the PC within a first preset time range for the NPU are not consistent, the PC can continue to obtain the first pieces of information of the NPU for comparison, which can further improve the accuracy of determining the model training deadlock fault.

[0076] S403 and PC determine that a deadlock fault exists during model training if each of the multiple second pieces of information meets the preset conditions.

[0077] For example, the second information satisfying the preset conditions includes any one of the following: at least one indicator included in the second information is less than or equal to the preset threshold corresponding to the indicator; at least one indicator included in the second information is less than or equal to the preset threshold corresponding to the indicator within a preset time range; or, the probability that at least one indicator included in the second information is less than or equal to the preset threshold corresponding to the indicator within a preset time range reaches a preset probability.

[0078] For example, the preset threshold can be any value between 0 and 20, such as a preset threshold of 0 or 10. The preset time range can be any value between 0 minutes and 30 minutes, such as a preset time range of 1 minute or 5 minutes. The preset probability can be any value between 80% and 100%, such as a preset probability of 90% or 98%.

[0079] For example, the second piece of information includes NPU utilization and I / O data volume, and the preset thresholds corresponding to NPU utilization and I / O data volume are both 0. The PC can determine whether NPU utilization and I / O data volume are both 0. If NPU utilization and I / O data volume are both 0, then it is determined that the model training has frozen. Alternatively, the PC can determine whether NPU utilization and I / O data volume are both 0 within 1 minute. If NPU utilization and I / O data volume are both 0 within 1 minute, then it is determined that the model training has frozen. Alternatively, the PC can determine whether the probability of NPU utilization and I / O data volume being 0 within 1 minute reaches 90%. If the probability of NPU utilization and I / O data volume being 0 within 1 minute reaches 90%, then it is determined that the model training has frozen.

[0080] The steps S401-S403 described above realize the model training fault identification in the embodiments of this application. Without waiting for the timeout period, it can immediately determine whether the processor's communication is normal by comparing the processor's most recent first information. If the processor's communication is abnormal, it can further determine whether the processor's indicators meet the preset conditions, thereby determining whether the model training is stuck, which improves the accuracy and efficiency of determining the model training stuck fault.

[0081] In some embodiments, the PC displays a first message indicating a stuck failure.

[0082] For example, the first prompt message displayed on the PC may be as shown in Figure 6, prompting the user that the model training has stalled so that they can find out the reason for the stall.

[0083] As shown in Figure 7, this embodiment of the application provides a completion process for model training fault identification. The PC obtains the NPU of at least one server used for model training. The user configures the cannevent data generation function for each server corresponding to the model training. Processes on the NPU within the server obtain the cannevent data of the currently executing communication operator and write the cannevent data to the corresponding files of each process. The user configures the file path for the cannevent data on the PC. The PC sends a first request to the corresponding server, which includes the file path. The server retrieves the cannevent data of each process from the corresponding file according to the file path and sends the cannevent data of each process to the PC. Cannevent data: The PC stores the cannevent data for each process. It can use the cannevent data of each process to calculate and store the hash value. The server collects the second information corresponding to each process and sends the second information of each process to the PC. The PC stores the second information corresponding to each process. Then, when the PC identifies whether the model training has been stuck, it compares the hash values ​​of the cannevent data collected multiple times for each process to see if they are the same. If the hash values ​​of the cannevent data collected multiple times for each process are the same, it is considered that the model training may be stuck. It checks whether the second information of each process meets the preset conditions. If the preset conditions are met, it is considered that the model training is stuck.

[0084] In some embodiments, after the PC determines that a deadlock occurs during model training, the PC obtains the type of the deadlock and displays the type of the deadlock.

[0085] For example, the types of stuck failures include at least one of the following: failure of the switching device connected to the NPU, failure of the storage device corresponding to model training, failure of the NPU port, failure of NPU hardware, and failure of the server where the NPU is located.

[0086] For example, in the case where the stuck failure is due to a failure of the NPU port of NPU3 on server 2, the displayed failure type is shown in Figure 8, prompting the user that the failure of the NPU port of NPU3 on server 2 has caused the model training to be stuck, so that the user can solve the problem and resume model training.

[0087] In some embodiments, when the PC obtains the fault type of the stuck fault, the PC first identifies at least one abnormal NPU among a plurality of NPUs, and then the PC determines the fault type of the stuck fault based on the abnormal NPU.

[0088] For example, the PC can determine at least one abnormal NPU among multiple NPUs in, but is not limited to, the following two ways:

[0089] Method 1: The abnormal NPU is the NPU among multiple NPUs whose operator delivery timed out.

[0090] For example, each of the multiple NPUs uses operators to train the model. Operator delivery timeouts include user-delivered operator timeouts and framework-delivered operator timeouts. If the server obtains an operator delivery timeout, it sends an alarm message about the NPU's operator delivery timeout to the PC. If the PC receives the alarm message about the operator delivery timeout, it will identify the corresponding NPU as an abnormal NPU.

[0091] Method 2: When the PC determines at least one abnormal NPU among multiple NPUs, firstly, the PC determines multiple communication domains. Then, for each communication domain, the PC obtains the first information of each of the multiple NPUs in the communication domain. Finally, the PC determines the abnormal NPU among the multiple NPUs based on the first information of each of the multiple NPUs.

[0092] For example, each of the multiple communication domains determined by the PC may include a PP communication domain or a non-PP communication domain, and the PC may identify the abnormal NPU in each communication domain.

[0093] Taking a non-PP communication domain as an example, the PC can obtain the cannevent data of each NPU in each non-PP communication domain. The communication status of the NPU can be obtained through the cannevent data. The communication status includes the start of event execution and the end of event execution. Under normal circumstances, NPUs in the same communication domain should execute the same event synchronously, and the communication status of NPUs in the same communication domain should be consistent. If the communication status of NPU1 in communication domain 2 of the non-PP domain is "end" and the communication status of NPU2 is "start", it may indicate that NPU1 has encountered a problem when executing the event, causing NPU1 to end prematurely or fail to synchronize correctly. The PC will identify NPU1 as an abnormal NPU.

[0094] As can be seen, PC can identify abnormal NPUs in two ways, thereby improving the accuracy of identifying abnormal NPUs.

[0095] In some embodiments, when the PC determines the type of the stuck fault based on the abnormal NPU, the PC needs to determine whether there is an error log of the error completion queue element (Error CQE) in each communication domain where the abnormal NPU is located. The PC determines the type of the stuck fault based on the Error CQE.

[0096] For example, if the PC determines that an Error CQE exists in any one or at least one communication domain where the abnormal NPU is located, the type of the stuck fault is determined based on the number of abnormal NPUs identified.

[0097] For example, if the PC determines that there is no Error CQE in each communication domain where the abnormal NPU is located, the PC displays the abnormal NPU to prompt the user to analyze the stuck fault.

[0098] In some embodiments, when the PC determines the type of a stuck fault based on the number of abnormal NPUs, if there is only one abnormal NPU, the PC first obtains at least one communication domain where the abnormal NPU is located. Then, the PC performs multi-layer detection on the at least one communication domain and obtains the detection results for each layer. Finally, if the detection results for any layer indicate that there is an abnormal NPU in at least one communication domain, the PC determines the fault type of the abnormal NPU as the type of a stuck fault.

[0099] For example, all communication domains where the abnormal NPU resides can be PP communication domains or non-PP communication domains (e.g., TP communication domains). Multi-layered detection is performed on all NPUs within each communication domain containing the abnormal NPU. This multi-layered detection includes network-level detection, storage-level detection, and NPU-level detection. Network-level detection includes detection of the switching devices connected to the NPUs; these switching devices are used for communication between NPUs. Storage-level detection includes detection of the storage devices corresponding to model training; these storage devices are used to store data used for model training and data generated during model training. NPU-level detection includes detection of at least one of the following: NPU port, NPU hardware, the server where the NPU resides, and the storage client where the NPU resides.

[0100] For example, the PC can first perform NPU-level detection on each NPU within each communication domain where the abnormal NPU resides. If the NPU-level detection result indicates an anomaly, the freeze fault type is classified as an NPU-level fault. If the NPU-level detection result indicates no anomaly, then network-level detection is performed on each NPU. If the network-level detection result indicates an anomaly, the freeze fault type is classified as a network-level fault. If the network-level detection result indicates no anomaly, then storage-level detection is performed on each NPU. If the storage-level detection result indicates an anomaly, the freeze fault type is classified as a storage-level fault. If the storage-level detection result indicates no anomaly, the PC can display the abnormal NPU to prompt the user to analyze the cause of the freeze fault. The above multi-level detection order is an example. In other examples, network-level detection can be performed first, followed by NPU-level detection, and finally storage-level detection. This embodiment does not limit the order of multi-level detection.

[0101] For example, when a PC performs NPU-level detection on an NPU, the PC can detect whether there is an alarm message for that NPU (at least one of the following: NPU port, NPU hardware, the server where the NPU is located, or the storage client where the NPU is located) from the alarm information of the server. Assuming that there is an alarm message for the NPU port of that NPU in the alarm information of the server, or if the alarm level of the alarm message for the NPU port of that NPU indicates a severe alarm, the PC can determine that the fault type of the stuck failure is an NPU port failure of that NPU, and the PC can display a prompt message indicating that the NPU port is faulty.

[0102] For example, when a PC performs network-level testing on an NPU, it can detect whether there are alarm messages from the network alarm information for the switching device connected to the NPU. If the network alarm information contains alarm messages for the switching device connected to the NPU, or if the alarm level of the alarm message for the switching device connected to the NPU indicates a severe alarm, the PC can determine that the fault type of the stuck fault is a fault of the switching device connected to the NPU, and the PC can display a prompt message indicating that the switching device connected to the NPU is faulty.

[0103] For example, when the PC performs storage-level testing on the NPU, the PC can detect whether there are alarm messages for the storage device corresponding to model training from the alarm messages detected by the storage device. Assuming that there are alarm messages for the storage device corresponding to model training in the alarm messages detected by the storage device, or if the alarm level of the alarm message for the storage device corresponding to model training indicates a severe alarm, the PC can determine that the fault type of the stuck failure is a fault of the storage device corresponding to model training, and the PC can display a prompt message indicating that the storage device corresponding to model training is faulty.

[0104] It is evident that multi-layered detection can improve the accuracy of PC in determining the type of model training freeze fault.

[0105] In some embodiments, when the PC determines the type of the stuck fault based on the number of abnormal NPUs, if there are multiple abnormal NPUs, the PC first performs NPU-level detection for each of the multiple abnormal NPUs. Then, if no abnormality is found in the NPU-level detection of each abnormal NPU, the PC performs network-level detection for each abnormal NPU. Finally, if an abnormal NPU is detected in multiple network-level detections, the PC determines that the type of the stuck fault is a network fault.

[0106] For example, when a PC performs NPU-level detection on multiple abnormal NPUs, for each abnormal NPU, the PC checks the alarm information of the server to see if there is an alarm for that abnormal NPU (at least one of NPU port, NPU hardware, or the server where the NPU is located). Assuming that the alarm information of the server contains alarm information for the NPU hardware of that abnormal NPU, or if the alarm level of the alarm information for the NPU hardware of that abnormal NPU indicates a severe alarm, the PC determines that the fault type of the stuck failure is an NPU hardware failure of that abnormal NPU, and the PC can display a prompt message indicating an NPU hardware failure of that abnormal NPU.

[0107] For example, when a PC performs network-level detection on multiple abnormal NPUs, for each abnormal NPU, the PC checks the alarm information of the NCE to see if there is an alarm information for the switching device connected to that abnormal NPU. Assuming that the alarm information of the NCE contains an alarm information for the switching device connected to that abnormal NPU, or if the alarm level of the alarm information for the switching device connected to that abnormal NPU indicates a severe alarm, the PC determines that the fault type of the stuck fault is a fault of the switching device connected to that abnormal NPU. The PC can display a prompt message indicating a fault of the switching device connected to that abnormal NPU. The PC can also display all abnormal NPUs for the user to view.

[0108] For example, if the detection results at both the NPU level and the network level of multiple abnormal NPUs are not abnormal, the PC displays multiple abnormal NPUs to allow users to analyze the cause of the model training freeze.

[0109] It is evident that when there are multiple abnormal NPUs, the causes of failure at the NPU level and network level of the abnormal NPUs can be considered first, which can further improve the efficiency of determining the type of model training deadlock.

[0110] For example, when a PC performs network-level detection, the network detection alarm information obtained can be obtained from a network detection device, a network detection platform, or a network detection tool. For instance, the PC can obtain NPU network-level alarm information from the network cloud engine (NCE). When a PC performs storage-level detection, the storage device detection alarm information obtained can be obtained from a storage detection device, a storage detection platform, or a storage detection tool. For instance, the PC can obtain NPU storage-level alarm information from the direct memory access (DME) protocol.

[0111] For example, as shown in Figure 9, before determining the type of model training deadlock, the PC obtains the NPU used for model training. The technician enables the generation function of cannevent data on the server corresponding to the NPU. All processes on the NPU in the server obtain the cannevent data of the communication operator being executed, Error CQE, timeout of user-issued operator, timeout of framework-issued operator, and other first information, and write the first information into the file corresponding to the process. The technician configures the file path to the PC. During deadlock detection, the PC determines the multiple NPUs used for model training and the file path for obtaining the first information of each NPU. Based on the file path, it generates a first request and sends it to the server where the NPU resides. The server retrieves the first information of each process from the corresponding file based on the first request and sends it back to the PC, which stores the first information. The server also retrieves NPU-level alarm information, such as NPU port failures, NPU hardware failures, NPU server failures, and NPU storage client failures, and sends these alarms back to the PC, which stores them. The NCE retrieves network-level alarm information from the switching devices connected to each NPU and sends it back to the PC, which stores it. The DME retrieves storage-level alarm information from the storage devices corresponding to model training and sends it back to the PC, which stores it. Finally, the PC can determine the type of model training deadlock based on the stored first information and alarm information.

[0112] For example, this application embodiment also provides a system, as shown in FIG10. The system includes an operation and maintenance platform (e.g., a cluster computing application engine, CCAE), multiple servers (server 1 to server n) for user model training, an NCE, and a DME. Each server includes multiple NPUs and a proxy service (e.g., CCAgent). During model training, the proxy service of each server sends the acquired NPU data and corresponding alarm information to a PC. The PC stores this data and alarm information. The NCE sends the acquired network-level alarm information corresponding to each NPU to the PC. The PC stores the network-level alarm information. The DME sends the acquired storage-level alarm information corresponding to each NPU to the PC. The PC stores the storage-level alarm information. The PC identifies whether a model training freeze has occurred based on the stored data. After determining that a model training freeze has occurred, the PC determines the cause of the model training freeze based on the stored data and alarm information. The NCE and DME can be devices or platforms independent of the PC, or they can be tools or services provided on the PC. The operation and maintenance platform can be used to implement the functions that can be achieved by the aforementioned electronic devices such as PCs. For example, the electronic device may include the operation and maintenance platform.

[0113] It is understood that, in order to achieve the above functions, the aforementioned electronic device includes hardware and / or software modules corresponding to perform each function. Based on the algorithmic steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0114] This application embodiment can divide the electronic device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0115] This application also provides an electronic device, as shown in FIG11, which may include one or more processors 1101, memory 1102 and communication interface 1103.

[0116] The memory 1102, communication interface 1103, and processor 1101 are coupled together. For example, the memory 1102, communication interface 1103, and processor 1101 can be coupled together via bus 1104.

[0117] The communication interface 1103 is used for data transmission with other devices. The memory 1102 stores computer program code. The computer program code includes computer instructions, which, when executed by the processor 1101, cause the electronic device to perform the model training fault identification method described in this embodiment.

[0118] The processor 1101 may be a processor or controller, such as a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with this disclosure. The processor may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0119] Bus 1104 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 1104 can be categorized as an address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in Figure 11, but this does not indicate that there is only one bus or one type of bus.

[0120] This application also provides a computer-readable storage medium storing computer program code. When the processor executes the computer program code, the electronic device executes the relevant method steps in the above method embodiments.

[0121] The electronic devices and computer storage media provided in this application are used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0122] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0123] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0124] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0125] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0126] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, in essence, or the part that contributes, or all or part of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0127] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model training fault identification method, characterized in that, The model training uses a processor from at least one server; the method includes: Acquire first information for each of the plurality of processors; wherein the first information is used to indicate the communication status of the processor; For each processor, if the first information obtained N times consecutively is consistent, second information for each processor is obtained; wherein the second information includes at least one indicator for indicating the performance of the processor; N is a positive integer greater than 1; If each of the second pieces of information satisfies a preset condition, it is determined that a deadlock fault exists during the model training process.

2. The method according to claim 1, characterized in that, The method further includes: Display a first message indicating the stuck fault.

3. The method according to claim 1 or 2, characterized in that, The step of obtaining the first information of each of the multiple processors includes: Send a first request to each of the at least one of the servers; wherein the first request is used to instruct the server to obtain first information of each of the at least one of the processors provided by the server; Receive the first information sent by the server.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: Display a first page; wherein the first page includes multiple controls, each of which is used to indicate a corresponding processor; In response to a user's selection of a control on the first page, multiple processors used in the model training process are determined; wherein the multiple processors come from the same server or from different servers.

5. The method according to any one of claims 1-4, characterized in that, The second information satisfies any of the following preset conditions: at least one indicator included in the second information is less than or equal to the preset threshold corresponding to the indicator; or, at least one indicator included in the second information is less than or equal to the preset threshold corresponding to the indicator within a preset time range.

6. The method according to any one of claims 1-5, characterized in that, The metric is the processor utilization rate or the amount of data input / output by the processor.

7. The method according to claim 1, characterized in that, After determining that a freezing fault exists during the model training process, the method further includes: Obtain the fault type of the aforementioned stuck fault; Display the fault type.

8. The method according to claim 7, characterized in that, The method of obtaining the fault type of the stuck fault includes: Identify at least one abnormal processor among the plurality of processors; The fault type of the stuck fault is determined based on the abnormality processor.

9. The method according to claim 8, characterized in that, Each of the plurality of processors uses an operator to train the model; the abnormal processor is the processor among the plurality of processors whose operator timeout occurs.

10. The method according to claim 8, characterized in that, The step of determining at least one abnormal processor among the plurality of processors includes: Multiple communication domains are defined; each of the multiple communication domains is a communication relationship consisting of multiple processors, wherein the multiple processors come from the same server or different servers; For each of the communication domains, obtain the first information of each of the multiple processors in the communication domain; Based on the first information of each of the plurality of processors, the abnormal processor among the plurality of processors is determined.

11. The method according to any one of claims 8-10, characterized in that, When there is only one exception handler, determining the fault type of the stuck fault based on the exception handler includes: Obtain at least one communication domain where the exception handler is located; Multi-layer detection is performed on the at least one communication domain to obtain the detection results for each layer; the multi-layer detection includes network layer detection, storage layer detection and processor layer detection; If the detection result for any level indicates that there is a processor with a detection anomaly in at least one communication domain, the fault type of the processor with the detection anomaly is determined as the fault type of the stuck fault.

12. The method according to any one of claims 8-10, characterized in that, When there are multiple exception handlers, determining the fault type of the stuck fault based on the exception handlers includes: Perform processor-level detection for each of the multiple exception handlers; If no anomalies are found at the processor level for each anomaly processor, then perform network-level detection for each anomaly processor. If a processor is found to be abnormal during multiple network-level detections, the type of the stuck fault is determined to be a network fault.

13. The method according to claim 11, characterized in that, The network-level detection includes the detection of switching devices connected to the processor; the switching devices are used for communication between processors; the storage-level detection includes the detection of storage devices corresponding to model training; the storage devices are used to store data used for model training and data generated by model training; the processor-level detection includes the detection of at least one of the following: processor port, processor hardware, and the server where the processor is located.

14. The method according to any one of claims 7-13, characterized in that, The types of the stuck failure include at least one of the following: failure of the switching device connected to the processor, failure of the storage device corresponding to model training, failure of the processor port, failure of the processor hardware, and failure of the server where the processor is located.

15. An electronic device, characterized in that, The device includes a memory and one or more processors; the memory is coupled to the processors; wherein the memory stores computer program code, the computer program code including computer instructions, which, when executed by the processor, cause the electronic device to perform the model training fault identification method as described in any one of claims 1-14.

16. A computer-readable storage medium, characterized in that, The method includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the model training fault identification method as described in any one of claims 1-14.

17. A computer program product, characterized in that, When the computer program product is run on a computer, the computer performs the model training fault identification method as described in any one of claims 1-14.

Images