Detection method and related apparatus

Abstract

The present application relates to the technical field of computers. Disclosed are a detection method and a related apparatus, which are applied to a model training scenario. A model training server includes a plurality of data parallel (DP) groups, each DP group comprises a plurality of accelerator cards, and the server performs iterative training on a model by means of the plurality of DP groups. The method comprises: on the basis of total duration data of first iteration, determining whether the first iteration is abnormal; when it is detected that the first iteration is abnormal, determining, on the basis of total waiting duration data of cross-DP group communication of accelerator cards in each DP group in the first iteration, whether a first DP group is abnormal; and when it is detected that the first DP group is abnormal, determining, on the basis of total model training duration data of each accelerator card in the first DP group, whether a first accelerator card in the first DP group is abnormal. In a positioning manner of reducing a troubleshooting range level by level, an improvement in the efficiency of determining an abnormal accelerator card is facilitated.

Description

Detection methods and related devices

[0001] This application claims priority to Chinese Patent Application No. 202510121647.8, filed on January 24, 2025, entitled "Detection Method and Related Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and in particular to a detection method and related apparatus. Background Technology

[0003] In the past two years, with the emergence of various powerful capabilities of large models, the field of large models has become a hot spot pursued by many manufacturers. Training large models requires a large number of artificial intelligence (AI) accelerator cards (such as graphics processing units (GPUs), neural network processing units (NPUs), etc.) to work together. Each accelerator card needs to communicate frequently to synchronize training progress and gradient results. In this process, as long as there is a slow accelerator card, it will drag down the training progress of the entire cluster. The definition of a slow accelerator card is: the time taken by the card to perform communication or computation work is significantly longer than that of other accelerator cards. For example: (1) the execution time of a certain function on the accelerator card is significantly longer than that of other cards; (2) the execution time of a certain operator on the accelerator card is significantly longer than that of other cards; (3) the computation time of the accelerator card for a batch of data is significantly longer than that of other cards; (4) the communication time is significantly longer than that of other cards, etc. However, due to the extremely large number of accelerator cards involved in training large models, and the interweaving of various complex parallel methods, a single accelerator card may belong to different data parallel (DP) groups, as well as different pipeline parallel (PP) groups and tensor parallel (TP) groups. Locating slow accelerator cards in the cluster is a difficult task.

[0004] Currently, the industry mainly uses a global screening approach to locate abnormal accelerator cards during model training. This process requires collecting a large amount of performance data of the accelerator cards. From data collection to locating abnormal accelerator cards, multiple steps are required, resulting in low efficiency in locating abnormal accelerator cards. Summary of the Invention

[0005] This application provides a detection method and related apparatus, which can help improve the efficiency of identifying abnormal accelerator cards.

[0006] Firstly, an embodiment of this application provides a detection method applied to a model training scenario. The model training server includes multiple dynamic programming (DP) groups, each DP group includes multiple accelerator cards, and the server iteratively trains the model through multiple DP groups, including:

[0007] Based on the total duration data of the first iteration, determine whether the first iteration is abnormal; if the first iteration is abnormal, then based on the total waiting time data of cross-DP group communication of accelerator cards within each DP group in the first iteration, determine whether the first DP group is abnormal; if the first DP group is abnormal, then based on the total model training time data of each accelerator card in the first DP group, determine whether the first accelerator card in the first DP group is abnormal.

[0008] As can be seen, in this scheme, abnormal iterations are first identified by the total duration data of each iteration. Then, abnormal DP groups are identified by the total waiting time data of cross-DP group communication between accelerator cards within the DP group from the abnormal iterations. Finally, abnormal accelerator cards are identified by the total model training time data of each accelerator card from the abnormal DP groups. Compared with global investigation, this method of locating abnormal accelerator cards by narrowing the investigation scope layer by layer is conducive to improving the efficiency of identifying abnormal accelerator cards.

[0009] In conjunction with the first aspect, in one possible implementation, determining whether the first iteration is abnormal based on the total duration data of the first iteration includes: obtaining a standard value for the total duration based on the first average and first standard deviation of the total duration data of multiple historical iterations; determining whether the first iteration is abnormal based on the standard value for the total duration and the total duration data of the first iteration; wherein, when the total duration data of the first iteration is greater than the standard value for the total duration, the first iteration is abnormal, and when the total duration data of the first iteration is less than or equal to the standard value for the total duration, the first iteration is normal.

[0010] First, a standard value for the total duration is derived using the first average and first standard deviation of the total duration data from multiple historical iterations. Since this standard value is based on a large amount of statistical data, it can serve as a scientifically reliable quantitative standard. Therefore, by comparing the total duration data of the first iteration with the standard value, it is possible to accurately determine whether the first iteration is abnormal. Once the first iteration is determined to be abnormal, it is unnecessary to investigate all accelerator cards in all iterations, thereby improving the efficiency of locating abnormal accelerator cards.

[0011] In conjunction with the first aspect, in one possible implementation, determining whether the first DP group is abnormal based on the total waiting time data of accelerator card communication across DP groups within each DP group in the first iteration includes: arranging multiple DP groups in descending order of the total waiting time data of accelerator card communication across DP groups to obtain a first DP group sequence; calculating the second average and second standard deviation of the total waiting time data of accelerator card communication across DP groups for the first proportion of DP groups in the first DP group sequence; obtaining a standard value of total waiting time based on the second average and second standard deviation; and determining whether the first DP group is abnormal based on the standard value of total waiting time and the total waiting time data of accelerator card communication across DP groups within the first DP group; wherein, when the total waiting time data of accelerator card communication across DP groups within the first DP group is less than the standard value of total waiting time, the first DP group is abnormal; when the total waiting time data of accelerator card communication across DP groups within the first DP group is greater than or equal to the standard value of total waiting time, the first DP group is normal.

[0012] First, the total waiting time data of multiple DP groups is sorted. Then, a certain proportion of the data is selected to calculate the standard value of the total waiting time. This avoids the influence of abnormal data in multiple total waiting time data on the standard value of the total waiting time, ensuring the accuracy of the standard value of the total waiting time. Then, by comparing the total time data of the first DP group with the standard value of the total waiting time, it is possible to accurately determine whether the first DP group is abnormal. In addition, after determining that the first DP group is abnormal, it is not necessary to check the accelerator cards of all DP groups. Only the accelerator cards in the first DP group need to be checked, thereby improving the efficiency of locating abnormal accelerator cards.

[0013] In conjunction with the first aspect, in one possible implementation, the total model training time data includes total computation time data, tensor parallel TP group intra-group communication waiting time data, and pipelined parallel PP group intra-group communication time data. Based on the total model training time data of each accelerator card in the first DP group, it is determined whether the first accelerator card in the first DP group is abnormal, including: determining whether the first accelerator card is abnormal based on the third average and third standard deviation of the total computation time data of each accelerator card in the first DP group, and the total computation time data of the first accelerator card in the first DP group; or, determining whether the first accelerator card is abnormal based on the fourth average and fourth standard deviation of the TP group intra-group communication waiting time data of each accelerator card in the first DP group, and the TP group intra-group communication waiting time data of the first accelerator card in the first DP group; or, determining whether the first accelerator card is abnormal based on the fifth average and fifth standard deviation of the PP group intra-group communication time data of each accelerator card in the first DP group, and the PP group intra-group communication time data of the first accelerator card in the first DP group.

[0014] Judging whether the first accelerator card is abnormal from multiple dimensions such as total calculation time, communication waiting time within the TP group, and communication time within the PP group can avoid the limitations of evaluation by a single indicator and improve the accuracy of locating abnormal accelerator cards.

[0015] In conjunction with the first aspect, in one possible implementation, determining whether a first accelerator card is abnormal is based on the third average and third standard deviation of the total computation time data of each accelerator card in the first DP group, and the total computation time data of the first accelerator card in the first DP group. This includes: arranging multiple accelerator cards in ascending order of their total computation time data to obtain a first accelerator card sequence; calculating the third average and third standard deviation of the total computation time data of the second proportion of accelerator cards in the first accelerator card sequence; obtaining a standard value for total computation time based on the third average and third standard deviation; and determining whether a first accelerator card is abnormal based on the standard value for total computation time and the total computation time data of the first accelerator card in the first DP group. Specifically, when the total computation time data of the first accelerator card is greater than the standard value for total computation time, the first accelerator card is abnormal; when the total computation time data of the first accelerator card is less than or equal to the standard value for total computation time, the first accelerator card is normal.

[0016] First, the total computation time data of multiple accelerator cards are sorted. Then, a certain proportion of the data is selected to calculate the standard value of the total computation time. This avoids the influence of abnormal data in the total computation time data on the standard value of the computation time, ensuring the accuracy of the standard value of the computation time. Then, the total computation time data of the first accelerator card is directly compared with the standard value of the computation time to determine whether the first accelerator card is abnormal. This can accurately and efficiently locate abnormal accelerator cards.

[0017] In conjunction with the first aspect, in one possible implementation, the determination of whether the first accelerator card is abnormal is based on the fourth average and fourth standard deviation of the intra-TP communication waiting time data of each accelerator card in the first DP group, and the intra-TP communication waiting time data of the first accelerator card in the first DP group. This includes: arranging multiple accelerator cards in descending order of intra-TP communication waiting time data to obtain a second accelerator card sequence; calculating the fourth average and fourth standard deviation of the intra-TP communication waiting time data of the third proportion of accelerator cards in the second accelerator card sequence; obtaining a standard value of intra-TP communication waiting time based on the fourth average and fourth standard deviation; and determining whether the first accelerator card is abnormal based on the standard value of intra-TP communication waiting time and the intra-TP communication waiting time data of the first accelerator card. Specifically, when the intra-TP communication waiting time data of the first accelerator card is less than the standard value of intra-TP communication waiting time, the first accelerator card is abnormal; when the intra-TP communication waiting time data of the first accelerator card is greater than or equal to the standard value of intra-TP communication waiting time, the first accelerator card is normal.

[0018] First, the communication waiting time data within the TP group of multiple accelerator cards is sorted. Then, a certain proportion of the data is selected to calculate the standard value of the communication waiting time within the TP group. This avoids the influence of abnormal data in the communication waiting time data within multiple TP groups on the standard value of the communication waiting time within the TP group, ensuring the accuracy of the standard value of the communication waiting time within the TP group. Then, by directly comparing the communication waiting time data within the TP group of the first accelerator card with the standard value of the communication waiting time within the TP group, it is determined whether the first accelerator card is abnormal. This can accurately and efficiently locate abnormal accelerator cards.

[0019] In conjunction with the first aspect, in one possible implementation, the determination of whether the first accelerator card is abnormal is based on the fifth average and fifth standard deviation of the communication duration data within the PP group of each accelerator card in the first DP group, and the communication duration data within the PP group of the first accelerator card in the first DP group. This includes: arranging multiple accelerator cards in ascending order of their communication duration data within the PP group to obtain a third accelerator card sequence; calculating the fifth average and fifth standard deviation of the communication duration data within the PP group of the fourth proportion of accelerator cards in the third accelerator card sequence; obtaining a standard value for the communication duration within the PP group based on the fifth average and fifth standard deviation; and determining whether the first accelerator card is abnormal based on the standard value for the communication duration within the PP group and the communication duration data within the PP group of the first accelerator card. Specifically, when the communication duration data within the PP group of the first accelerator card is greater than the standard value for the communication duration within the PP group, the first accelerator card is abnormal; when the communication duration data within the PP group of the first accelerator card is less than or equal to the standard value for the communication duration within the PP group, the first accelerator card is normal.

[0020] First, the communication duration data within the PP group of multiple accelerator cards is sorted. Then, a certain proportion of the data is selected to calculate the standard value of the communication duration within the PP group. This avoids the influence of abnormal data in the communication duration data of multiple PP groups on the standard value of the communication duration within the PP group, ensuring the accuracy of the standard value of the communication duration within the PP group. Then, by directly comparing the communication duration data within the PP group of the first accelerator card with the standard value of the communication duration within the PP group, it is determined whether the first accelerator card is abnormal. This can accurately and efficiently locate abnormal accelerator cards.

[0021] In conjunction with the first aspect, in one possible implementation, the method further includes: if an anomaly is detected in the first accelerator card in the first DP group, then a standard value for the total execution time is obtained based on the sixth average and sixth standard deviation of the total execution time data of each function or operator in the first accelerator card on a normal accelerator card; functions in the first accelerator card whose total execution time data is greater than the standard value for the total execution time are identified as problematic functions, and operators in the first accelerator card whose total execution time data is greater than the standard value for the total execution time are identified as problematic operators.

[0022] First, the standard value of total execution time is derived by using the sixth average and sixth standard deviation of the total execution time data of each function or operator in the first accelerator card on a normal accelerator card. Since the standard value of total execution time is based on a large amount of statistical data, it can serve as a scientifically reliable quantitative standard. Therefore, by directly comparing the total execution time data of functions and operators in the first accelerator card with the standard value of total execution time, the problematic functions or operators in the first accelerator card can be accurately and efficiently identified, thereby determining the cause of the anomaly of the first accelerator card.

[0023] Secondly, a detection device provided in the embodiments of this application is applied to a model training scenario. The model training server includes multiple DP groups, each DP group includes multiple accelerator cards, and the server iteratively trains the model through multiple DP groups. The detection device includes:

[0024] The first judgment unit is used to determine whether the first iteration is abnormal based on the total duration data of the first iteration;

[0025] The second judgment unit is used to detect an abnormality in the first iteration and then determine whether the first DP group is abnormal based on the total waiting time data of the acceleration card cross-DP group communication in each DP group in the first iteration.

[0026] The third judgment unit is used to detect an anomaly in the first DP group. Based on the total training time data of each accelerator card in the first DP group, it determines whether the first accelerator card in the first DP group is abnormal.

[0027] Thirdly, a computer provided in the embodiments of this application includes a processor, a memory, and a computer program or instructions stored in the memory, wherein the processor executes the computer program or instructions to implement the steps in the method designed in the first aspect above.

[0028] Fourthly, an embodiment of this application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program or instructions, which, when executed, implement the steps in the method designed in the first aspect described above.

[0029] Fifthly, a computer program product provided in this application includes a computer program or instructions, wherein when the computer program or instructions are executed, the steps in the method designed in the first aspect described above are performed. Exemplarily, the computer program product may be a software installation package.

[0030] The beneficial effects of the technical solutions in the second to fifth aspects can be found in the technical effects of the technical solutions in the first or second aspects, and will not be repeated here. Attached Figure Description

[0031] Figure 1 is a schematic diagram of the architecture of a detection system provided in an embodiment of this application;

[0032] Figure 2 is a flowchart illustrating a detection method provided in an embodiment of this application;

[0033] Figure 3 is a flowchart illustrating another detection method provided in an embodiment of this application;

[0034] Figure 4 is a schematic diagram of a time consumption anomaly detection provided in an embodiment of this application;

[0035] Figure 5 is a functional unit block diagram of a detection device provided in an embodiment of this application;

[0036] Figure 6 is a schematic diagram of the structure of a computer provided in an embodiment of this application. Detailed Implementation

[0037] It should be understood that the terms "first," "second," etc., used in the embodiments of this application are used to distinguish different objects, rather than to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, software, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may also include steps or units not listed, or may also include other steps or units inherent to these processes, methods, products, or devices.

[0038] The term "embodiment" as used in the embodiments of this application means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0039] In the embodiments of this application, "at least one" or "at least one item" refers to one or more, and "multiple" refers to two or more.

[0040] In this application's embodiments, "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent the following three cases: A exists alone, A and B exist simultaneously, and B exists alone. A and B can be singular or plural. The character " / " can indicate that the preceding and following associated objects are in an "or" relationship. Additionally, the character " / " can represent a division sign, such as A / B, which means A divided by B.

[0041] In the embodiments of this application, "at least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent the following seven cases: a, b, c, a and b, a and c, b and c, a, b, and c. Each of a, b, and c can be an element or a set containing one or more elements.

[0042] In the embodiments of this application, the terms "of," "corresponding (relevant)," "corresponding," "associated (related)," and "mapped" may sometimes be used interchangeably. It should be noted that when no distinction is emphasized, the concepts or meanings expressed are consistent.

[0043] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0044] The technical solutions of the embodiments of this application will be described in detail below.

[0045] [Acceleration Card]

[0046] An accelerator card is a hardware device that plugs into a computer motherboard to offload computing tasks from the CPU (Central Processing Unit), providing more powerful processing capabilities for specific types of computations. Its core function is to rapidly process large amounts of data and complex computational tasks through a parallel computing architecture. Common types of accelerator cards include GPU accelerator cards and NPU accelerator cards.

[0047]

DP Group

[0048] Data parallelism involves dividing the training data into multiple subsets, with each subset being trained independently on different computing devices. The training results from each device are then aggregated in some way to update the model's parameters. A DP group is a set of computing devices formed based on a data parallelism strategy.

[0049] Each DP group has a complete copy of the model, and different DP groups process different subsets of data. Within a DP group, each accelerator card simultaneously performs forward propagation, calculates the loss, and backpropagates to update parameters on its own processed subset of data. Then, the parameter update information from each accelerator card is aggregated and averaged (or other aggregation methods are used) through a communication mechanism to obtain the final model parameter update.

[0050]

PP Group

[0051] Pipeline parallelism distributes different layers or stages of a model across different computing devices, much like an assembly line in industrial production. Data flows sequentially between different devices, with each device responsible for processing one stage of the data, thus achieving parallel computing. A PP group is a set of computing devices organized according to this pipeline parallelism strategy.

[0052] The model is divided into different stages, each handled by one or more accelerator cards from the PP group. Data first enters the first stage of the pipeline, is processed by the accelerator card in that stage, and then passed to the next stage's accelerator card, and so on, until the data has completed computations across all stages of the pipeline. During this process, accelerator cards in different stages can process different batches of data simultaneously, achieving parallel computing.

[0053]

TP Group

[0054] Tensor parallelism involves partitioning a tensor (such as the weight matrix or input data tensor in a model) across different computing devices (such as accelerator cards), allowing each device to process only a portion of the tensor, thus achieving parallel computation. A TP group is a set of computing devices partitioned based on this tensor parallelism strategy.

[0055] In the TP group, the model's tensors are partitioned along specific dimensions, and each accelerator card is responsible for processing a partitioned portion of the tensor. For example, when performing matrix multiplication, the matrix can be divided into rows or columns and distributed across different accelerator cards. Each accelerator card computes its assigned portion in parallel, and then the results are aggregated through a communication mechanism to obtain the final computation result.

[0056] Please refer to Figure 1, which is a schematic diagram of the architecture of a detection system provided in an embodiment of this application. As shown in Figure 1, the detection system 10 may include a client 101 and a server 102. The client 101 can communicate with the server 102 via a network. The client 101 refers to a device used by the user, such as a smartphone or computer. The user can interact with the system server 102 through the client 101. In this solution, the client 101 sends the model to be trained or the data to be detected during model training to the server 102. The client 101 is also used to receive and display the detection results obtained by the server 102. In addition, the client 101 is also used to promptly determine the processing strategy for the abnormal acceleration card after receiving the abnormal acceleration card and its root cause sent by the server 102.

[0057] Server 102 refers to a remote computer used to process large amounts of computing tasks and store data. Server 102 is equipped with accelerator cards required for model training, and it can group these AI accelerator cards into DP, TP, and PP groups. Server 102 is responsible for detecting abnormal accelerator cards during model training and locating the cause of abnormality in the detection system 10. In this scheme, server 102 determines whether the first iteration is abnormal based on the total duration data of the first iteration. Then, when an abnormality is detected in the first iteration, it determines whether the first DP group is abnormal based on the total waiting time data of cross-DP group communication between accelerator cards within each DP group in the first iteration. Finally, when an abnormality is detected in the first DP group, it determines whether the first accelerator card in the first DP group is abnormal based on the total model training time data of each accelerator card in the first DP group. Of course, Figure 1 is only an example of the architecture of a detection system and does not constitute a limitation on the architecture of the detection system in this embodiment.

[0058] Based on the above, an example of a detection method according to an embodiment of this application will be described below. Please refer to Figure 2, which is a schematic flowchart of a detection method provided by an embodiment of this application. As shown in Figure 2, the method includes the following steps:

[0059] S201, based on the total duration data of the first iteration, determine whether the first iteration is abnormal.

[0060] The execution subject of this method can be the server 102 of the detection system 10.

[0061] The first iteration can be any one of the multiple iterations in model training. In model training, an iteration refers to the process of updating model parameters using a portion of the training data. The total iteration duration refers to the total time spent on each iteration, that is, the total time spent completing such a parameter update process. It includes the total time consumed by each step, such as data loading, forward propagation, loss calculation, backpropagation, and parameter update.

[0062] The total duration of the first iteration can be obtained through training logs or performance analysis tools. Specifically, most deep learning frameworks generate detailed training logs during model training. These logs typically record the start and end times of each iteration, and the total time for that iteration can be calculated by the difference between these two times. For example, frameworks like TensorFlow and PyTorch have corresponding log output functions, allowing developers to obtain the data based on the timestamps in the logs. Furthermore, specialized performance analysis tools can be used to monitor various metrics during model training, including the duration of each iteration. These tools may display the time information visually for easy viewing and analysis. For instance, NVIDIA's Nsight Systems can perform detailed performance analysis on GPU-based training processes, including iteration time and other information.

[0063] In one possible implementation, determining whether the first iteration is abnormal based on the total duration data of the first iteration includes:

[0064] The standard value of total duration is obtained by taking the first average and the first standard deviation of the total duration data of multiple historical iterations; based on the standard value of total duration and the total duration data of the first iteration, it is determined whether the first iteration is abnormal; when the total duration data of the first iteration is greater than the standard value of total duration, the first iteration is abnormal, and when the total duration data of the first iteration is less than or equal to the standard value of total duration, the first iteration is normal.

[0065] When determining whether the first iteration is abnormal based on the total duration data of the first iteration, mathematical methods of hypothesis testing can be used, as well as thresholding methods or machine learning-based methods (e.g., using machine learning algorithms to train a detection model, and using the detection model to detect whether the iteration is abnormal based on the total duration data of the iteration). Here, no restrictions are placed on the method of anomaly detection for the first iteration.

[0066] Specifically, taking the first iteration as the latest iteration as an example, the total duration data of each previous iteration is collected, and the average (avg_t) and standard deviation (std_t) of the total duration data of these iterations are calculated. When the total duration data t1 of the latest iteration is obtained, a judgment is made. If the formula: t1≤avg_t+α1·std_t is satisfied;

[0067] If the total duration of the latest iteration is normal, then the total duration of the latest iteration is considered to be significantly greater than that of other iterations.

[0068] Here, α1 is a constant greater than 0. The value of α1 affects the sensitivity of the algorithm; the smaller the value, the more sensitive the detection. For example, the value of α1 can be 3.

[0069] As can be seen, in this example, the standard value of total duration is first derived using the first average and first standard deviation of the total duration data from multiple historical iterations. Since the standard value of total duration is based on a large amount of statistical data, it can serve as a scientifically reliable quantitative standard. Therefore, by comparing the total duration data of the first iteration with the standard value of total duration, it is possible to accurately determine whether the first iteration is abnormal. After determining that the first iteration is abnormal, it is not necessary to check the accelerator cards of all iterations, thereby improving the efficiency of locating abnormal accelerator cards.

[0070] S202, if an anomaly is detected in the first iteration, then determine whether the first DP group is abnormal based on the total waiting time data of the accelerator card communication across DP groups in each DP group during the first iteration.

[0071] In the training process of a system or model, multiple iterations are typically performed. Each iteration can be viewed as a complete computation cycle, during which multiple DP groups (possibly distributed processing groups) in the system work together to complete a specific task. A DP group is the basic unit in the system that possesses a complete copy of the model. Different DP groups are responsible for performing training computations using different subsets of training data and communicate with other DP groups to synchronize training gradient results.

[0072] The total waiting time for accelerator card communication across DP groups refers to the sum of waiting times spent by accelerator cards within a DP group communicating with accelerator cards in other DP groups during a specific iteration process. This waiting time is the time interval between initiating a communication request and the actual start of data transmission.

[0073] In one possible implementation, the presence or absence of an anomaly in the first DP group is determined based on the total waiting time data for cross-DP group communication of the accelerator cards within each DP group during the first iteration, including:

[0074] Multiple DP groups are arranged in descending order of the total waiting time data for cross-DP group communication of accelerator cards to obtain the first DP group sequence. The second average and second standard deviation of the total waiting time data for cross-DP group communication of accelerator cards within the first proportion of DP groups in the first DP group sequence are calculated. The standard value of the total waiting time is obtained based on the second average and the second standard deviation. Based on the standard value of the total waiting time and the total waiting time data for cross-DP group communication of accelerator cards within the first DP group, it is determined whether the first DP group is abnormal. Specifically, when the total waiting time data for cross-DP group communication of accelerator cards within the first DP group is less than the standard value of the total waiting time, the first DP group is abnormal; when the total waiting time data for cross-DP group communication of accelerator cards within the first DP group is greater than or equal to the standard value of the total waiting time, the first DP group is normal.

[0075] When an anomaly is detected in the first iteration, for each DP group in the first iteration, a batch of data processed by that DP group is tracked, with each DP group and batch of data streams corresponding one-to-one. Key events and times in the processing of the corresponding data streams by each DP group are tracked, including: the start and end times of the forward computation of each accelerator card, the communication time between accelerator cards and DP groups, and the backpropagation time of each accelerator card. Based on the collected time information, the total waiting time of the accelerator cards in each DP group during cross-DP group communication can be calculated. Each DP group corresponds to a total waiting time value, and this total waiting time value is the total waiting duration data.

[0076] If the total waiting time of a DP group during cross-DP group communication is significantly less than that of other DP groups, then the DP group is considered to be slowing down other DP groups as a whole.

[0077] One approach is to identify slow DP groups using hypothesis testing. Specifically, the waiting times (total waiting time data for accelerator card communication across DP groups) of each DP group are sorted from largest to smallest to obtain the first DP group sequence. From this sequence, a certain proportion of the DP groups with the longest waiting times are selected (as a reference, for example, 2 / 3 of the DP groups can be selected). The average (avg_wait) and standard deviation (std_wait) of the waiting times of this portion of DP groups are calculated. Then, the total waiting time data t2 for accelerator card communication across DP groups of other DP groups is evaluated. If the formula is satisfied: t2≥avg_wait-α2·std_wait;

[0078] If the total waiting time data for cross-DP group communication of the accelerator card in the DP group is normal, then the DP group is considered to be an abnormal DP group.

[0079] Here, α2 is a constant greater than 0. The value of α2 affects the sensitivity of the algorithm; the smaller the value, the more sensitive the detection. For example, the value of α2 can be 3.

[0080] As can be seen, in this example, sorting the total waiting time data of multiple DP groups first, and then selecting a certain proportion of the data to calculate the standard value of the total waiting time, can avoid the influence of abnormal data in multiple total waiting time data on the standard value of the total waiting time, and ensure the accuracy of the standard value of the total waiting time. Then, by comparing the total time data of the first DP group with the standard value of the total time, it is possible to accurately determine whether the first DP group is abnormal. In addition, after determining that the first DP group is abnormal, it is not necessary to check the accelerator cards of all DP groups. Only the accelerator cards in the first DP group need to be checked, thereby improving the efficiency of locating abnormal accelerator cards.

[0081] S203 If an anomaly is detected in the first DP group, then determine whether the first accelerator card in the first DP group is abnormal based on the total model training time data of each accelerator card in the first DP group.

[0082] The total model training time data includes total computation time, communication waiting time within the TP group, and communication time within the PP group. Specifically, total computation time refers to the sum of time spent by the accelerator cards performing model computation tasks. During model training, accelerator cards (such as GPUs) perform various computational operations on the input data, including but not limited to matrix multiplication, convolution operations, activation function calculations, and gradient calculations. Communication waiting time within the TP group refers to the time that an accelerator card needs to wait while communicating within the TP group. This includes the waiting time incurred when accelerator cards need to exchange information, share data, or perform synchronization operations. Communication time within the PP group refers to the time spent by the accelerator cards communicating within the PP group. In pipelined parallel training, different accelerator cards may be responsible for different parts of the model, and they need to pass intermediate results to each other; the time spent in this process is the communication time within the PP group.

[0083] As can be seen in this example, the abnormal iterations are first identified by the total duration data of each iteration. Then, the abnormal DP groups are identified by the total waiting time data of cross-DP group communication between accelerator cards within the DP group. Finally, the abnormal accelerator cards are identified by the total model training time data of each accelerator card in the abnormal DP groups. Compared with global investigation, this method of locating abnormal accelerator cards by narrowing the investigation scope layer by layer is beneficial to improving the efficiency of identifying abnormal accelerator cards.

[0084] In one possible implementation, determining whether the first accelerator card in the first DP group is abnormal is based on the total model training time data of each accelerator card in the first DP group. This includes: determining whether the first accelerator card is abnormal based on the third average and third standard deviation of the total computation time data of each accelerator card in the first DP group, and the total computation time data of the first accelerator card in the first DP group; or, determining whether the first accelerator card is abnormal based on the fourth average and fourth standard deviation of the communication waiting time data within the TP group of each accelerator card in the first DP group, and the communication waiting time data within the TP group of the first accelerator card in the first DP group; or, determining whether the first accelerator card is abnormal based on the fifth average and fifth standard deviation of the communication time data within the PP group of each accelerator card in the first DP group, and the communication time data within the PP group of the first accelerator card in the first DP group.

[0085] This involves using hypothesis testing to identify slow acceleration cards (abnormal acceleration cards) within the slow DP group. Based on the total training time data for different models, multi-dimensional detection can be performed. Each detection method yields independent results, and the results from multiple methods can be combined for judgment. The judgment criteria can be: if any one detection method indicates the first acceleration card is an abnormal acceleration card, then the first acceleration card is determined to be abnormal; or, if two or more detection methods indicate the first acceleration card is an abnormal acceleration card, then the first acceleration card is determined to be abnormal. The specific implementation of anomaly detection using multiple detection methods is not limited here.

[0086] As can be seen, in this example, judging whether the first accelerator card is abnormal from multiple dimensions such as total calculation time, communication waiting time within the TP group, and communication time within the PP group can avoid the limitations of single indicator evaluation and improve the accuracy of locating abnormal accelerator cards.

[0087] In one possible implementation, the determination of whether the first accelerator card is abnormal is based on the third average and third standard deviation of the total computation time data of each accelerator card in the first DP group, and the total computation time data of the first accelerator card in the first DP group, including:

[0088] Multiple accelerator cards are arranged in ascending order of total computation time to obtain the first accelerator card sequence. The third average and third standard deviation of the total computation time of the second-ranked accelerator cards in the first accelerator card sequence are calculated. The standard value of total computation time is obtained based on the third average and third standard deviation. Based on the standard value of total computation time and the total computation time of the first accelerator card in the first DP group, it is determined whether the first accelerator card is abnormal. Specifically, when the total computation time of the first accelerator card is greater than the standard value of total computation time, the first accelerator card is abnormal; when the total computation time of the first accelerator card is less than or equal to the standard value of total computation time, the first accelerator card is normal.

[0089] One approach is to identify slow accelerator cards based on hypothesis testing. Specifically, first, all accelerator cards are sorted by computation time from smallest to largest to obtain the first accelerator card sequence. A certain proportion (e.g., 2 / 3) of these cards with the shortest computation time is then selected, and the average (avg_comp) and standard deviation (std_comp) of their computation time are calculated. Next, the total computation time (t3) of the other accelerator cards in the first DP group is evaluated one by one. If the formula t3 ≤ avg_comp + α3·std_comp is satisfied;

[0090] If the total computation time of the accelerator card is normal, it is considered to be normal; otherwise, it is considered to be an abnormal accelerator card.

[0091] Here, α3 is a constant greater than 0. The value of α3 affects the sensitivity of the algorithm; the smaller the value, the more sensitive the detection. For example, the value of α3 can be 3.

[0092] As can be seen, in this example, the total computation time data of multiple accelerator cards is sorted first, and then a certain proportion of the data is selected to calculate the standard value of the total computation time. This avoids the influence of abnormal data in the total computation time data on the standard value of the computation time, ensuring the accuracy of the standard value of the computation time. Then, by directly comparing the total computation time data of the first accelerator card with the standard value of the computation time, it is possible to determine whether the first accelerator card is abnormal, which can accurately and efficiently locate abnormal accelerator cards.

[0093] In one possible implementation, the determination of whether the first accelerator card is abnormal is based on the fourth average and fourth standard deviation of the intra-TP communication wait time data of each accelerator card in the first DP group, and the intra-TP communication wait time data of the first accelerator card in the first DP group, including:

[0094] Multiple accelerator cards are arranged in descending order of their intra-TP group communication waiting time data to obtain a second accelerator card sequence. The fourth average and fourth standard deviation of the intra-TP group communication waiting time data for the top three proportion of accelerator cards in the second accelerator card sequence are calculated. A standard value for intra-TP group communication waiting time is obtained based on the fourth average and fourth standard deviation. Based on the standard value and the intra-TP group communication waiting time data of the first accelerator card, it is determined whether the first accelerator card is abnormal. Specifically, if the intra-TP group communication waiting time data of the first accelerator card is less than the standard value, the first accelerator card is abnormal; if the intra-TP group communication waiting time data of the first accelerator card is greater than or equal to the standard value, the first accelerator card is normal.

[0095] One approach is to identify slow-speed acceleration cards using hypothesis testing. Specifically, all acceleration cards are sorted from largest to smallest based on their communication latency within the TP group, resulting in a second acceleration card sequence. A certain proportion (e.g., 2 / 3) of these cards with the longest latency are then selected, and the average (avg_tpwait) and standard deviation (std_tpwait) of their TP group communication latency are calculated. Then, the TP group communication latency data t4 of the other acceleration cards is evaluated one by one. If the formula t4 ≥ avg_tpwait - α4·std_tpwait is satisfied;

[0096] If the data shows that the TP group communication waiting time of the accelerator card is normal, then the accelerator card is considered to be an abnormal accelerator card.

[0097] Here, α4 is a constant greater than 0. The value of α4 affects the sensitivity of the algorithm; the smaller the value, the more sensitive the detection. For example, the value of α4 can be 3.

[0098] As can be seen, in this example, the communication waiting time data within the TP group of multiple accelerator cards is first sorted, and then a certain proportion of the data is selected to calculate the standard value of the communication waiting time within the TP group. This avoids the influence of abnormal data in the communication waiting time data within multiple TP groups on the standard value of the communication waiting time within the TP group, ensuring the accuracy of the standard value of the communication waiting time within the TP group. Then, by directly comparing the communication waiting time data within the TP group of the first accelerator card with the standard value of the communication waiting time within the TP group, it is determined whether the first accelerator card is abnormal, which can accurately and efficiently locate abnormal accelerator cards.

[0099] In one possible implementation, the determination of whether the first accelerator card is abnormal is based on the fifth average and fifth standard deviation of the communication duration data within the PP group for each accelerator card in the first DP group, and the communication duration data within the PP group for the first accelerator card in the first DP group, including:

[0100] Multiple accelerator cards are arranged in ascending order of their PP group communication duration data to obtain a third accelerator card sequence. The fifth average and fifth standard deviation of the PP group communication duration data for the top-ranked accelerator cards in the third accelerator card sequence are calculated. The standard value of the PP group communication duration is obtained based on the fifth average and fifth standard deviation. Based on the standard value of the PP group communication duration and the PP group communication duration data of the first accelerator card, it is determined whether the first accelerator card is abnormal. Specifically, if the PP group communication duration data of the first accelerator card is greater than the standard value, the first accelerator card is abnormal; if the PP group communication duration data of the first accelerator card is less than or equal to the standard value, the first accelerator card is normal.

[0101] One approach is to identify slow-speed acceleration cards using hypothesis testing. Specifically, all acceleration cards are sorted by their communication time within the PP group from smallest to largest to obtain the third acceleration card sequence. A certain proportion (e.g., 2 / 3) of these cards with the shortest communication time is selected, and the average (avg_ppcomm) and standard deviation (std_ppcomm) of their communication time within the PP group are calculated. Then, the communication time data t5 within the PP group for the other acceleration cards is evaluated one by one. If the formula t5 ≤ avg_ppcomm + α5·std_ppcomm is satisfied;

[0102] If the data shows normal communication duration within the PP group, then the accelerator card is considered to be normal; otherwise, the accelerator card is considered to be abnormal.

[0103] Here, α5 is a constant greater than 0. The value of α5 affects the sensitivity of the algorithm; the smaller the value, the more sensitive the detection. For example, the value of α5 can be 3.

[0104] As can be seen, in this example, the communication duration data within the PP group of multiple accelerator cards is first sorted, and then a certain proportion of the data is selected to calculate the standard value of the communication duration within the PP group. This avoids the influence of abnormal data in the communication duration data of multiple PP groups on the standard value of the communication duration within the PP group, ensuring the accuracy of the standard value of the communication duration within the PP group. Then, by directly comparing the communication duration data within the PP group of the first accelerator card with the standard value of the communication duration within the PP group, it is determined whether the first accelerator card is abnormal, which can accurately and efficiently locate abnormal accelerator cards.

[0105] In one possible implementation, the method also includes:

[0106] If an anomaly is detected in the first accelerator card in the first DP group, the standard value of the total execution time is obtained based on the sixth average and sixth standard deviation of the total execution time data of each function or operator in the first accelerator card on the normal accelerator card; functions whose total execution time data in the first accelerator card is greater than the standard value of the total execution time are identified as problematic functions, and operators whose total execution time data in the first accelerator card is greater than the standard value of the total execution time are identified as problematic operators.

[0107] After locating the slow acceleration card (abnormal acceleration card), the software call stack on the single acceleration card is first used as input. Then, multiple functions or operators are introduced as performance indicators on the basis of the original single acceleration card. The indicators of the slow acceleration card and other normal acceleration cards are calculated. Finally, the root cause function or root cause operator of the slow card is found through hypothesis testing.

[0108] The software call stack, also known as the call stack, is a data structure used to manage function (or method) calls and returns during software execution. It records the call relationships between functions and operators executed on the accelerator card, including the names of all executed functions and operators.

[0109] The specific steps are as follows: Collect and parse the training logs; for each function and operator in the software call stack, statistically calculate its performance metrics on each accelerator card, including the number of calls and total execution time. Then, analyze each function and operator individually to determine if it is the root cause of the performance problem.

[0110] This process involves selecting a specific function or operator and summarizing its performance metrics on various normal accelerator cards. Then, a hypothesis testing method is used to determine whether the total execution time of this function or operator on a slow accelerator card is significantly greater than its total execution time on a normal accelerator card.

[0111] Specifically, calculate the average (avg_cost) and standard deviation (std_cost) of the total execution time of the function and operator on each normal accelerator card, and then determine whether the total execution time data t6 of the function and operator on the slow accelerator card satisfies the following formula: t6≤avg_cost+α6·std_cost;

[0112] If the total execution time of the function or operator is normal, then the function or operator is considered to be a problematic function or operator.

[0113] Here, α6 is a constant greater than 0. The value of α6 affects the sensitivity of the algorithm; the smaller the value, the more sensitive the detection. For example, the value of α6 can be 3.

[0114] As can be seen in this example, the standard value of total execution time is first derived using the sixth average and sixth standard deviation of the total execution time data for each function or operator in the first accelerator card on a normal accelerator card. Since the standard value of total execution time is based on a large amount of statistical data, it can serve as a scientifically reliable quantitative standard. Therefore, by directly comparing the total execution time data of functions and operators in the first accelerator card with the standard value of total execution time, the problematic functions or operators in the first accelerator card can be accurately and efficiently identified, thereby determining the cause of the anomaly of the first accelerator card.

[0115] Please refer to Figure 3, which is a flowchart illustrating another detection method provided in an embodiment of this application. As shown in Figure 3, the method includes the following steps:

[0116] S301, Model Training.

[0117] The input to model training can include target data and model structure. The corpus data is the raw material used to train the model, containing various information that the model needs to learn and understand. For example, in natural language processing tasks, the corpus data can be a large number of text files, including news articles, novels, academic papers, and social media texts; in image recognition tasks, the corpus data is a large number of images, covering various objects and scenes. The model structure specifically defines the overall architecture and components of the model. For example, in deep learning, common model structures include convolutional neural networks (CNNs), recurrent neural networks (RNNs), and Transformers. Taking the Transformer architecture, which is commonly used in large models, as an example, it consists of embedding layers, attention layers, decoding layers, etc., each layer having a specific function and connection method.

[0118] Model training requires setting model information, cluster size, model corpus, and parallel parameters. Model information includes detailed information about the model to be trained, such as model name, version number, and algorithm type. Cluster size refers to the number of computing nodes participating in model training and the hardware configuration information of each node. Model corpus is a further refinement and description of the input corpus data, which may include preprocessing of the corpus data such as text segmentation and standardization. Parallel parameters refer to a series of parameters related to parallel computing that need to be set when training in a distributed manner or using multi-core, multi-GPU systems. For example, in training large models, it is necessary to set TP parallelism, PP parallelism, and global_batch_size (the number of data batches processed in each iteration).

[0119] S302, Iteration time anomaly detection.

[0120] Based on the aforementioned model training settings (including corpus data, model structure, cluster size, model corpus, and parallel parameter settings), the model will undergo multiple iterations to optimize the parameters. Anomaly detection of iteration time will monitor and analyze the time spent on each iteration.

[0121] In this process, by detecting whether the iteration time is abnormal, it can be determined whether the current iteration is abnormal. For example, please refer to Figure 4, which is a schematic diagram of an abnormal time consumption detection provided by an embodiment of this application. As shown in Figure 4, the horizontal axis can be different iterations, and the vertical axis is the time data corresponding to different iterations. At this time, a first iteration time threshold and a second iteration time threshold can be set, namely line A and line B. If the iteration time data is not between line A and line B, then it can be considered that the iteration time is abnormal, and the abnormal iteration data is highlighted.

[0122] The setting of the first iteration duration threshold and the second iteration duration threshold can be based on experience, or obtained by statistical analysis of the duration data of previous iterations. Alternatively, only one iteration duration threshold can be set, and the abnormal iteration time can be determined based on the relationship between the iteration duration data and the iteration duration threshold.

[0123] S303, Data Flow Critical Event Tracking.

[0124] Upon detecting anomalies in iteration time, the batch data streams corresponding to different DP groups are used as the tracking objects to track important events and their occurrence times in the data stream processing. Specifically, the data stream is the flow and processing of data during model training, which runs through all stages of model training, from the input of corpus data, its transfer within the model structure, through various preprocessing and computational operations, to the final output and parameter updates. Data stream key event tracking involves the detailed recording and analysis of these key nodes and processes.

[0125] S304, Abnormal DP group location.

[0126] Specifically, by obtaining time information for different DP groups based on key events in the data stream, key indicators are calculated based on the collected time information, and hypothesis testing methods are used to locate abnormal DP groups.

[0127] Please refer to Figure 4 again. As shown in Figure 4, the horizontal axis can be different DP groups, and the vertical axis can be the duration data corresponding to different DP groups. At this time, the duration threshold of the first DP group and the duration threshold of the second DP group can be set, namely line A and line B. If the duration data of the DP group is not between line A and line B, then it can be considered that the DP group has abnormal time consumption, and the abnormal DP group will be highlighted.

[0128] Optionally, the duration thresholds for the first DP group and the second DP group can be set based on experience.

[0129] Optionally, in actual large model training scenarios, abnormal DP groups account for only a minority of all DP groups, and most DP groups are normal DP groups. Based on this, the duration thresholds of the first DP group and the second DP group can be obtained by analyzing the time information of most DP groups, and then it can be used to determine whether there are obviously abnormal DP groups.

[0130] Optionally, you can set only one DP group duration threshold and determine whether the DP group time consumption is abnormal based on the relationship between the DP group duration data and the DP group duration threshold.

[0131] S305, DP Group Critical Event Tracking.

[0132] Once a slow DP group is detected, a batch of data streams is used as the tracking object to track and record the computation and communication time within the DP group.

[0133] S306, Abnormal acceleration card location.

[0134] Among them, key time consumption indicators are calculated based on the collected time information, and hypothesis testing methods are used to locate abnormal acceleration cards.

[0135] Please refer to Figure 4 again. As shown in Figure 4, the horizontal axis can be different accelerator cards, and the vertical axis can be the duration data corresponding to different accelerator cards. At this time, the first accelerator card duration threshold and the second accelerator card duration threshold can be set, namely line A and line B. If the duration data of the accelerator card is not between line A and line B, then it can be considered that the accelerator card time consumption is abnormal, and the abnormal accelerator card will be highlighted.

[0136] Optionally, the first and second acceleration card duration thresholds can be set based on experience.

[0137] Optionally, in actual large model training scenarios, abnormal accelerator cards account for only a minority of all accelerator cards in the same iteration, while most accelerator cards are normal. Based on this, the first accelerator card duration threshold and the second accelerator card duration threshold can be obtained by analyzing the time information of most accelerator cards in the same iteration, and then it can be used to determine whether there are obviously abnormal accelerator cards.

[0138] Optionally, you can set only one acceleration card duration threshold and determine whether the acceleration card time consumption is abnormal based on the relationship between the acceleration card duration data and the acceleration card duration threshold.

[0139] It should be noted that the specific implementation of each operation in the method embodiment shown in Figure 3 can be found in the description of the method embodiment shown in Figure 2 above, and will not be repeated here.

[0140] Please refer to Figure 5, which is a functional unit block diagram of a detection device provided in an embodiment of this application. As shown in Figure 5, the detection device 500 includes:

[0141] The first judgment unit 501 is used to determine whether the first iteration is abnormal based on the total duration data of the first iteration;

[0142] The second judgment unit 502 is used to detect the first iteration abnormality and then determine whether the first DP group is abnormal based on the total waiting time data of the acceleration card cross-DP group communication in each DP group in the first iteration.

[0143] The third judgment unit 503 is used to detect an anomaly in the first DP group and then determine whether the first accelerator card in the first DP group is abnormal based on the total model training time data of each accelerator card in the first DP group.

[0144] In one possible implementation, in determining whether the first iteration is abnormal based on the total duration data of the first iteration, the first judgment unit 501 is specifically used to: obtain a total duration standard value based on the first average and the first standard deviation of the total duration data of multiple historical iterations; and determine whether the first iteration is abnormal based on the total duration standard value and the total duration data of the first iteration; wherein, when the total duration data of the first iteration is greater than the total duration standard value, the first iteration is abnormal, and when the total duration data of the first iteration is less than or equal to the total duration standard value, the first iteration is normal.

[0145] In one possible implementation, regarding determining whether the first DP group is abnormal based on the total waiting time data of the accelerator card communication across DP groups in the first iteration, the second judgment unit 502 is specifically used to: arrange multiple DP groups in descending order of the total waiting time data of the accelerator card communication across DP groups to obtain a first DP group sequence; calculate the second average and second standard deviation of the total waiting time data of the accelerator card communication across DP groups in the first DP group sequence; obtain a standard value of total waiting time based on the second average and the second standard deviation; and determine whether the first DP group is abnormal based on the standard value of total waiting time and the total waiting time data of the accelerator card communication across DP groups in the first DP group; wherein, when the total waiting time data of the accelerator card communication across DP groups in the first DP group is less than the standard value of total waiting time, the first DP group is abnormal, and when the total waiting time data of the accelerator card communication across DP groups in the first DP group is greater than or equal to the standard value of total waiting time, the first DP group is normal.

[0146] In one possible implementation, regarding determining whether the first accelerator card in the first DP group is abnormal based on the total model training time data of each accelerator card in the first DP group, the third judgment unit 503 is specifically used to: determine whether the first accelerator card is abnormal based on the third average and third standard deviation of the total computation time data of each accelerator card in the first DP group, and the total computation time data of the first accelerator card in the first DP group; or, determine whether the first accelerator card is abnormal based on the fourth average and fourth standard deviation of the communication waiting time data within the TP group of each accelerator card in the first DP group, and the communication waiting time data within the TP group of the first accelerator card in the first DP group; or, determine whether the first accelerator card is abnormal based on the fifth average and fifth standard deviation of the communication time data within the PP group of each accelerator card in the first DP group, and the communication time data within the PP group of the first accelerator card in the first DP group.

[0147] In one possible implementation, regarding determining whether a first accelerator card is abnormal based on the third average and third standard deviation of the total computation time data of each accelerator card in the first DP group, and the total computation time data of the first accelerator card in the first DP group, the third judgment unit 503 is specifically used to: arrange multiple accelerator cards in ascending order of their total computation time data to obtain a first accelerator card sequence; calculate the third average and third standard deviation of the total computation time data of the second proportion of accelerator cards in the first accelerator card sequence; obtain a standard value of total computation time based on the third average and third standard deviation; and determine whether the first accelerator card is abnormal based on the standard value of total computation time and the total computation time data of the first accelerator card in the first DP group; wherein, when the total computation time data of the first accelerator card is greater than the standard value of total computation time, the first accelerator card is abnormal, and when the total computation time data of the first accelerator card is less than or equal to the standard value of total computation time, the first accelerator card is normal.

[0148] In one possible implementation, regarding determining whether a first accelerator card is abnormal based on the fourth average and fourth standard deviation of the intra-TP communication waiting time data of each accelerator card in the first DP group, and the intra-TP communication waiting time data of the first accelerator card in the first DP group, the third judgment unit 503 is specifically used to: arrange multiple accelerator cards in descending order of intra-TP communication waiting time data to obtain a second accelerator card sequence; calculate the fourth average and fourth standard deviation of the intra-TP communication waiting time data of the third proportion of accelerator cards in the second accelerator card sequence; obtain the standard value of intra-TP communication waiting time based on the fourth average and fourth standard deviation; and determine whether the first accelerator card is abnormal based on the standard value of intra-TP communication waiting time and the intra-TP communication waiting time data of the first accelerator card; wherein, when the intra-TP communication waiting time data of the first accelerator card is less than the standard value of intra-TP communication waiting time, the first accelerator card is abnormal, and when the intra-TP communication waiting time data of the first accelerator card is greater than or equal to the standard value of intra-TP communication waiting time, the first accelerator card is normal.

[0149] In one possible implementation, regarding determining whether a first accelerator card is abnormal based on the fifth average and fifth standard deviation of the communication duration data within the PP group for each accelerator card in the first DP group, and the communication duration data within the PP group for the first accelerator card in the first DP group, the third judgment unit 503 is specifically used to: arrange multiple accelerator cards in ascending order of their communication duration data within the PP group to obtain a third accelerator card sequence; calculate the fifth average and fifth standard deviation of the communication duration data within the PP group for the fourth proportion of accelerator cards in the third accelerator card sequence; obtain a standard value for the communication duration within the PP group based on the fifth average and fifth standard deviation; and determine whether the first accelerator card is abnormal based on the standard value for the communication duration within the PP group and the communication duration data within the PP group for the first accelerator card, wherein when the communication waiting time data within the PP group for the first accelerator card is greater than the standard value for the communication duration within the PP group, the first accelerator card is abnormal; when the communication waiting time data within the PP group for the first accelerator card is less than or equal to the standard value for the communication duration within the PP group, the first accelerator card is normal.

[0150] In one possible implementation, the third judgment unit 503 is further configured to: detect an anomaly in the first accelerator card in the first DP group, and obtain a standard value for the total execution time based on the sixth average value and the sixth standard deviation of the total execution time data of each function or operator in the first accelerator card on a normal accelerator card; determine that the function whose total execution time data in the first accelerator card is greater than the standard value for the total execution time is a problem function, and determine that the operator whose total execution time data in the first accelerator card is greater than the standard value for the total execution time is a problem operator.

[0151] It is worth noting that the specific functional implementation of the detection device 500 is described in the detection method shown in Figure 2 above. For example, the first judgment unit 501 is used to implement the relevant content of S201, the second judgment unit 502 is used to implement the relevant content of S202, and the third judgment unit 503 is used to implement the relevant content of S203. Each unit or module in the detection device 500 can be individually or entirely merged into one or more other units or modules, or some of the units or modules can be further divided into multiple functionally smaller units or modules. This achieves the same operation without affecting the technical effect of the embodiments of this application. The above-mentioned units or modules are divided according to logical functions. In practical applications, the function of one unit (or module) is implemented by multiple units (or modules), or the function of multiple units (or modules) is implemented by one unit (or module).

[0152] Based on the description of the above method embodiments and related device embodiments, please refer to FIG6, which is a schematic diagram of the structure of a computer provided in an embodiment of this application. The computer 600 shown in FIG6 includes a processor 601, a memory 602, a communication interface 603, and a bus 604. The processor 601, the memory 602, and the communication interface 603 are interconnected through the bus 604.

[0153] Optionally, the memory 602 can be ROM, a static storage device, a dynamic storage device, or RAM.

[0154] The memory 602 is capable of storing executable program code. When the executable program code stored in the memory 602 is executed by the processor 601, the processor 601 and the communication interface 603 are used to execute the various steps of the detection method of the embodiment shown in FIG2 or FIG3.

[0155] The processor 601 employs a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), GPU, NPU, or one or more integrated circuits to execute relevant programs to perform the detection method of the method embodiment of this application.

[0156] The processor 601 can also be an integrated circuit chip with signal processing capabilities. During implementation, each step of the detection method of this application can be completed through the integrated logic circuitry of the hardware in the processor 601 or through software instructions. Optionally, the processor 601 can be a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor is a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. Optional software modules are located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 602. The processor 601 reads the information in memory 602 and, in conjunction with its hardware, completes the functions required by the modules included in the detection device 500 of this application embodiment, or executes the detection method of the embodiment shown in FIG. 2 or FIG. 3 of this application.

[0157] Communication interface 603 uses transceiver-related devices such as, but not limited to, transceivers.

[0158] Bus 604 may include a pathway for transmitting information between various components of computer 600 (e.g., memory 602, processor 601, communication interface 603).

[0159] It should be noted that although the computer 600 shown in Figure 6 only illustrates the memory, processor, and communication interface, those skilled in the art should understand that in specific implementations, the computer 600 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the computer 600 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that the computer 600 may only include the devices necessary for implementing the embodiments of this application, and not necessarily all the devices shown in Figure 6.

[0160] This application provides a computer-readable storage medium storing a computer program for electronic data interchange. The computer program includes execution instructions for performing some or all of the steps of any of the detection methods described in the above-described detection method embodiments. The computer includes an electronic terminal device.

[0161] This application provides a computer program product, which includes a computer program operable to enable a computer to perform some or all of the steps of any of the detection methods described in the above method embodiments. The computer program product may be a software installation package.

[0162] It should be noted that, for the sake of simplicity, each of the aforementioned detection method embodiments is described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.

[0163] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principle and implementation of a detection method and apparatus of this application. The description of the above embodiments is only for the purpose of helping to understand the method and its core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation and application scope based on the idea of ​​a detection method and apparatus of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

[0164] This application is described with reference to flowchart illustrations and / or block diagrams of methods, hardware products, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0165] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams. The storage medium may include: flash drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, etc.

[0166] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce a good effect.

[0167] Those skilled in the art will understand that all or part of the steps in the various methods of any of the above-described detection methods can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage device, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk, or an optical disk, etc.

[0168] It is understood that any product that is controlled or configured to perform the processing method of the flowchart described in an embodiment of a detection method of this application, such as the apparatus and computer program product of the above flowchart, falls within the scope of the related products described in this application.

[0169] Obviously, those skilled in the art can make various modifications and variations to the detection method and apparatus provided in this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

Claims

1. A detection method, characterized in that, In model training scenarios, the server for model training comprises multiple data-parallel processing (DP) groups, each DP group including multiple accelerator cards. The server iteratively trains the model using multiple DP groups, including: Based on the total duration data of the first iteration, determine whether the first iteration is abnormal; If an anomaly is detected in the first iteration, the first DP group is determined to be abnormal based on the total waiting time data of cross-DP group communication of the accelerator card in each DP group in the first iteration. If an anomaly is detected in the first DP group, then the anomaly of the first accelerator card in the first DP group is determined based on the total model training time data of each accelerator card in the first DP group.

2. The method as described in claim 1, characterized in that, The step of determining whether the first iteration is abnormal based on the total duration data of the first iteration includes: The standard value of total duration is obtained based on the first average and first standard deviation of the total duration data from multiple historical iterations; Based on the total duration standard value and the total duration data of the first iteration, it is determined whether the first iteration is abnormal; wherein, when the total duration data of the first iteration is greater than the total duration standard value, the first iteration is abnormal, and when the total duration data of the first iteration is less than or equal to the total duration standard value, the first iteration is normal.

3. The method as described in claim 1 or 2, characterized in that, The step of determining whether the first DP group is abnormal based on the total waiting time data of cross-DP group communication of the accelerator cards in each DP group in the first iteration includes: Arrange multiple DP groups in descending order of the total waiting time for cross-DP group communication of the accelerator card to obtain the first DP group sequence; Calculate the second average and second standard deviation of the total waiting time data of the first proportion of DP group accelerator cards communicating across DP groups in the first DP group sequence. The standard value of total waiting time is obtained based on the second average value and the second standard deviation; Based on the total waiting time standard value and the total waiting time data of the accelerator card communication across DP groups within the first DP group, it is determined whether the first DP group is abnormal; wherein, when the total waiting time data of the accelerator card communication across DP groups within the first DP group is less than the total waiting time standard value, the first DP group is abnormal, and when the total waiting time data of the accelerator card communication across DP groups within the first DP group is greater than or equal to the total waiting time standard value, the first DP group is normal.

4. The method according to any one of claims 1-3, characterized in that, The total training time data for the model includes total computation time data, tensor parallel TP group intra-group communication waiting time data, and pipelined parallel PP group intra-group communication time data. The step of determining whether the first accelerator card in the first DP group is abnormal based on the total model training time data of each accelerator card in the first DP group includes: Based on the third average and third standard deviation of the total computation time data for each accelerator card in the first DP group, and the total computation time data of the first accelerator card in the first DP group, determine whether the first accelerator card is abnormal; or, Based on the fourth average and fourth standard deviation of the intra-TP communication waiting time data for each accelerator card in the first DP group, and the intra-TP communication waiting time data for the first accelerator card in the first DP group, determine whether the first accelerator card is abnormal; or, Based on the fifth average value and fifth standard deviation of the communication duration data within the PP group of each accelerator card in the first DP group, and the communication duration data within the PP group of the first accelerator card in the first DP group, determine whether the first accelerator card is abnormal.

5. The method as described in claim 4, characterized in that, The step of determining whether the first accelerator card is abnormal based on the third average and third standard deviation of the total computation time data of each accelerator card in the first DP group, and the total computation time data of the first accelerator card in the first DP group, includes: Arrange multiple accelerator cards in ascending order of total computation time to obtain the first accelerator card sequence; Calculate the third average and third standard deviation of the total computation time data of the second proportion of accelerator cards in the first accelerator card sequence; The standard value for the total duration is obtained based on the third average and the third standard deviation. Based on the standard value of total computation time and the total computation time data of the first accelerator card in the first DP group, it is determined whether the first accelerator card is abnormal; wherein, when the total computation time data of the first accelerator card is greater than the standard value of total computation time, the first accelerator card is abnormal, and when the total computation time data of the first accelerator card is less than or equal to the standard value of total computation time, the first accelerator card is normal.

6. The method as described in claim 4 or 5, characterized in that, The step of determining whether the first accelerator card is abnormal based on the fourth average and fourth standard deviation of the intra-TP communication waiting time data of each accelerator card in the first DP group, and the intra-TP communication waiting time data of the first accelerator card in the first DP group, includes: The multiple accelerator cards are arranged in descending order of communication waiting time data within the TP group to obtain the second accelerator card sequence; Calculate the fourth average and fourth standard deviation of the intra-TP communication waiting time data of the third proportion of accelerator cards in the second accelerator card sequence; The standard value of communication waiting time within the TP group is obtained based on the fourth average value and the fourth standard deviation. Based on the standard value of the communication waiting time within the TP group and the communication waiting time data within the TP group of the first accelerator card, it is determined whether the first accelerator card is abnormal; wherein, when the communication waiting time data within the TP group of the first accelerator card is less than the standard value of the communication waiting time within the TP group, the first accelerator card is abnormal, and when the communication waiting time data within the TP group of the first accelerator card is greater than or equal to the standard value of the communication waiting time within the TP group, the first accelerator card is normal.

7. The method according to any one of claims 4-6, characterized in that, The step of determining whether the first accelerator card is abnormal based on the fifth average value and fifth standard deviation of the communication duration data within the PP group of each accelerator card in the first DP group, and the communication duration data within the PP group of the first accelerator card in the first DP group, includes: The multiple accelerator cards are arranged in ascending order of communication duration data within the PP group to obtain the third accelerator card sequence; Calculate the fifth average and fifth standard deviation of the communication duration data within the PP group of the fourth proportion of accelerator cards in the third accelerator card sequence; The standard value of communication duration within the PP group is obtained based on the fifth average value and the fifth standard deviation. Based on the standard value of the communication duration within the PP group and the communication duration data within the PP group of the first accelerator card, it is determined whether the first accelerator card is abnormal. Specifically, when the communication duration data within the PP group of the first accelerator card is greater than the standard value of the communication duration within the PP group, the first accelerator card is abnormal. When the communication duration data within the PP group of the first accelerator card is less than or equal to the standard value of the communication duration within the PP group, the first accelerator card is normal.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: If an anomaly is detected in the first accelerator card in the first DP group, the standard value of the total execution time is obtained based on the sixth average and the sixth standard deviation of the total execution time data of each function or operator in the first accelerator card on the normal accelerator card. Functions whose total execution time in the first accelerator card is greater than the standard value of total execution time are identified as problem functions, and operators whose total execution time in the first accelerator card is greater than the standard value of total execution time are identified as problem operators.

9. A detection device, characterized in that, Applied to model training scenarios, the model training server contains multiple data-parallel processing (DP) groups, each DP group includes multiple accelerator cards. The server iteratively trains the model through multiple DP groups. The detection device includes: The first judgment unit is used to determine whether the first iteration is abnormal based on the total duration data of the first iteration; The second judgment unit is used to detect the first iteration abnormality and then determine whether the first DP group is abnormal based on the total waiting time data of the acceleration card cross-DP group communication in each DP group in the first iteration. The third judgment unit is used to detect an anomaly in the first DP group and then determine whether the first accelerator card in the first DP group is abnormal based on the total model training time data of each accelerator card in the first DP group.

10. A computer, characterized in that, The computer includes a memory, a processor, and executable program code stored in the memory and executable on the processor, wherein the processor executes the executable program code and performs the steps of the method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by the electronic device, implement the steps of the method as described in any one of claims 1-8.

12. A computer program product, characterized in that, The computer program product includes a computer program for causing a computer to perform the steps of the method as described in any one of claims 1-8.

Images