Silent data corruption detection method in model training process, and model training system

Abstract

The present application relates to the technical field of AI. Disclosed are a silent data corruption detection method in a model training process, and a model training system. In the method, a plurality of operators sent by a host to an accelerator card comprise a detection operator, which is configured to detect, on the basis of input data of a communication operator, whether silent data corruption has occurred on the accelerator card. On this basis, when executing a computation operator to acquire the input data of the communication operator, the accelerator card executes the detection operator to acquire an execution result of the detection operator, and then determines, by means of the execution result of the detection operator, whether silent data corruption has occurred on the accelerator card. In this way, before the accelerator card executes the communication operator, it is determined whether silent data corruption has occurred on the current accelerator card, thereby preventing the propagation of silent data corruption between accelerator cards, i.e., preventing silent data corruption from propagating throughout the AI model, and thus improving model training accuracy.

Description

Silent data error detection method and model training system during model training

[0001] This application claims priority to Chinese Patent Application No. 202510070211.0, filed on January 15, 2025, entitled "Method for Detecting Silent Data Errors and Model Training System in Model Training Process", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method for detecting silent data errors during model training and a model training system. Background Technology

[0003] Silent data corruption (SDC) refers to the phenomenon where data corruption occurs in a system without any alerts or error messages. Currently, as AI model training systems continue to scale up, the frequency of SDC during model training is increasing significantly.

[0004] In related technologies, error-correcting code (ECC) technology is used to prevent SDC from occurring during model training. For example, when transmitting or storing data related to AI models, the data is encoded based on the ECC algorithm to generate redundant check bits. When a single-bit error is detected in the data, the redundant check bits are used to automatically correct the erroneous data, so that subsequent computing units (such as accelerator cards) receive the correct data.

[0005] However, the above methods can only prevent SDCs that occur during data transmission or storage, but cannot detect SDCs that occur during computation. This means that the transmitted or stored data may be erroneous data obtained after computation, and the SDCs that occur during computation can propagate to the entire AI model, affecting the model training accuracy. Summary of the Invention

[0006] This application provides a method and system for detecting silent data errors during model training, which can prevent silent data errors from propagating to the entire AI model and improve model training accuracy.

[0007] Firstly, a silent data error detection method is provided during model training, applied to a model training system including a host computer and an accelerator card. The method includes:

[0008] The host sends multiple operators to the accelerator card to train an artificial intelligence (AI) model. These operators include computation operators, communication operators, and detection operators. The communication operators are used to transmit the execution results of the computation operators between different accelerator cards, and the detection operators are used to detect whether silent data errors have occurred on the accelerator card based on the input data of the communication operators.

[0009] The accelerator card executes computation operators and obtains the input data for communication operators based on the execution results of the computation operators;

[0010] The accelerator card executes the detection operator based on the input data of the communication operator and obtains the execution result of the detection operator.

[0011] The AI ​​model involved in this application is used to perform reasoning on input data to output reasoning results. Application areas of the AI ​​model include text reasoning, image reasoning, video reasoning, audio reasoning, etc. In the above method, the host sends multiple operators to the accelerator card, including a detection operator. This detection operator is used to detect whether a silent data error has occurred on the accelerator card based on the input data of the communication operator. Based on this, when the accelerator card executes the computation operator to obtain the input data of the communication operator, it executes the detection operator to obtain the execution result of the detection operator. Then, the execution result of the detection operator is used to determine whether a silent data error has occurred on the accelerator card. Thus, before the accelerator card executes the communication operator, it is determined whether a silent data error has occurred, preventing the propagation of silent data errors between accelerator cards, that is, preventing silent data errors from spreading to the entire AI model, thereby improving the model training accuracy. Moreover, this method can accurately locate the accelerator card where a silent data error has occurred, providing technical support for subsequent fault-tolerance measures for silent data errors.

[0012] In some embodiments, the host sends multiple operators to the accelerator card, including: during the backpropagation phase of the AI ​​model training process, the host sends multiple operators to the accelerator card, and the input data of the communication operators is the activation value gradient of the network layer in the AI ​​model; the accelerator card executes detection operators according to the input data of the communication operators and obtains the execution result of the detection operators, including: the accelerator card executes detection operators according to the activation value gradient and obtains the execution result of the detection operators.

[0013] The above method involves performing silent data error detection during the backpropagation phase of the AI ​​model training process. Since SDC (Silent Data Error) during the backpropagation phase can have a significant impact on the model training results, detecting silent data errors that may occur during the backpropagation phase of the AI ​​model can save computing resources and improve detection performance while preventing silent data errors from spreading to the entire AI model.

[0014] In some embodiments, the accelerator card executes a detection operator based on the activation value gradient to obtain the execution result of the detection operator, including: the accelerator card determines the statistical information of the activation value gradient, the statistical information of the activation value gradient indicating the distribution characteristics of the numerical values ​​in the activation value gradient; the accelerator card determines the execution result of the detection operator based on the relationship between the statistical information of the activation value gradient and a threshold; wherein, if the statistical information of the activation value gradient is greater than the threshold, the execution result of the detection operator indicates that a silent data error has occurred on the accelerator card, and if the statistical information of the activation value gradient is less than or equal to the threshold, the execution result of the detection operator indicates that no silent data error has occurred on the accelerator card.

[0015] In the above method, the statistical information of the activation value gradient includes, for example, the norm, variance, maximum value, or standard deviation of the activation value gradient. Since the statistical information of the activation value gradient usually exhibits certain patterns at different stages of training, when SDC (Signal Distortion Cancellation) occurs, the statistical information of the activation value gradient will show abnormal peaks or changes compared to the threshold, indicating an error in the data or calculation process that disrupts the normal training state. Therefore, the above method can accurately determine whether SDC has occurred on the accelerator card. Furthermore, since gradient calculation is usually involved in AI model training, the above method has strong versatility and can be applied to various neural network architectures.

[0016] In some embodiments, the method further includes: the accelerator card calculates a threshold based on historical statistical information and probability inequalities, wherein the historical statistical information refers to the statistical information of historical activation value gradients obtained by the accelerator card when executing communication operators in the historical iterations of the AI ​​model; if the execution result of the detection operator indicates that no silent data error has occurred on the accelerator card, the accelerator card updates the historical statistical information based on the statistical information of activation value gradients.

[0017] By combining probability inequalities and historical statistical information to calculate the threshold in the above manner, we can continuously collect statistical information on the new activation value gradient as the iteration progresses. This helps to capture the dynamic changes and trends of statistical information and improve the accuracy of the detection operator's execution results.

[0018] In some embodiments, the method further includes: if the execution result of the detection operator indicates that a silent data error has occurred on the accelerator card, the accelerator card replaces the activation value gradient with 0 and executes the communication operator based on the replaced activation value gradient.

[0019] In this way, the accelerator card replaces the activation value gradient with 0 before transmitting it to other accelerator cards, avoiding errors caused by SDC propagation. Moreover, since the activation value gradient has been suppressed, silent data errors will no longer be detected on other accelerator cards. At this point, the accelerator card that issues the silent data error alarm is unique, and the alarming accelerator card is also the one where the silent data error occurred, achieving precise SDC card-level location.

[0020] In some embodiments, the method further includes: during the forward propagation phase of the AI ​​model training process, the host labels data to participate in the backpropagation phase, and when calculating the activation value gradient based on the labeled data during the backpropagation phase, sets the value of the first variable to a target value; during the backpropagation phase of the AI ​​model training process, the host sends multiple operators to the accelerator card, including: when the host detects that the value of the first variable is the target value during the backpropagation phase, the host sends multiple operators to the accelerator card.

[0021] The above method enables the host to identify the timing when to send detection operators during the backpropagation phase, providing technical support for the host to send detection operators to the accelerator card during the backpropagation phase.

[0022] In some embodiments, the host sends multiple operators to the accelerator card, including: the host sending detection operators and communication operators to the first task queue of the accelerator card, and sending computation operators to the second task queue of the accelerator card.

[0023] The first task queue can be understood as the communication flow on the accelerator card, and the second task queue can be understood as the computing flow on the accelerator card. Typically, the computing flow and communication flow on the accelerator card can be processed in parallel. In this way, on the one hand, performance loss can be reduced, and on the other hand, it can be ensured that the accelerator card executes the detection operator before executing the communication operator.

[0024] In some embodiments, the method further includes: if the execution result of the detection operator indicates that a silent data error has occurred on the accelerator card, the accelerator card performs processing actions for this silent data error based on the accelerator card's operating mode, the operating mode including at least one of the following:

[0025] The first working mode refers to the accelerator card recording an error in this silent data recording.

[0026] The second working mode refers to the accelerator card recording an error in the silent data, sending an alarm to the host and interrupting the training.

[0027] The third working mode refers to the accelerator card recording the current silent data error and the execution status of the accelerator card, alarming the host and interrupting the training.

[0028] In some embodiments, the host stores a second variable that indicates the type of operating mode on the accelerator card.

[0029] The above methods provide a variety of working modes on the accelerator card, which can be selected according to different application scenarios and system requirements.

[0030] Secondly, a silent data error detection method is provided during model training, applied to the host computer in a model training system, which also includes an accelerator card. The method includes:

[0031] Multiple operators are sent to the accelerator card to train an artificial intelligence (AI) model. These multiple operators include a computation operator, a communication operator, and a detection operator. The communication operator is used to transmit the execution results of the computation operator between different accelerator cards. The detection operator is used to detect whether a silent data error has occurred on the accelerator card based on the input data of the communication operator. The execution order of the detection operator is before the execution order of the communication operator.

[0032] In some embodiments, multiple operators are sent to the accelerator card, including:

[0033] During the backpropagation phase of the AI ​​model training process, multiple operators are sent to the accelerator card. The input data of the communication operators are the activation gradients of the network layers in the AI ​​model.

[0034] In some embodiments, the method further includes:

[0035] During the forward propagation phase of AI model training, data is labeled for participation in the backpropagation phase, and when calculating the activation value gradient based on the labeled data during the backpropagation phase, the value of the first variable is set to the target value.

[0036] During the backpropagation phase of the AI ​​model training process, multiple operators are sent to the accelerator card, including: when the value of the first variable is detected as the target value during the backpropagation phase, multiple operators are sent to the accelerator card.

[0037] In some embodiments, multiple operators are sent to the accelerator card, including:

[0038] The detection and communication operators are sent to the first task queue of the accelerator card, and the computation operator is sent to the second task queue of the accelerator card.

[0039] In some embodiments, the host stores a second variable that indicates the operating mode of the accelerator card in the event of a silent data error; the operating mode includes at least one of the following:

[0040] The first working mode refers to the accelerator card recording an error in this silent data recording.

[0041] The second working mode refers to the accelerator card recording an error in the silent data, sending an alarm to the host and interrupting the training.

[0042] The third working mode refers to the accelerator card recording the current silent data error and the execution status of the accelerator card, alarming the host and interrupting the training.

[0043] Thirdly, a silent data error detection method is provided during model training, applied to an accelerator card in a model training system, which also includes a host computer. The method includes:

[0044] The receiver sends multiple operators for training an AI model, including a computation operator, a communication operator, and a detection operator. The communication operator is used to transmit the execution results of the computation operator between different accelerator cards, and the detection operator is used to detect whether a silent data error has occurred on the accelerator card based on the input data of the communication operator.

[0045] Execute the computation operator and obtain the input data of the communication operator based on the execution result of the computation operator;

[0046] The detection operator is executed based on the input data of the communication operator, and the execution result of the detection operator is obtained.

[0047] In some embodiments, the receiving host sends multiple operators for training an AI model, including:

[0048] During the backpropagation phase of the AI ​​model training process, multiple operators are received, and the input data of the communication operators are the activation value gradients of the network layers in the AI ​​model.

[0049] The detection operator is executed based on the input data of the communication operator, and the execution result of the detection operator is obtained, including: executing the detection operator based on the activation value gradient and obtaining the execution result of the detection operator.

[0050] In some embodiments, a detection operator is executed based on the activation value gradient to obtain the execution result of the detection operator, including:

[0051] Determine the statistical information of the activation value gradient, which indicates the distribution characteristics of the numerical magnitudes in the activation value gradient;

[0052] The execution result of the detection operator is determined based on the statistical information of the activation value gradient and the relationship between the threshold.

[0053] If the statistical message of the activation value gradient is greater than the threshold, the execution result of the detection operator indicates that a silent data error has occurred on the accelerator card; if the statistical message of the activation value gradient is less than or equal to the threshold, the execution result of the detection operator indicates that no silent data error has occurred on the accelerator card.

[0054] In some embodiments, the method further includes:

[0055] Based on historical statistical information and probability inequalities, a threshold is calculated. The historical statistical information refers to the statistical information of the historical activation value gradient obtained by the accelerator card when executing the communication operator in the historical iterations of the AI ​​model.

[0056] If the execution result of the detection operator indicates that no silent data error has occurred on the accelerator card, the historical statistics are updated based on the statistics of the activation value gradient.

[0057] In some embodiments, the method further includes:

[0058] If the execution result of the detection operator indicates a silent data error on the accelerator card, the activation value gradient is replaced with 0, and the communication operator is executed based on the replaced activation value gradient.

[0059] In some embodiments, the receiving host sends multiple operators for training an AI model, including:

[0060] The detection and communication operators are received through the first task queue on the accelerator card, and the computation operators are received through the second task queue on the accelerator card.

[0061] In some embodiments, the method further includes:

[0062] If the execution result of the detection operator indicates that a silent data error has occurred on the accelerator card, the processing action for this silent data error is executed based on the accelerator card's operating mode, which includes at least one of the following:

[0063] The first working mode refers to the accelerator card recording an error in this silent data recording.

[0064] The second working mode refers to the accelerator card recording an error in the silent data, sending an alarm to the host and interrupting the training.

[0065] The third working mode refers to the accelerator card recording the current silent data error and the execution status of the accelerator card, alarming the host and interrupting the training.

[0066] Fourthly, a model training system is provided, comprising a host computer and an accelerator card, wherein the host computer controls the accelerator card, and the model training system is used to implement the silent data error detection method in the model training process provided by the first aspect or any possible implementation thereof.

[0067] Fifthly, a processing chip is provided that is communicatively connected to an accelerator card. The processing chip is used to control the accelerator card and to implement the silent data error detection method during model training provided in the second aspect or any possible implementation thereof.

[0068] In a sixth aspect, an acceleration chip is provided, which is communicatively connected to a host computer. The acceleration chip is used to accept control from the host computer, and the processing chip is used to implement the silent data error detection method in the model training process provided by the third aspect or any possible implementation of the third aspect.

[0069] In a seventh aspect, a computing device is provided, the computing device including a processor and a memory, the processor being configured to execute at least a segment of program code stored in the memory, so that the computing device implements the silent data error detection method in the model training process provided by the first aspect or any possible implementation thereof.

[0070] Eighthly, a computer-readable storage medium is provided for storing at least one piece of program code, which, when executed by a computing device, causes the computing device to implement the silent data error detection method during model training as provided in the first aspect or any possible implementation thereof. The storage medium includes, but is not limited to, volatile memory, such as random access memory, and non-volatile memory, such as flash memory, hard disk drive (HDD), and solid-state drive (SSD).

[0071] Ninthly, a computer program product is provided that, when run on a computing device, enables the computing device to implement the silent data error detection method during model training as provided in the first aspect or any possible implementation thereof. The computer program product can be a software installation package, which can be downloaded and executed on the computing device when the aforementioned AI model training method needs to be implemented.

[0072] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0073] Figure 1 is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0074] Figure 2 is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0075] Figure 3 is a schematic diagram of the structure of an accelerator card provided in an embodiment of this application;

[0076] Figure 4 is a schematic diagram of the functional architecture of a model training system provided in an embodiment of this application;

[0077] Figure 5 is a schematic diagram of a model training system performing silent data error detection according to an embodiment of this application;

[0078] Figure 6 is a schematic diagram of the architecture of a model training system provided in an embodiment of this application;

[0079] Figure 7 is a flowchart of a silent data error detection method during model training provided in an embodiment of this application;

[0080] Figure 8 is a schematic diagram of a backpropagation identification method provided in an embodiment of this application;

[0081] Figure 9 is a schematic diagram of a communication operator in the backpropagation stage provided in an embodiment of this application;

[0082] Figure 10 is a schematic diagram of a silent data error detection method in the model training process provided in an embodiment of this application;

[0083] Figure 11 is a schematic diagram of a silent data error detection device in the model training process provided in an embodiment of this application;

[0084] Figure 12 is a schematic diagram of another silent data error detection device in the model training process provided in an embodiment of this application. Detailed Implementation

[0085] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application are authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the training requests for AI models and the input data of operators involved in this application are obtained with full authorization.

[0086] To facilitate understanding, the key terms and concepts involved in this application will be explained below.

[0087] Artificial intelligence (AI) models are a class of mathematical algorithm models that use machine learning concepts to solve practical problems. Typically, AI models include a large number of parameters and calculation formulas (or calculation rules).

[0088] An accelerator card, also known as an acceleration chip, acceleration device, accelerator, or computing card, is a specialized hardware device or computer system designed to accelerate computation in AI scenarios. In this application, an accelerator card may include, for example, a graphics processing unit (GPU), a neural network processing unit (NPU), an intelligent processing unit (IPU), a tensor processing unit (TPU), or a domain-specific architecture (DSA) chip, and is not limited to these.

[0089] An operator (OP) is a computational unit or function that runs on a computing device. In the field of deep learning, neural network layers and even the entire model are composed of operators, which correspond to the computational logic within the neural network layers. For example, a convolutional layer is an operator; the weight summation process in a fully-connected layer (FC layer) is also an operator.

[0090] In this application, a tensor refers to the data of an operator during model training, such as the input data, output data, and intermediate data of an operator.

[0091] Collective communication, also known as aggregate communication, is a widely used communication pattern in parallel and distributed computing. It enables data transfer between different processes in a distributed computing environment. Collective communication typically involves multiple devices (such as accelerator cards) simultaneously interacting with each other. Specific communication primitives (or operators) coordinate data transfer, reduction, and synchronization among these nodes to achieve collaborative progress of the overall task in distributed computing. Common communication operators in collective communication include all-to-all, broadcast, barrier synchronization, reduce, scatter, gather, all-gather, reduce-scatter, all-reduce, send, and receive.

[0092] Silent data corruption (SDC) refers to the phenomenon where data corruption occurs in a system without any alarms or error messages. Also known as silent faults, silent data corruption can occur during data transmission, computation, or storage. However, its occurrence is relatively random and difficult to detect, making it easy for the system to use erroneous (or corrupted) data for processing, resulting in incorrect data processing results (such as training an AI model with low accuracy or even non-convergence). For example, in a model training system, if the storage unit used to store AI model-related data (such as AI model parameters) is damaged (e.g., memory causing bit flipping due to electromagnetic interference or voltage disturbances, or disk aging or magnetic decay), the data stored in the storage unit becomes corrupted, and the system uses this corrupted data for computation, resulting in incorrect calculation results. For example, when an accelerator card in a model training system executes operators related to an AI model, the data may be corrupted due to voltage disturbances, high-energy particle impacts, circuit aging, or transistor etching. This corrupted data can then lead to incorrect calculation results. Alternatively, data may be corrupted during transmission due to electromagnetic interference or other external disturbances, causing the accelerator card to use this corrupted data and obtain incorrect calculation results.

[0093] Hook functions, or hooks, are mechanisms in deep learning used to obtain, modify, or monitor intermediate variables (such as parameter weights and outputs) of a model. They allow users to insert custom functions at specific stages of model execution to perform certain operations. Illustratively, by hooking parameters onto their weights, users can obtain the values ​​of the parameters at each forward propagation. For example, when training a deep neural network, registering hooks on the weights of a convolutional layer allows obtaining the weights of the convolutional kernel at each forward propagation. Furthermore, registering hooks on the model's output allows obtaining the model's predictions. For instance, registering a hook on the output of a network layer allows obtaining the output of that network layer.

[0094] The application scenarios of this application are described below.

[0095] This application is applied to AI model training scenarios, where the AI ​​model is used to perform reasoning on input data to output reasoning results. The application areas of the AI ​​model include text reasoning, image reasoning, video reasoning, audio reasoning, and so on. For example, when applied to text reasoning, the input data of the AI ​​model is text, and the AI ​​model is used to analyze the text to output a text summary; or, the AI ​​model is used to reason on question text to output the answer to the question, and so on. As another example, when applied to image reasoning scenarios, the input data of the AI ​​model is an image, and the AI ​​model is used to recognize the image to output an image recognition result. Furthermore, the input data can also be audio or video, and is not limited to these. The AI ​​model training scenarios involved in this application can also employ distributed parallel strategies to implement AI model training. These distributed parallel strategies include, for example, data parallelism, tensor parallelism, pipeline parallelism, or optimizer parallelism. Pipeline parallelism includes, but is not limited to, naive pipeline parallelism, virtual pipeline parallelism, micro-batch pipeline parallelism, and so on.

[0096] As AI model training systems continue to scale up, the frequency of Service Defects (SDCs) during model training increases significantly. Related technologies employ error-correcting codes (ECCs) to prevent SDCs during model training. For example, when transmitting or storing AI model data, the data is encoded using an ECC algorithm to generate redundant check bits. When a single-bit error is detected, the redundant check bits automatically correct the erroneous data, ensuring that subsequent computing units (such as accelerator cards) receive the correct data. However, this method only prevents SDCs occurring during data transmission or storage; it cannot detect SDCs occurring during computation. This means that the transmitted or stored data may contain erroneous data obtained after computation, allowing SDCs occurring during computation to propagate throughout the entire AI model and affect training accuracy.

[0097] Based on this, this application provides a method for detecting silent data errors during model training. By detecting the input data of the communication operator during model training, it can promptly determine whether any accelerator cards in the model training system have experienced silent data errors. This allows for the determination of whether a silent data error has occurred on an accelerator card before the communication operator is executed, preventing the propagation of silent data errors between accelerator cards and thus preventing them from spreading to the entire AI model. Furthermore, this method can accurately locate the accelerator card experiencing the silent data error, providing technical support for subsequent fault-tolerance measures. For example, during the backpropagation phase of an AI model, accelerator cards typically execute communication operators to transmit activation value gradients between different accelerator cards. By detecting whether the gradient of the activation value to be transmitted is abnormal, it can be determined whether a silent data error (SDC) has occurred on the accelerator card. If an abnormal gradient of the activation value to be transmitted is detected, it indicates that an SDC has occurred on that accelerator card when calculating the activation value gradient. In this case, by issuing an alarm or interrupting training, it is possible to determine whether the current accelerator card has experienced an SDC before the communication operator is executed, preventing the SDC from spreading to the entire AI model.

[0098] The implementation environment of this application is described below.

[0099] Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of this application. As shown in Figure 1, the implementation environment includes a model training system 100, which includes at least one computing node. Each computing node includes a host 101 and an accelerator card 102, and the host 101 and the accelerator card 102 are communicatively connected. In some embodiments, the accelerator card 102 is also referred to as an acceleration device, accelerator, acceleration chip, computing card, training card, etc., and this application is not limited thereto. In addition, the number of computing nodes, hosts, and accelerator cards shown in the figure are illustrative and do not constitute a limitation on this application. For example, the number of computing nodes can be one or more, and the number of accelerator cards in each computing node can be one or more, etc.

[0100] The model training system 100 is used to provide AI model training services. An AI model refers to a model that includes one or more network layers; this application does not limit the type of AI model. Taking a large language model as an example, a large language model is typically based on a Transformer architecture, including multiple Transformer layers, also known as Transformer blocks. A Transformer block refers to the basic building block in a Transformer model, which may include components such as multi-head self-attention, a feedforward network, normalization, and residual connections. Multiple Transformer blocks are stacked sequentially to form a complete Transformer model, enabling effective feature extraction and representation learning of the input sequence. The computing nodes in the model training system 100 can jointly deploy the large language model. For example, a distributed parallel strategy can be used, deploying multiple Transformer layers of the large language model on different computing nodes. This application does not limit the distributed parallel strategy.

[0101] For any computing node in the model training system 100, the host 101 on that computing node can control the accelerator card 102 on that computing node to execute relevant operators involved in the AI ​​model training process. Illustratively, in response to the training request of the AI ​​model, the host sends the operators corresponding to the training stage (including forward propagation and back propagation stages) of the AI ​​model to the accelerator card to train the AI ​​model. This process can also be understood as loading the AI ​​model's training task into the accelerator card 102 for execution. The accelerator card 102 is used to provide computing power for the AI ​​model's training task to accelerate the AI ​​model's training process. The accelerator card 102 can be a GPU, NPU, IPU, TPU, DSA chip, etc., and this application is not limited to these. For example, the accelerator card 102 receives operators sent by the host 101, processes the input data of the operators, and obtains output data.

[0102] The forward and backward propagation phases of the AI ​​model both involve communication and computation operators. Communication operators are used to transmit data between different accelerator cards, ensuring collaborative operation between the various accelerator cards and computing nodes in the model training system 100. Computation operators are used to perform various numerical calculations and logical operations on the input data. Examples of computation operators include matrix multiplication, convolution, and matrix addition, among others.

[0103] In this embodiment, the model training system 100 also provides a silent error detection function during model training. During model training, the host 101 on the computing node can control the accelerator card 102 to execute a detection operator to determine whether a silent data error has occurred on the accelerator card 102. The detection operator is used to detect whether a silent data error has occurred on the accelerator card 102 based on the input data of the communication operator during model training. The type of communication operator is described above and will not be repeated here. It should be noted that when the number of parameters and training data of the AI ​​model is large, in order to complete training on an accelerator card with limited video memory, a training cluster composed of a large number of accelerator cards is needed, and the AI ​​model and training data are split and distributed across various accelerator cards. The training cluster exchanges data between different accelerator cards through aggregated communication, thereby ensuring that the final training result is consistent with the training result on an ideal accelerator card with a large amount of video memory. During aggregated communication, inter-card propagation of SDC (Software Divided Data) may occur. Therefore, by detecting SDC (Software-Defined Dispute) on accelerator cards based on the input data of communication operators in the training cluster, it is possible to monitor SDCs that affect model training results, prevent SDC propagation between accelerator cards, and achieve card-level localization of SDCs. Of course, once the accelerator card experiencing SDC is located, the compute node where the SDC occurred can also be quickly located, providing technical support for relevant personnel to implement fault-tolerant measures for the compute node experiencing SDC.

[0104] In some embodiments, the computing nodes in the model training system 100 are general-purpose physical servers, or computing devices such as desktop computers and terminal devices capable of deploying AI models. For example, taking a computing node as a server, the computing node includes a host 101 and multiple accelerator cards 102. The host 101 and the accelerator cards 102 are connected via a peripheral component interconnect express (PCIe) link, and the host 101 and the accelerator cards 102 interact with each other via the PCIe link. The various accelerator cards 102 are connected via a high-speed interconnect link (or chip bus), such as Huawei cache coherent system (HCCS), RDMA over converged Ethernet (RoCE), NVIDIA Link, compute express link (CXL), universal chiplet interconnect express (UCIe), cache coherent interconnect for accelerators (CCIX), etc., and this application is not limited to these.

[0105] In some embodiments, the model training system 100 is deployed on a cloud platform, and the computing nodes are virtualized nodes such as virtual machines or containers. In this case, the virtual machines or containers can be deployed in at least one computing device (e.g., a server). Here, cloud platform is short for cloud computing platform, referring to a service that provides computing, networking, and storage capabilities based on hardware and software resources. Through the network "cloud," massive amounts of data are processed and analyzed remotely before being returned to the user, featuring large scale, distributed nature, virtualization, high availability, scalability, on-demand service, and security. Cloud platforms can achieve rapid provisioning and release of configurable computing resources with relatively low management costs or low interaction complexity between users and service providers.

[0106] In some embodiments, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The networks are typically Transmission Control Protocol / Internet Protocol (TCP / IP) networks and RDMA networks in data center networks, such as RDMA over Converged Ethernet (RoCE) networks, InfiniBand (IB) networks, etc., and are not limited thereto. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0107] Based on the implementation environment shown in Figure 1, this application provides a computing device that can be configured as a computing node in the model training system 100 described above. Accordingly, the implementation environment can be implemented by one or more computing devices. For example, the model training system can be deployed on multiple computing devices, each configured with a host and at least one accelerator card. Alternatively, the model training system can be deployed on a single computing device, which is configured with a host and multiple accelerator cards. The structure of the computing device will now be described with reference to Figure 2.

[0108] Figure 2 is a schematic diagram of a computing device provided in an embodiment of this application. As shown in Figure 2, the computing device 200 includes a memory 201, a processor 202, a communication interface 203, a bus 204, and an accelerator card 205. The memory 201, processor 202, communication interface 203, and accelerator card 205 are interconnected via the bus 204.

[0109] The memory 201 can be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or it can be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. In this application, the memory 201 is used to store at least one piece of program code. When the program code stored in the memory 201 is executed by the processor 202, the processor 202 is used to perform the steps executed by the host in the silent data error detection method during the model training process described below.

[0110] The processor 202 may be a network processor (NP), a central processing unit (CPU), an application-specific integrated circuit (ASIC), or an integrated circuit used to control the execution of the program of the present application. The processor 202 may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. There may be one or more processors 202.

[0111] The communication interface 203 uses a transceiver module, such as a transceiver, to enable communication between the computing device 200 and other devices or communication networks. For example, data can be acquired through the communication interface 203.

[0112] The memory 201 and the processor 202 can be set separately or integrated together.

[0113] Bus 204 may include a pathway for transmitting information between various components of computing device 200 (e.g., memory 201, processor 202, communication interface 203, accelerator card 205).

[0114] Accelerator card 205 may be, for example, a GPU, NPU, IPU, TPU, DSA chip, etc., and this application does not limit it to this. In this application, accelerator card 205 is used to perform the steps executed by the accelerator card in the silent data error detection method during the model training process described below. Schematic, the structure of accelerator card 205 is shown in FIG3, which is a schematic diagram of the structure of an accelerator card provided in an embodiment of this application. As shown in FIG3, accelerator card 205 includes a communication interface 301, an AI processing core 302, a processor 303, a memory 304, and a bus 305. The communication interface 301, AI processing core 302, processor 303, and memory 304 are interconnected via the bus 305.

[0115] The communication interface 301 is used to provide program instructions and / or data to the AI ​​processing core 302. The communication interface 301 includes a PCIe communication interface, other general peripheral interfaces, etc., which are not limited in this application. For example, data exchange can be achieved between the processor 202 in the computing device 200 and the PCIe communication interface. Another example is that the accelerator card 205 can communicate with other devices or communication networks through the peripheral interface.

[0116] The AI ​​processing core 302 is used to perform matrix calculations, vector calculations, and scalar calculations during the AI ​​model training process.

[0117] The processor 303 can be a CPU, an ASIC, or an integrated circuit used to control the execution of the program in this application. The processor 303 can be a single-core processor or a multi-core processor. The number of processors 303 can be one or more. Taking a multi-core processor as an example, the multiple cores can be functionally divided into a control CPU dedicated to controlling the overall operation of the accelerator card 205 and an AI CPU dedicated to handling complex non-matrix calculations. The number of CPU cores occupied by these two types of tasks can be dynamically allocated by the software according to the actual system operation; this application does not limit this allocation.

[0118] Memory 304 includes, but is not limited to, high bandwidth memory (HBM), double data rate memory (DDR), static random-access memory (SRAM), or other types of dynamic storage devices capable of storing information and instructions, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer.

[0119] Bus 305 may include a pathway for transmitting information between various components of accelerator card 205 (e.g., communication interface 301, AI processing core 302, processor 303, memory 304).

[0120] This application also provides a computing device cluster, which includes multiple computing devices connected to each other via a network. The network can be a wide area network (WAN) or a local area network (LAN), etc. This computing device cluster is used to implement the silent data error detection method provided in this application during model training.

[0121] The following section introduces the silent data error detection method provided in this application during the model training process.

[0122] For ease of understanding, please refer to Figure 4 first, and take the aforementioned model training system as an example to introduce the principle of this application. Figure 4 is a functional architecture diagram of a model training system provided in an embodiment of this application. As shown in Figure 4, the model training system is used to provide model training function 401, detection operator sending function 402, detection operator execution function 403, and fault handling function 404.

[0123] The model training function 401 is implemented collaboratively by the host and accelerator card in the model training system. Taking any computing node in the model training system as an example, the host on the computing node responds to the training request of the AI ​​model and sends the operator corresponding to the training stage (including the forward propagation stage and the back propagation stage) to the accelerator card to train the AI ​​model.

[0124] The detection operator sending function 402 is implemented by the host in the model training system. As described above, in this application, the detection operator is used to detect whether a silent data error has occurred on the accelerator card based on the input data of the communication operator during model training. Schematic, when the host sends a communication operator to the accelerator card, it also sends a detection operator for that communication operator. This allows the accelerator card to execute the detection operator based on the input data of the communication operator before executing the communication operator, thus obtaining the execution result of the detection operator. The accelerator card can then determine whether a silent data error has occurred based on the execution result of the detection operator. The host can send both the communication operator and the detection operator simultaneously, or they can be sent sequentially according to the execution order of the communication operator and the detection operator.

[0125] In some embodiments, the host sends detection operators to the accelerator card during both the forward and backward propagation phases of the AI ​​model. That is, it detects silent data errors that may occur during the entire training process of the AI ​​model, thus preventing silent data errors from propagating to the entire AI model.

[0126] In other embodiments, the host sends detection operators to the accelerator card during the backpropagation phase of the AI ​​model. Since SDC (Silent Data Conversion) during the backpropagation phase can significantly impact the model training results, detecting silent data errors that may occur during the backpropagation phase of the AI ​​model can save computing resources and improve detection performance while preventing silent data errors from propagating to the entire AI model. In this scenario, the model training system also provides a backpropagation identification function 405, implemented by the host. For example, the host marks data to be used in the backpropagation phase during the forward propagation phase of the AI ​​model, and when calculating activation value gradients based on the marked data during the backpropagation phase, it identifies the backpropagation phase and uses the activation value gradients that will be communicated during the backpropagation phase as the data to be detected, i.e., as the data for the detection operator. Based on this, the detection operator is sent to the accelerator card. It should be understood that the underlying deep learning framework does not identify the training phase to which the communication operator belongs. Through the aforementioned backpropagation identification function 405, the host can promptly identify the timing when to send detection operators during the backpropagation phase, providing technical support for the host to send detection operators to the accelerator card during the backpropagation phase.

[0127] The detection operator execution function 403 is implemented by the accelerator card in the model training system. The accelerator card obtains the input data of the communication operator based on the detection operator sent by the host, executes the detection operator, obtains the execution result, and determines whether an SDC (Signal Disturbance Consequence) has occurred on the accelerator card. For example, taking the input data of the communication operator in the backpropagation stage as an example, this input data is the activation value gradient of the network layer in the AI ​​model. The statistical information of the activation value gradient is calculated to indicate the distribution characteristics of the numerical value. This statistical information is compared with the statistical information of the historical activation value gradient obtained when executing the same communication operator in previous iterations to determine whether the current activation value gradient is abnormal. If abnormal, it indicates that an SDC has occurred on the accelerator card. If the activation value gradient is not abnormal, the accelerator card executes the communication operator to transmit the activation value gradient to other accelerator cards, i.e., continues normal training. Since the statistical information of the activation value gradient usually exhibits certain patterns at different stages of training, when an SDC occurs, the statistical information of the activation value gradient will show abnormal peaks or changes compared to the threshold, indicating that there is an error in the data or calculation process, disrupting the normal training state. Therefore, the above method can accurately determine whether SDC has occurred on the accelerator card. Furthermore, since gradient calculation is typically involved in AI model training, the above method is highly versatile and can be applied to various neural network architectures.

[0128] The fault handling function 404 is implemented by the accelerator card in the model training system, or by the accelerator card and the host working together. After the accelerator card determines that an SDC has occurred on the accelerator card through the detection operator execution function 403, the accelerator card can record the SDC and alert the host. In some embodiments, after receiving the alert, the host can interrupt training or roll back to a point before the SDC occurred and continue training, etc. In other embodiments, after determining that an SDC has occurred, the accelerator card can also perform fault suppression for the SDC. For example, during the backpropagation phase, the activation value gradient is replaced with 0 before being transmitted to other accelerator cards to prevent the SDC from propagating and causing other accelerator cards to report errors. It should be understood that the activation value gradient is used to update the weights of the network layers in the AI ​​model. Since the activation value gradient received by other accelerator cards is 0, the activation value gradients subsequently calculated by other accelerator cards are also 0, and therefore the weights of the network layers on other accelerator cards will not be updated, thereby preventing the SDC from continuously spreading during the AI ​​model training process. In addition, the model training system can also provide other post-processing interfaces, such as resuming training from the previous checkpoint to start training again, in order to mitigate the impact of silent data errors, or to handle the accelerator card that has SDC, such as replacing the accelerator card, etc. This application does not limit this.

[0129] It should be noted that the functional division of the model training system is not limited to what is shown in Figure 4. In practical applications, more functions can be set according to user needs. For example, the model training system also includes storage functions, scheduling functions, etc. This application does not limit this.

[0130] Referring to Figure 5, the following example illustrates the function of the model training system shown in Figure 4, using the silent data error detection performed by the model training system during the backpropagation phase of the AI ​​model. Figure 5 is a schematic diagram of a model training system performing silent data error detection according to an embodiment of this application. As shown in Figure 5, during the forward propagation phase of the AI ​​model, the model training system uses the backpropagation identification function to mark the data used in the backpropagation phase. When calculating the activation value gradient based on the marked data during the backpropagation phase, the system identifies the backpropagation phase and uses the activation value gradient to be communicated during the backpropagation phase as the data to be detected. Through the detection operator sending and execution functions, the system obtains the execution result of the detection operator. If the execution result of the detection operator indicates that the accelerator card has experienced SDC (Sound-Confirmed Dispute), the system uses the fault handling function to suppress the SDC, locates the accelerator card where the SDC occurred, and processes the accelerator card with the SDC through other post-processing interfaces. If the execution result of the detection operator indicates that the accelerator card has not experienced SDC, normal training continues.

[0131] Schematic illustration: The functionality provided by the aforementioned model training system can be installed as a software toolkit component in a computing device and run by the host and accelerator card within that device. For example, when the computing device employs a heterogeneous computing architecture (including architectures using computing units with different instruction sets), users can install heterogeneous computing frameworks on the device. One such framework is the Compute Architecture for NeuroNet (CANN), a heterogeneous computing framework for neural networks. CANN can support users in quickly building AI applications by providing multi-layered programming interfaces. Additionally, users can install deep learning frameworks on the computing device to compile methods for implementing models, construct large-scale computational graphs, and automatically perform gradient calculations within those graphs. The functionality provided by the aforementioned model training system can be interfaced and adapted with deep learning frameworks and heterogeneous computing frameworks.

[0132] Referring to Figure 6, the relationship between the functions provided by the model training system and the deep learning framework and heterogeneous computing framework will be illustrated below. Figure 6 is a schematic diagram of the architecture of a model training system provided in an embodiment of this application. As shown in Figure 6, the model training system includes a training script 601, a model training framework 602, and a heterogeneous computing framework 603. The training script 601 is provided by the user, and executing the training script 601 calls the model training framework 602. The model training framework 602 includes third-party libraries (such as Megatron, DeepSpeed, AscendSpeed, etc.) and deep learning modules (such as PyTorch, MindSpore, etc.). The deep learning module is used to implement the detection operator sending function 402 and the backpropagation recognition function 406 shown in Figure 4 above. For example, the backpropagation recognition function is implemented through the Python layer, and the detection operator sending function is implemented through the C++ layer. It should be understood that the C++ layer is located at the bottom layer of PyTorch and is responsible for efficient computation execution and performance optimization, providing stable and high-speed computational support for the upper-layer Python code. The heterogeneous computing framework 603, for example, is CANN, used to implement the detection operator execution function 403 and fault handling function 404 shown in Figure 4 above. For example, it can be implemented by an AI CPU on the NPU used to handle complex non-matrix calculations. It should be noted that Figure 6 is only an example, and the above-mentioned model training framework 602 and heterogeneous computing framework 603 may include more or fewer functions or components, which is not limited in this application.

[0133] Referring to the embodiment shown in Figure 7, the silent data error detection method provided in this application is described below, taking the silent data error detection performed by the model training system during the backpropagation stage of the AI ​​model as an example.

[0134] Figure 7 is a flowchart of a silent data error detection method during model training provided in an embodiment of this application. As shown in Figure 7, the method is executed by the model training system described in Figures 1 to 6 above. The model training system includes a host computer and an accelerator card. Schematic, the method includes the following steps 701 to 705.

[0135] 701. The model training system responds to the training request of the AI ​​model and executes the forward propagation phase of the AI ​​model.

[0136] In this embodiment, the host and accelerator card in the model training system work together to perform the forward propagation phase of the AI ​​model. This process is described above and will not be repeated here.

[0137] It should be understood that the training process of an AI model typically involves multiple iterations. Illustrated, each iteration includes the following stages: Forward Propagation: Training data is input into the input layer of the AI ​​model, and forward propagation is performed layer by layer to calculate the predicted output of the output layer. Loss Calculation: The loss value between the predicted output and the true label is calculated. Backpropagation: Based on the loss value between the predicted output and the true label, the activation gradient of the output layer of the AI ​​model is calculated, and the activation gradient is propagated layer by layer from the output layer to the input layer using the chain rule. Layer-by-layer propagation of the activation gradient refers to calculating the activation gradient of the current network layer, and then backpropagating it to the next network layer based on the current network layer's activation gradient. The activation gradient of a network layer includes the activation gradients of neurons within that layer; it is the set of activation gradients of neurons in the network layer. For example, the activation gradient of the output layer includes the activation gradients of neurons in the output layer. Model update phase: The weight gradient of each network layer is calculated based on the activation value gradient of each network layer. Then, the weight gradient is sent to the optimizer of the network layer. The optimizer calculates the update amount required for the weight update of the network layer. After that, the weight of the network layer is updated according to the update amount to minimize the error of the entire AI model.

[0138] This embodiment uses any iteration during AI model training as an example. During the forward propagation phase of the AI ​​model, the host computer marks data for the backpropagation phase. This allows for timely identification of when to send detection operators when calculating activation gradients based on the marked data in subsequent backpropagation phases. This provides technical support for the host computer to send detection operators to the accelerator card during backpropagation, or in other words, it filters out communication operators used to transmit activation gradients during backpropagation, reducing detection overhead. In some embodiments, the host computer indicates whether a communication operator belongs to the forward or backpropagation phase by setting the value of a first variable. For example, hook functions are registered on the outermost input, the first parameter weight, the outermost output, and the last parameter weight of the AI ​​model. These hook functions trigger the setting of the first variable to a target value. It should be understood that the outermost input and the first parameter weight of the AI ​​model correspond to the final position traversed during backpropagation; while the outermost output and the last parameter weight of the AI ​​model correspond to the starting position traversed during backpropagation.

[0139] Schematic, referring to Figure 8, is a schematic diagram of a backpropagation identification method provided by an embodiment of this application. As shown in Figure 8, the host creates a global variable (i.e., the aforementioned first variable) to indicate whether it is in the backpropagation stage, for example named "IS_BACKWARD". During forward propagation, the host registers hook functions on the outermost input, the first parameter weight (represented as W1 in the figure), the outermost output, and the last parameter weight (represented as Wn in the figure) of the AI ​​model. Specifically, by registering hook functions on the last parameter weight and the outermost output of the model, when backpropagation reaches the calculation of the activation value gradient, the hook function is triggered. After triggering, the operation of setting "IS_BACKWARD" to "True" (i.e., the aforementioned target value) is executed, thereby marking that backpropagation has started to the corresponding key position and that a detection operator needs to be sent to the communication operator. By registering hook functions on the first parameter weight and the outermost input, these hook functions are triggered when backpropagation reaches this point. Upon triggering, the value of "IS_BACKWARD" is set to "False," indicating that backpropagation has reached its final critical point and is about to end. It is evident that by setting the global variable "IS_BACKWARD," the state of backpropagation can be clearly understood, providing technical support for subsequent detection operators. In some scenarios, when the model training system trains the AI ​​model using a data-parallel strategy, since the data-parallel strategy executes communication operators on the activation gradients after the AI ​​model's backpropagation phase, the detection operator can be executed on the first tensor where "requires_grad" is "True." Here, "requires_grad" is an attribute in the deep learning framework PyTorch used to control whether a tensor needs to calculate activation gradients; the first tensor with "requires_grad" set to "True" is the first tensor that requires activation gradient calculation.

[0140] 702. During the backpropagation phase of the AI ​​model, the host in the model training system sends multiple operators to the accelerator card. These operators include computation operators, communication operators, and detection operators.

[0141] In this embodiment, a communication operator is used to transmit the execution results of a computation operator between different accelerator cards. A detection operator is used to detect whether a silent data error has occurred on the accelerator card based on the input data of the communication operator. The input data of the communication operator is the activation value gradient of the network layer in the AI ​​model, and the computation operator is used to calculate the activation value gradient. After identifying the backpropagation stage through the above step 701, the input data of the communication operator can be obtained when the communication operator is called. By monitoring the communication operators in the backpropagation stage, the possible propagation exit of the fault can be blocked, ensuring that the fault is detected before it spreads, and achieving card-level localization. It is possible to monitor all or part of the communication operators in the backpropagation stage; this application does not limit this and it can be set according to business needs.

[0142] Schematic illustration: setting the value of a first variable indicates that the invoked communication operator belongs to the backpropagation phase. That is, during the forward propagation phase of the AI ​​model, the host labels the data to be used in the backpropagation phase, and during the backpropagation phase, when calculating the activation value gradient based on the labeled data, sets the value of the first variable to the target value. Based on this, when the host detects that the value of the first variable is the target value during the backpropagation phase, it sends the aforementioned multiple operators to the accelerator card.

[0143] In some embodiments, the host sends multiple operators to the accelerator card, including: sending detection and communication operators to a first task queue of the accelerator card, and sending computation operators to a second task queue of the accelerator card. The first and second task queues are different. The operators enter the task queues on the accelerator card in the form of instruction packets. The accelerator card then sequentially retrieves tasks from the task queues and executes them on the corresponding computing units (such as AI processing cores or AI CPUs). The first task queue can be understood as the communication flow on the accelerator card, and the second task queue can be understood as the computation flow on the accelerator card. Typically, the computation and communication flows on the accelerator card can be processed in parallel. This approach reduces performance overhead and ensures that the accelerator card executes the detection operator before executing the communication operator. It should be noted that this application does not limit the number of instructions that the host needs to send to the accelerator card when the detection operator is implemented. For example, the detection operator can be split according to its function. First, an instruction to obtain the statistical information of the input data of the communication operator can be sent, and then an instruction to determine whether SDC has occurred on the accelerator card based on the statistical information can be sent. In practical applications, it can be flexibly adjusted according to the deep learning framework applicable to the model training system. This application does not limit this.

[0144] Schematic, referring to Figure 9, is a schematic diagram of a communication operator in the backpropagation stage provided by an embodiment of this application. As shown in Figure 9, multiple computational and communication operators are executed on the accelerator card to train the AI ​​model (the relationship and number of computational and communication operators in the figure are only illustrative). The model training system can use various distributed parallel strategies to train the AI ​​model, such as pipeline parallelism, tensor parallelism, and expert parallelism. In the backpropagation stage of the AI ​​model, corresponding detection operators are issued for communication operators such as Send, AllGather, Reduce-Scatter, and All-to-All, so that the accelerator card can detect the input data of these communication operators, ensuring that faults are detected before they spread and achieving card-level localization.

[0145] 703. The accelerator card executes computation operators and obtains the input data of the communication operator based on the execution results of the computation operators. The input data of the communication operator is the activation value gradient of the network layer in the AI ​​model.

[0146] In this application, the accelerator card executes computational operators to obtain the activation gradients of the network layers in the AI ​​model, which is also the input data for the communication operators.

[0147] 704. The accelerator card executes the detection operator based on the input data of the communication operator and obtains the execution result of the detection operator.

[0148] In this embodiment, the input data of the communication operator refers to the activation value gradient of the network layer in the AI ​​model. The accelerator card executes the detection operator based on the activation value gradient to obtain the execution result of the detection operator. In some embodiments, the accelerator card determines the statistical information of the activation value gradient, which indicates the distribution characteristics of the numerical magnitude in the activation value gradient; based on the relationship between the statistical information of the activation value gradient and a threshold, the execution result of the detection operator is determined. For example, if the statistical information of the activation value gradient is greater than the threshold, it indicates that the activation value gradient is abnormal, and thus indicates that SDC has occurred on the accelerator card; if the activation value gradient is less than or equal to the threshold, it indicates that the activation value gradient is not abnormal, and thus indicates that SDC has not occurred on the accelerator card. Since the statistical information of the activation value gradient usually exhibits certain patterns at different stages of training, when SDC occurs, the statistical information of the activation value gradient will show abnormal peaks or changes compared to the threshold, indicating that there is an error in the data or calculation process, which disrupts the normal training state. Therefore, the above method can accurately determine whether SDC has occurred on the accelerator card. In addition, since gradient calculation is usually involved in AI model training, the above method has strong versatility and can be applied to various neural network architectures.

[0149] Schematic illustration: The statistical information of the activation value gradient may be, for example, the norm, variance, maximum value, or standard deviation of the activation value gradient. It should be noted that in real-world scenarios, the statistical information of the activation value gradient can also be other values, such as a combination of the above-mentioned statistical values, the median of the activation value gradient, etc. This application does not impose any specific limitations.

[0150] In some embodiments, the aforementioned threshold is related to the historical statistical information corresponding to the communication operator. The historical statistical information refers to the statistical information of the historical activation value gradients obtained by the accelerator card when executing the communication operator in the historical iterations of the AI ​​model. This threshold can be manually configured based on the historical statistical information, or it can be calculated based on the historical statistical information and probability inequalities. For example, the probability inequalities can be Chebyshev's inequality, Markov's inequality, etc., and this application does not limit this. In this scenario, if the execution result of the detection operator indicates that no silent data error has occurred on the accelerator card, the accelerator card updates the historical statistical information based on the statistical information of the activation value gradients obtained this time.

[0151] Schematic example, taking the probability inequality as a Markov inequality, the accelerator card calculates the threshold based on the following formula (1).

[0152] In the above formula (1), X refers to the statistical information of the activation value gradient, which can also be understood as a feature index; E t-1 [X] represents historical statistics, specifically the mean of statistics obtained based on historical activation value gradients; t represents the number of times the accelerator card executes the communication operator. For example, when t = 5, E t-1 [X] refers to the mean of the statistical information obtained based on the activation value gradients during the first four executions of the communication operator; 1 / a is the false detection rate, where a is, for example, 10. 6 Or 10 8 It can be configured according to business needs, and this application does not impose any limitations on it; aE t-1 [X] represents the threshold. If the statistical information of the currently calculated activation value gradient is greater than the threshold, it indicates that an SDC has occurred on the accelerator card. If it is less than or equal to the threshold, it indicates that an SDC has not occurred on the accelerator card. The accelerator card can update the historical statistical information based on the following formula (2).

[0153] E t [x]=βE t-1 [X]+(1-β)X (2)

[0154] In the above formula (2), β is a value close to 1, such as 0.999, which can be set according to business needs, and this application does not limit it. By combining probability inequalities and historical statistical information to calculate the threshold in the above way, it is possible to continuously collect statistical information of new activation value gradients as the iteration progresses, which helps to capture the dynamic changes and trends of statistical information and improve the accuracy of the detection operator execution results.

[0155] 705. If the execution result indicates that a silent data error has occurred on the accelerator card, the accelerator card will replace the activation value gradient with 0 and execute the communication operator based on the replaced activation value gradient.

[0156] In this embodiment, the accelerator card can suppress the silent data error. That is, the accelerator card replaces the activation value gradient with 0 before transmitting it to other accelerator cards, preventing errors from occurring on other accelerator cards after SDC propagation. Furthermore, since the activation value gradient has been suppressed, the silent data error will not be detected on other accelerator cards. Therefore, the accelerator card issuing the silent data error alarm is unique, and the alarming accelerator card is the one where the silent data error occurred, achieving precise SDC card-level location.

[0157] In some embodiments, the accelerator card can also perform processing actions for this silent data error based on its operating mode. The operating mode of the accelerator card includes at least one of the following:

[0158] The first working mode refers to the accelerator card recording the current silent data error. That is, after detecting a silent data error, the accelerator card does not issue an alarm, but only prints an exception log. Recording the current silent error means recording it in the form of an exception log. For example, it may record a timestamp, accelerator card identifier, operator information, error description, error code, etc., but this application does not limit this.

[0159] The second working mode refers to the accelerator card recording the silent data error, alerting the host, and interrupting training. In other words, after detecting a silent data error, the accelerator card prints an exception log, alerts the host, and interrupts training.

[0160] The third working mode refers to the accelerator card recording the silent data error and its execution status, alerting the host, and interrupting training. The accelerator card's execution status is recorded in the form of a debug log. That is, after detecting a silent data error, the accelerator card not only prints an exception log but also prints a debug log based on its execution status. For example, the debug log records operator information, hardware status, driver version, etc., but this application does not limit this.

[0161] In some embodiments, the host stores a second variable to indicate the operating mode of the accelerator card in the event of a silent data error. Illustratively, before performing step 702 above, the host determines whether a detection operator needs to be sent to the accelerator card and the accelerator card's operating mode based on the value of the second variable. For example, an environment variable (i.e., the second variable) corresponding to the silent data error detection switch is created, named "NPU_ASD_ENABLE," and its values ​​and corresponding functional descriptions are as follows:

[0162] NPU_ASD_ENABLE=0: Disable silent data error detection.

[0163] NPU_ASD_ENABLE=1: Enables silent data error detection, but does not issue an alarm after detecting a silent data error, only prints an exception log, which corresponds to the first working mode mentioned above.

[0164] NPU_ASD_ENABLE=2: Enables silent data error detection. Upon detecting a silent data error, it prints an exception log and generates an alarm, while simultaneously interrupting training. This corresponds to the second working mode mentioned above.

[0165] NPU_ASD_ENABLE=3: Enables silent data error detection. After detecting a silent data error, it prints an exception log and a debug log, generates an alarm, and interrupts training, which corresponds to the third working mode mentioned above.

[0166] In addition, after the accelerator card executes step 704, if the execution result indicates that no silent data error has occurred on the accelerator card, the accelerator card executes the communication operator and continues normal training.

[0167] Referring to Figure 10, the methods shown in steps 701 to 705 above will be summarized below. Figure 10 is a schematic diagram of a silent data error detection method in the model training process provided by an embodiment of this application. As shown in Figure 10, in response to the training request of the AI ​​model, the model training system first obtains the information of the silent data error detection switch on the accelerator card, that is, it determines whether the silent data error detection function of the accelerator card is enabled by the value of the second variable. After confirming that the accelerator card has enabled silent data error detection, the model training system executes the forward propagation phase of the AI ​​model. During this phase, data for the backpropagation phase is labeled. When calculating activation gradients based on the labeled data during backpropagation, the first variable is set to a target value. This identifies the backpropagation phase, and the activation gradients to be communicated during backpropagation are used as the data to be detected. A detection operator is sent to the accelerator card, and this operator is distributed to the accelerator card's communication stream. The accelerator card, upon receiving the input data for the communication operator, executes the detection operator based on that data. Specifically, it obtains statistical information about the activation gradients and uses probability inequalities to determine if a silent data error has occurred on the accelerator card. If a silent data error occurs, the activation gradient is replaced with 0, the error is recorded, and other post-processing is performed to accurately locate the accelerator card where the error occurred. If no silent data error occurs, the communication operator is executed, and normal training continues.

[0168] In summary, the method provided in this application detects the input data of the communication operators during model training, promptly identifying whether any accelerator cards in the model training system have silent data errors. This allows for the determination of silent data errors before the accelerator cards execute the communication operators, preventing the propagation of silent data errors between accelerator cards and thus avoiding their spread to the entire AI model. Furthermore, this method accurately locates the accelerator card experiencing the silent data error, making card-level localization simple and direct, with high detection efficiency, and unaffected by clock asynchrony between cards. This provides technical support for subsequent fault-tolerant measures against silent data errors. Additionally, the above method is applicable to all models that utilize backpropagation training and employ common communication operators for cluster communication, regardless of model structure, and exhibits good generalization across various accelerator cards, model architectures, and parallel strategies.

[0169] This application also provides a silent data error detection device during model training. This model training device can be implemented as part or all of the functions of the host in the aforementioned model training system through software, hardware, or a combination of both. Schematic, Figure 11 is a structural diagram of a silent data error detection device during model training provided in an embodiment of this application. As shown in Figure 11, the device is configured in the host of the model training system, and the device includes a sending module 1101.

[0170] The sending module 1101 is used to send multiple operators to the accelerator card to train an artificial intelligence (AI) model. The multiple operators include a computation operator, a communication operator, and a detection operator. The communication operator is used to transmit the execution results of the computation operator between different accelerator cards. The detection operator is used to detect whether a silent data error has occurred on the accelerator card based on the input data of the communication operator. The execution order of the detection operator is before the execution order of the communication operator.

[0171] In some embodiments, the sending module 1101 is configured to:

[0172] During the backpropagation phase of the AI ​​model training process, multiple operators are sent to the accelerator card. The input data of the communication operators are the activation gradients of the network layers in the AI ​​model.

[0173] In some embodiments, the device further includes a backpropagation identification module 1102, for:

[0174] During the forward propagation phase of AI model training, data is labeled for participation in the backpropagation phase, and when calculating the activation value gradient based on the labeled data during the backpropagation phase, the value of the first variable is set to the target value.

[0175] The sending module 1101 is used to send multiple operators to the accelerator card when the value of the first variable is detected as the target value during the backpropagation phase.

[0176] In some embodiments, the sending module 1101 is configured to:

[0177] The detection and communication operators are sent to the first task queue of the accelerator card, and the computation operator is sent to the second task queue of the accelerator card.

[0178] In some embodiments, the device further includes a storage module 1103 for storing a second variable, the second variable being used to indicate the operating mode of the accelerator card in the event of a silent data error; the operating mode includes at least one of the following:

[0179] The first working mode refers to the accelerator card recording an error in this silent data recording.

[0180] The second working mode refers to the accelerator card recording an error in the silent data, sending an alarm to the host and interrupting the training.

[0181] The third working mode refers to the accelerator card recording the current silent data error and the execution status of the accelerator card, alarming the host and interrupting the training.

[0182] With the aforementioned device, the host sends multiple operators to the accelerator card, including a detection operator. This detection operator is used to detect whether a silent data error has occurred on the accelerator card based on the input data of the communication operator. Based on this, when the accelerator card executes the computation operator to obtain the input data of the communication operator, it executes the detection operator to obtain the execution result of the detection operator. Then, it determines whether a silent data error has occurred on the accelerator card through the execution result of the detection operator. In this way, it can determine whether a silent data error has occurred on the current accelerator card before the accelerator card executes the communication operator, thus preventing the silent data error from spreading between accelerator cards and preventing the silent data error from spreading to the entire AI model, thereby improving the model training accuracy.

[0183] Of course, the device can also include other functional units to implement other functions involved in the host in the above method embodiments. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the device can be divided into different functional units to complete all or part of the functions described above. In addition, the device provided in the above embodiments and the above method embodiments belong to the same concept, and its specific implementation process can be found in the method embodiments, which will not be repeated here.

[0184] This application also provides another silent data error detection device during model training. This model training device can be implemented by software, hardware, or a combination of both to become part or all of the functions of the accelerator card in the aforementioned model training system. Schematic, Figure 12 is a structural diagram of a silent data error detection device during model training provided in an embodiment of this application. As shown in Figure 12, the device is configured in the accelerator card of the model training system. The device includes a receiving module 1201, an execution module 1202, and a detection module 1203.

[0185] The receiving module 1201 is used to receive multiple operators sent by the host for training an AI model. The multiple operators include a computation operator, a communication operator, and a detection operator. The communication operator is used to transmit the execution results of the computation operator between different accelerator cards, and the detection operator is used to detect whether a silent data error has occurred on the accelerator card based on the input data of the communication operator.

[0186] Execution module 1202 is used to execute the computation operator and obtain the input data of the communication operator based on the execution result of the computation operator;

[0187] The detection module 1203 is used to execute the detection operator based on the input data of the communication operator and obtain the execution result of the detection operator.

[0188] In some embodiments, the receiving module 1201 is configured to:

[0189] During the backpropagation phase of the AI ​​model training process, multiple operators are received, and the input data of the communication operators are the activation value gradients of the network layers in the AI ​​model.

[0190] The detection module 1203 is used to: execute a detection operator based on the activation value gradient and obtain the execution result of the detection operator.

[0191] In some embodiments, the detection module 1203 is configured to:

[0192] Determine the statistical information of the activation value gradient, which indicates the distribution characteristics of the numerical magnitudes in the activation value gradient;

[0193] The execution result of the detection operator is determined based on the statistical information of the activation value gradient and the relationship between the threshold.

[0194] Specifically, if the statistical message of the activation value gradient is greater than the threshold, the execution result of the detection operator indicates that a silent data error has occurred on the accelerator card; if the statistical message of the activation value gradient is less than or equal to the threshold, the execution result of the detection operator indicates that no silent data error has occurred on the accelerator card.

[0195] In some embodiments, the device further includes a threshold calculation module 1204, configured to:

[0196] Based on historical statistical information and probability inequalities, the threshold is calculated. Historical statistical information refers to the statistical information of historical activation value gradients obtained by the accelerator card when executing communication operators in the historical iterations of the AI ​​model.

[0197] If the execution result of the detection operator indicates that no silent data error has occurred on the accelerator card, the historical statistics are updated based on the statistics of the activation value gradient.

[0198] In some embodiments, the device further includes a fault suppression module 1205, for:

[0199] If the execution result of the detection operator indicates a silent data error on the accelerator card, the activation value gradient is replaced with 0, and the communication operator is executed based on the replaced activation value gradient.

[0200] In some embodiments, the receiving module 1201 is configured to:

[0201] The detection and communication operators are received through the first task queue on the accelerator card, and the computation operators are received through the second task queue on the accelerator card.

[0202] In some embodiments, if the execution result of the detection operator indicates that a silent data error has occurred on the accelerator card, the apparatus is further configured to: perform processing actions for the silent data error based on the operating mode of the accelerator card, wherein the operating mode includes at least one of the following:

[0203] The first working mode refers to the accelerator card recording an error in this silent data recording.

[0204] The second working mode refers to the accelerator card recording an error in the silent data, sending an alarm to the host and interrupting the training.

[0205] The third working mode refers to the accelerator card recording the current silent data error and the execution status of the accelerator card, alarming the host and interrupting the training.

[0206] With the aforementioned device, the host sends multiple operators to the accelerator card, including a detection operator. This detection operator is used to detect whether a silent data error has occurred on the accelerator card based on the input data of the communication operator. Based on this, when the accelerator card executes the computation operator to obtain the input data of the communication operator, it executes the detection operator to obtain the execution result of the detection operator. Then, it determines whether a silent data error has occurred on the accelerator card through the execution result of the detection operator. In this way, it can determine whether a silent data error has occurred on the current accelerator card before the accelerator card executes the communication operator, thus preventing the silent data error from spreading between accelerator cards and preventing the silent data error from spreading to the entire AI model, thereby improving the model training accuracy.

[0207] Of course, the device can also include other functional units to implement other functions involved in the accelerator card in the above method embodiments. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the device can be divided into different functional units to complete all or part of the functions described above. In addition, the device provided in the above embodiments and the above method embodiments belong to the same concept, and its specific implementation process can be found in the method embodiments, which will not be repeated here.

[0208] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with substantially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or execution order. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another. For example, without departing from the various examples described, a first operator can be referred to as a second operator, and similarly, a second operator can be referred to as a first operator. Both the first and second operators can be operators, and in some cases, they can be separate and distinct operators.

[0209] In this application, the term "at least one" means one or more, and the term "multiple" means two or more. For example, multiple operators means two or more operators.

[0210] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0211] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of program structure information. This program structure information includes one or more program instructions. When these program instructions are loaded and executed on a computing device, the processes or functions according to the embodiments of this application are generated, in whole or in part.

[0212] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0213] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for detecting silent data errors during model training, characterized in that, Applied to a model training system, the model training system including a host and an accelerator card, the method includes: The host sends multiple operators to the accelerator card to train an artificial intelligence (AI) model. The multiple operators include a computation operator, a communication operator, and a detection operator. The communication operator is used to transmit the execution results of the computation operator between different accelerator cards. The detection operator is used to detect whether a silent data error has occurred on the accelerator card based on the input data of the communication operator. The accelerator card executes the computation operator and obtains the input data of the communication operator based on the execution result of the computation operator; The accelerator card executes the detection operator based on the input data of the communication operator and obtains the execution result of the detection operator.

2. The method according to claim 1, characterized in that, The host sends multiple operators to the accelerator card, including: the host sends the multiple operators to the accelerator card during the backpropagation phase of the AI ​​model training process, wherein the input data of the communication operators is the activation value gradient of the network layer in the AI ​​model; The accelerator card executes the detection operator based on the input data of the communication operator to obtain the execution result of the detection operator, including: the accelerator card executes the detection operator based on the activation value gradient to obtain the execution result of the detection operator.

3. The method according to claim 2, characterized in that, The accelerator card executes the detection operator according to the activation value gradient to obtain the execution result of the detection operator, including: The accelerator card determines the statistical information of the activation value gradient, and the statistical information of the activation value gradient indicates the distribution characteristics of the numerical magnitude in the activation value gradient; The accelerator card determines the execution result of the detection operator based on the statistical information of the activation value gradient and the magnitude relationship between the threshold. If the statistical message of the activation value gradient is greater than the threshold, the execution result of the detection operator indicates that a silent data error has occurred on the accelerator card; if the statistical message of the activation value gradient is less than or equal to the threshold, the execution result of the detection operator indicates that no silent data error has occurred on the accelerator card.

4. The method according to claim 3, characterized in that, The method further includes: The accelerator card calculates the threshold based on historical statistical information and probability inequalities. The historical statistical information refers to the statistical information of the historical activation value gradient obtained by the accelerator card when executing the communication operator in the historical iteration of the AI ​​model. If the execution result of the detection operator indicates that no silent data error has occurred on the accelerator card, the accelerator card updates the historical statistical information based on the statistical information of the activation value gradient.

5. The method according to any one of claims 2 to 4, characterized in that, The method further includes: If the execution result of the detection operator indicates that a silent data error has occurred on the accelerator card, the accelerator card replaces the activation value gradient with 0 and executes the communication operator based on the replaced activation value gradient.

6. The method according to any one of claims 2 to 5, characterized in that, The method further includes: During the forward propagation phase of the AI ​​model training process, the host computer marks the data to be used in the backpropagation phase, and sets the value of the first variable to the target value when calculating the activation value gradient based on the marked data during the backpropagation phase. During the backpropagation phase of the AI ​​model training process, the host sends the plurality of operators to the accelerator card, including: when the host detects that the value of the first variable is the target value during the backpropagation phase, the host sends the plurality of operators to the accelerator card.

7. The method according to any one of claims 1 to 6, characterized in that, The host sends multiple operators to the accelerator card, including: The host sends the detection operator and the communication operator to the first task queue of the accelerator card, and sends the calculation operator to the second task queue of the accelerator card.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: If the execution result of the detection operator indicates that a silent data error has occurred on the accelerator card, the accelerator card performs processing actions for this silent data error based on the accelerator card's operating mode, which includes at least one of the following: The first working mode refers to the accelerator card recording an error in this silent data recording. The second working mode refers to the acceleration card recording an error in the current silent data, sending an alarm to the host and interrupting the training. The third working mode refers to the accelerator card recording the current silent data error and the execution status of the accelerator card, alarming the host and interrupting the training.

9. The method according to claim 8, characterized in that, The host computer stores a second variable that indicates the type of operating mode on the accelerator card.

10. A method for detecting silent data errors during model training, characterized in that, A host computer used in a model training system, the model training system further including an accelerator card, the method comprising: Multiple operators are sent to the accelerator card to train an artificial intelligence (AI) model. The multiple operators include a computation operator, a communication operator, and a detection operator. The communication operator is used to transmit the execution results of the computation operator between different accelerator cards. The detection operator is used to detect whether a silent data error has occurred on the accelerator card based on the input data of the communication operator. The execution order of the detection operator is before the execution order of the communication operator.

11. The method according to claim 10, characterized in that, The sending of multiple operators to the accelerator card includes: During the backpropagation phase of the AI ​​model training process, the multiple operators are sent to the accelerator card, and the input data of the communication operators are the activation value gradients of the network layers in the AI ​​model.

12. The method according to claim 11, characterized in that, The method further includes: During the forward propagation phase of the AI ​​model training process, data used to participate in the backpropagation phase is labeled, and when calculating the activation value gradient based on the labeled data during the backpropagation phase, the value of the first variable is set to the target value. The step of sending the plurality of operators to the accelerator card during the backpropagation phase of the AI ​​model training process includes: sending the plurality of operators to the accelerator card when the value of the first variable is detected to be the target value during the backpropagation phase.

13. The method according to any one of claims 10 to 12, characterized in that, The sending of multiple operators to the accelerator card includes: The detection operator and the communication operator are sent to the first task queue of the accelerator card, and the calculation operator is sent to the second task queue of the accelerator card.

14. The method according to any one of claims 10 to 13, characterized in that, The host computer stores a second variable, which indicates the operating mode of the accelerator card in the event of a silent data error; the operating mode includes at least one of the following: The first working mode refers to the accelerator card recording an error in this silent data recording. The second working mode refers to the acceleration card recording an error in the current silent data, sending an alarm to the host and interrupting the training. The third working mode refers to the accelerator card recording the current silent data error and the execution status of the accelerator card, alarming the host and interrupting the training.

15. A method for detecting silent data errors during model training, characterized in that, An accelerator card used in a model training system, the model training system further including a host computer, the method comprising: The system receives multiple operators sent by the host for training an AI model. These multiple operators include a computation operator, a communication operator, and a detection operator. The communication operator is used to transmit the execution results of the computation operator between different accelerator cards. The detection operator is used to detect whether a silent data error has occurred on the accelerator card based on the input data of the communication operator. Execute the computation operator, and obtain the input data of the communication operator based on the execution result of the computation operator; The detection operator is executed based on the input data of the communication operator to obtain the execution result of the detection operator.

16. The method according to claim 15, characterized in that, The receiving of multiple operators for training the AI ​​model sent by the host includes: During the backpropagation phase of the AI ​​model training process, the multiple operators are received, and the input data of the communication operators is the activation value gradient of the network layer in the AI ​​model. The step of executing the detection operator based on the input data of the communication operator and obtaining the execution result of the detection operator includes: executing the detection operator based on the activation value gradient and obtaining the execution result of the detection operator.

17. The method according to claim 16, characterized in that, The step of executing the detection operator according to the activation value gradient and obtaining the execution result of the detection operator includes: Determine the statistical information of the activation value gradient, wherein the statistical information of the activation value gradient indicates the distribution characteristics of the numerical magnitude in the activation value gradient; The execution result of the detection operator is determined based on the statistical information of the activation value gradient and the magnitude relationship between the threshold. If the statistical message of the activation value gradient is greater than the threshold, the execution result of the detection operator indicates that a silent data error has occurred on the accelerator card; if the statistical message of the activation value gradient is less than or equal to the threshold, the execution result of the detection operator indicates that no silent data error has occurred on the accelerator card.

18. The method according to claim 17, characterized in that, The method further includes: The threshold is calculated based on historical statistical information and probability inequalities. The historical statistical information refers to the statistical information of the historical activation value gradient obtained by the accelerator card when executing the communication operator in the historical iteration of the AI ​​model. If the execution result of the detection operator indicates that no silent data error has occurred on the accelerator card, the historical statistical information is updated based on the statistical information of the activation value gradient.

19. The method according to any one of claims 16 to 18, characterized in that, The method further includes: If the execution result of the detection operator indicates that a silent data error has occurred on the accelerator card, the activation value gradient is replaced with 0, and the communication operator is executed based on the replaced activation value gradient.

20. The method according to any one of claims 15 to 19, characterized in that, The receiving of multiple operators for training the AI ​​model sent by the host includes: The detection operator and the communication operator are received through the first task queue on the accelerator card, and the calculation operator is received through the second task queue on the accelerator card.

21. The method according to any one of claims 15 to 20, characterized in that, The method further includes: If the execution result of the detection operator indicates that a silent data error has occurred on the accelerator card, processing actions for this silent data error are performed based on the operating mode of the accelerator card, wherein the operating mode includes at least one of the following: The first working mode refers to the accelerator card recording an error in this silent data recording. The second working mode refers to the acceleration card recording an error in the current silent data, sending an alarm to the host and interrupting the training. The third working mode refers to the accelerator card recording the current silent data error and the execution status of the accelerator card, alarming the host and interrupting the training.

22. A model training system, characterized in that, The model training system includes a host computer and an accelerator card. The host computer is used to control the accelerator card. The model training system is used to implement the silent data error detection method in the model training process as described in any one of claims 1 to 9.

23. A processing chip, characterized in that, The processing chip is communicatively connected to the accelerator card, and the processing chip is used to control the accelerator card. The processing chip is used to implement the silent data error detection method in the model training process as described in any one of claims 10 to 14.

24. An accelerator chip, characterized in that, The acceleration chip is communicatively connected to the host computer, and the acceleration chip is used to accept the control of the host computer. The processing chip is used to implement the silent data error detection method in the model training process as described in any one of claims 15 to 21.

25. A computing device, characterized in that, The computing device includes a processor and a memory, the processor being configured to execute at least one piece of program code stored in the memory to enable the computing device to implement the silent data error detection method during the model training process as described in any one of claims 1 to 9.

26. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store at least one piece of program code, which, when executed by a computing device, causes the computing device to implement the silent data error detection method during the model training process as described in any one of claims 1 to 9.

27. A computer program product, characterized in that, When the computer program product is run on a computing device, the computing device implements the silent data error detection method during the model training process as described in any one of claims 1 to 9.

Images