Fault detecting method for machine learning and related apparatuses

Abstract

The present disclosure provides a fault detecting method for machine learning and related apparatuses. The fault detecting method includes: detecting whether a fault is presented in a current gradient from a current iteration of the machine learning, based on the current gradient and at least one of an upper boundary for the current iteration or a lower boundary for the current iteration, where the at least one of the upper boundary for the current iteration or the lower boundary for the current iteration is obtained based on statistical information from an iteration before the current iteration, and the statistical information indicates a distribution characteristic of at least one previous gradient from at least one previous iteration of the machine learning; and updating, based on a result of the detection, the statistical information to obtain updated statistical information for an iteration after the current iteration. By utilizing dynamically calculated upper and lower boundaries based on statistical information, the scheme can more accurately detect abnormal gradients caused by bitflip errors or other failures.

Description

FAULT DETECTING METHOD FOR MACHINE LEARNING AND RELATED APPARATUSESTECHNICAL FIELD

[0001] The present disclosure relates to the field of machine learning, and in particular to a fault detecting method for machine learning and related apparatuses.BACKGROUND

[0002] Deep neural networks (DNN) are widely used in many technical fields. Neural network training requires more and more computational and memory resources. During active use of computational resources an increase in technical failures is observed, such as switch of 1 to 0 (and vice versa) in a binary representation of a number (so-called bitflip). This type of errors cannot be observed immediately and sometimes they are called “soft errors” or “silent data corruption”. Such errors cause the convergence and stability problems in neural network training.

[0003] This background information is provided to reveal information believed by the applicant to be of possible relevance to the present disclosure. No admission is necessarily intended, nor should be construed, that any of the preceding information constitutes prior art against the present disclosure.SUMMARY

[0004] In a first aspect, an embodiment of the present disclosure provides a fault detecting method for machine learning, including:detecting whether a fault is presented in a current gradient from a current iteration of the machine learning, based on the current gradient and at least one of an upper boundary for the current iteration or a lower boundary for the current iteration,where the at least one of the upper boundary for the current iteration or the lower boundary for the current iteration is obtained based on statistical information from an iteration before the current iteration, and the statistical information indicates a distribution characteristic of at least one previous gradient from at least one previous iteration of the machine learning; andupdating, based on a result of the detection, the statistical information to obtain updated statistical information for an iteration after the current iteration.

[0005] By using the fault detecting method in which whether a fault is presented in a current gradient from a current iteration of the machine learning is detected, based on the current gradient and at least one of an upper boundary for the current iteration or a lower boundary for the current iteration, and the statistical information is updated to obtain updated statistical information for an iteration after the current iteration. The at least one of the upper boundary for the current iteration or the lower boundary for the current iteration is obtained based on statistical information from an iteration before the current iteration, and the statistical information indicates a distribution characteristic of at least one previous gradient from at least one previous iteration of the machine learning. The at least one of the upper boundary or the lower boundary can be adjusted dynamically to detect faults in the current gradients. By utilizing dynamically calculated upper and lower boundaries based on statistical information, the scheme can more accurately detect abnormal gradients caused by bitflip errors or other failures. In addition, the scheme reduces the impact of bitflip errors on the neural network training process by detecting and responding to them promptly, thereby enhancing the stability and robustness of training.

[0006] In a possible implementation of the first aspect, the result of the detection indicates that at least one preset bit in an exponent of a binary representation of the current gradient is faulty, where the at least one preset bit includes one or more of highest three bits in the exponent. The fault detecting method can be used to identify errors in specific bits within the exponent part of the binary representation of the gradients.

[0007] In a possible implementation of the first aspect, when the detection of the presence of the fault is based on the lower boundary for the current iteration, the detection includes:comparing the current gradient with the lower boundary for the current iteration;when the current gradient is smaller than the lower boundary for the current iteration, determining that the fault is presented on a second bit and / or a third bit of the highest three bits in the exponent.

[0008] The fault detector can not only detect whether a fault is present but also can achieve a more granular identification of faults in the gradient.

[0009] In a possible implementation of the first aspect, the result of the detection indicates at least one of following information:the current gradient being smaller than the lower boundary for the current iteration; or,the fault being presented on the second bit and / or the third bit.

[0010] The result explicitly or implicitly indicating that the fault is presented on the first bit of the current gradient can pinpoint the fault to specific bits, so the system can take more targeted measures to address the issue, enhancing the robustness and reliability of the machine learning model training.

[0011] In a possible implementation of the first aspect, when the detection of the presence of the fault is based on the upper boundary for the current iteration, the detection includes:comparing the current gradient with the upper boundary for the current iteration;when the current gradient is larger than the upper boundary for the current iteration, determining that the fault is presented on a first bit of the highest three bits in the exponent.

[0012] The fault detector can not only detect whether a fault is present but also provide detailed information about the position of the fault, which can be crucial for diagnosingand addressing the error effectively.

[0013] In a possible implementation of the first aspect, the result of the detection indicates at least one of following information:the current gradient being larger than the upper boundary for the current iteration; or,the fault being presented on the first bit.

[0014] The result explicitly or implicitly indicating that the fault is presented on the first bit of the current gradient can pinpoint the fault to specific bits, so the system can take more targeted measures to address the issue, enhancing the robustness and reliability of the machine learning model training.

[0015] In a possible implementation of the first aspect, the lower boundary for the current iteration is obtained by flipping a second bit, or a second bit and a third bit of highest three bits in an exponent of a binary representation of the upper boundary for the current iteration. By using this method to obtain the lower boundary, the fault detector can dynamically adjust the lower boundary based on the upper boundary, and the fault detector can quickly estimate the lower boundary, which is useful especially for rapid fault detection in scenarios where computational resources are limited or real¬ time processing is required.

[0016] In a possible implementation of the first aspect, the statistical information includes a mean of the at least one previous gradient and a variance of the at least one previous gradient, the method further includes:employing a statistical algorithm to obtain a mean of logarithms of gradients for the current iteration and a variance of logarithms of gradients for the current iteration, based on the statistical information and a gradient distribution of absolute values of the current gradient and the at least one previous gradient, where the gradient distribution is subject to zeroes exclusion;obtaining the lower boundary based on the mean of logarithms of gradients for the current iteration and the variance of logarithms of gradients for the current iteration.

[0017] In a possible implementation of the first aspect, the lower boundary for the current iteration is obtained based on the following formula:>where Bloweris the lower boundary for the current iteration, t is a number of iterations, gtlJis an element at an i-th row and j-th column in the gradient distribution, k is a hyperparameter corresponding to a number of a standard deviation of the at least one previous gradient, alta2are hyperparameters between 0 and 1 for adjusting weights for the at least one previous gradient and the current gradient, mtrepresents a mean of logarithm of the at least one previous gradient, andrepresents a variance of logarithm of the at least one previous gradient, mtis the mean of logarithms of gradients for the current iteration and vtis the variance of logarithms of gradients for the current iteration.

[0018] In a possible implementation of the first aspect, the statistical information includes a mean of the at least one previous gradient and a variance of the at least one previous gradient;where the upper boundary for the current iteration is obtained based on the following formula:"where Bupperis the upper boundary for the current iteration, / is a number of iterations, Mt^ is the mean of the at least one previous gradient,is the variance of the at least one previous gradient, k is a hyperparameter corresponding to a standard deviation of a number of the at least one previous gradient.

[0019] In a possible implementation of the first aspect, the updating of the statistical information includes:upon detecting that the fault is presented in the current gradient, skippingupdating the statistical information.

[0020] By ensuring that only accurate data is used in the training process, the reliability of the trained model can be improved.

[0021] In a possible implementation of the first aspect, the updating of the statistical information further includes:upon detecting that the fault is not presented in the current gradient, updating, based on the current gradient, the statistical information for the iteration after the current iteration.

[0022] Boundaries are calculated based on previous gradients, and both upper and lower boundaries should be updated during training process. The statistical information used for fault detection can be continuously updated based on the most recent data, ensuring that the fault detection system remains responsive to changes in the training dynamics.

[0023] In a possible implementation of the first aspect, the statistical information includes a mean of the at least one previous gradient and a variance of the at least one previous gradient;where the updating of the mean of the at least one previous gradient and the updating of the variance of the at least one previous gradient is based on the following formulas:where Mtis an updated mean in the updated statistical information, Mt_! is the mean of the at least one previous gradient, Vtis an updated variance in the updated statistical information, Vf_ i is the variance of the at least one previous gradient, ||gtl|oo is a gradient norm of the current gradient,and fi2arehyperparameters between 0 and 1.

[0024] In a possible implementation of the first aspect, the machine learning includes a training of a Deep Neural Network (DNN) model, where the DNN model is a DNN model with parallelism, and a type of the parallelism includes one of Tensor Parallelism(TP), Expert Parallelism (EP), Data Parallelism (DP), or Pipeline Parallelism (PP).

[0025] In a possible implementation of the first aspect, the training of the DNN model involves multiple devices, and each time when the input data generated by each of at least one of the multiple devices is required to be communicated with one or more other devices in the multiple devices, the detection of the presence of the fault and the updating of the statistical information based on the result of the detection are performed. In this way, faults can be identified early in the data exchange process, reducing the propagation of erroneous data. This implementation can allow for real-time monitoring and fault management at critical points in the training process where data is exchanged between devices.

[0026] In a second aspect, an embodiment of the present disclosure provides a fault detecting apparatus for machine learning, including:a detecting module, configured to detect whether a fault is presented in a current gradient from a current iteration of the machine learning, based on the current gradient and at least one of an upper boundary for the current iteration or a lower boundary for the current iteration, where the at least one of the upper boundary for the current iteration or the lower boundary for the current iteration is obtained based on statistical information from an iteration before the current iteration, and the statistical information indicates a distribution characteristic of at least one previous gradient from at least one previous iteration of the machine learning; andan updating module, configured to update, based on a result of the detection, the statistical information to obtain updated statistical information for an iteration after the current iteration.

[0027] In a possible implementation of the second aspect, the result of the detection indicates that at least one preset bit in an exponent of a binary representation of the current gradient is faulty, where the at least one preset bit includes one or more of highest three bits in the exponent.

[0028] In a possible implementation of the second aspect, when the detection of the presence of the fault is based on the lower boundary for the current iteration, thedetecting module is configured to:compare the current gradient with the lower boundary for the current iteration;when the current gradient is smaller than the lower boundary for the current iteration, determine that the fault is presented on a second bit and / or a third bit of the highest three bits in the exponent.

[0029] In a possible implementation of the second aspect, the result of the detection indicates at least one of following information:the current gradient being smaller than the lower boundary for the current iteration; or,the fault being presented on the second bit and / or the third bit.

[0030] In a possible implementation of the second aspect, when the detection of the presence of the fault is based on the upper boundary for the current iteration, the detecting module is configured to:compare the current gradient with the upper boundary for the current iteration;when the current gradient is larger than the upper boundary for the current iteration, determine that the fault is presented on a first bit of the highest three bits in the exponent.

[0031] In a possible implementation of the second aspect, the result of the detection indicates at least one of following information:the current gradient being larger than the upper boundary for the current iteration; or,the fault being presented on the first bit.

[0032] In a possible implementation of the second aspect, the lower boundary for the current iteration is obtained by flipping a second bit, or a second bit and a third bit of highest three bits in an exponent of a binary representation of the upper boundary for the current iteration.

[0033] In a possible implementation of the second aspect, the statistical informationincludes a mean of the at least one previous gradient and a variance of the at least one previous gradient, the detecting module is configured to:employ a statistical algorithm to obtain a mean of logarithms of gradients for the current iteration and a variance of logarithms of gradients for the current iteration, based on the statistical information and a gradient distribution of absolute values of the current gradient and the at least one previous gradient, where the gradient distribution is subject to zeroes exclusion;obtain the lower boundary based on the mean of logarithms of gradients for the current iteration and the variance of logarithms of gradients for the current iteration.

[0034] In a possible implementation of the second aspect, the lower boundary for the current iteration is obtained based on the following formula:where Bioweris the lower boundary for the current iteration, t is a number of iterations, gtlJis an element at an i-th row and j-th column in the gradient distribution, k is a hyperparameter corresponding to a number of a standard deviation of the at least one previous gradient, a , a2are hyperparameters between 0 and 1 for adjusting weights for the at least one previous gradient and the current gradient,represents a mean of logarithm of the at least one previous gradient, andrepresents a variance of logarithm of the at least one previous gradient, mtis the mean of logarithms of gradients for the current iteration and vtis the variance of logarithms of gradients for the current iteration .

[0035] In a possible implementation of the second aspect, the statistical information includes a mean of the at least one previous gradient and a variance of the at least one previous gradient;where the upper boundary for the current iteration is obtained based on thefollowing formula:"where Bupperis the upper boundary for the current iteration, t is a number of iterations, Mt-xis the mean of the at least one previous gradient, Vt-i isthe variance of the at least one previous gradient, k is a hyperparameter corresponding to a number of a standard deviation of the at least one previous gradient.

[0036] In a possible implementation of the second aspect, the updating module is configured to:upon detecting that the fault is presented in the current gradient, skip updating the statistical information.

[0037] In a possible implementation of the second aspect, the updating module is configured to:upon detecting that the fault is not presented in the current gradient, update, based on the current gradient, the statistical information for the iteration after the current iteration.

[0038] In a possible implementation of the second aspect, the statistical information includes a mean of the at least one previous gradient and a variance of the at least one previous gradient;where the updating of the mean of the at least one previous gradient and the updating of the variance of the at least one previous gradient is based on the following formulas:>>where Mtis an updated mean in the updated statistical information,is the mean of the at least one previous gradient, Vtis an updated variance in the updated statistical information, Vt-1is the variance of the at least one previous gradient, ||gt||oo is a gradient norm of the current gradient,and (?2are hyperparameters between 0 and 1.

[0039] In a possible implementation of the second aspect, the machine learning includes a training of a Deep Neural Network (DNN) model, where the DNN model is a DNN model with parallelism, and a type of the parallelism includes one of Tensor Parallelism (TP), Expert Parallelism (EP), Data Parallelism (DP), or Pipeline Parallelism (PP).

[0040] In a possible implementation of the second aspect, the training of the DNN model involves multiple devices, and each time when the input data generated by each of at least one of the multiple devices is required to be communicated with one or more other devices in the multiple devices, the detection of the presence of the fault and the updating of the statistical information based on the result of the detection are performed.

[0041] In a third aspect, an embodiment of the present disclosure provides a fault detecting apparatus including processing circuitry for executing the fault detecting method for machine learning according to the first aspect or any possible implementation of the first aspect.

[0042] In a fourth aspect, an embodiment of the present disclosure provides a machine learning system including the fault detecting apparatus according to the second aspect or any possible implementation of the second aspect, or the fault detecting apparatus according to the third aspect.

[0043] In a fifth aspect, an embodiment of the present disclosure provides a fault detecting system including a fault detecting component for executing the fault detecting method for machine learning according to the first aspect or any possible implementation of the first aspect.

[0044] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable medium has instructions stored therein which, when executed by a processor, causes the processor to execute the fault detecting method for machine learning according to the first aspect or any possible implementation of the first aspect.

[0045] In a seventh aspect, an embodiment of the present disclosure provides a computer program product including computer execution instructions which, when executed by a processor, causes the processor to execute the fault detecting method for iimachine learning according to the first aspect or any possible implementation of the first aspect.

[0046] In an eighth aspect, an embodiment of the present disclosure provides a computer program, when executed by a processor, causes the processor to execute the fault detecting method for machine learning according to the first aspect or any possible implementation of the first aspect.

[0047] In a ninth aspect, an embodiment of the present disclosure provides a computing device, including a processor and a memory. The processor is connected to the memory. The memory is configured to store instructions, the processor is configured to execute the instructions. When the processor executes the instructions stored in the memory, the processor is enabled to perform the method in the first aspect or any possible design of the first aspect.

[0048] In a tenth aspect, an embodiment of the present disclosure provides a computing device cluster, including at least one computing device, where the computing device includes a processor and a memory coupled with the processor, where the memory is configured to store a computer program, and the processor is configured to invoke and run the computer program stored in the memory, so that the computing device executes the method in the first aspect or any possible design of the first aspect.

[0049] Embodiments of the present disclosure provide a fault detecting method for machine learning and related apparatuses. The fault detecting method in which whether a fault is presented in a current gradient from a current iteration of the machine learning is detected, based on the current gradient and at least one of an upper boundary for the current iteration or a lower boundary for the current iteration, and the statistical information is updated to obtain updated statistical information for an iteration after the current iteration. The at least one of the upper boundary for the current iteration or the lower boundary for the current iteration is obtained based on statistical information from an iteration before the current iteration, and the statistical information indicates a distribution characteristic of at least one previous gradient from at least one previous iteration of the machine learning. The at least one of the upper boundary or the lowerboundary can be adjusted dynamically to detect faults in the current gradients. By utilizing dynamically calculated upper and lower boundaries based on statistical information, the scheme can more accurately detect abnormal gradients caused by bitflip errors or other failures. In addition, the scheme reduces the impact of bitflip errors on the neural network training process by detecting and responding to them promptly, thereby enhancing the stability and robustness of training.BRIEF DESCRIPTION OF DRAWINGS

[0050] Reference will now be made, by way of example, to the accompanying drawings which show example embodiments of the present disclosure.

[0051] FIG. 1 is a schematic flowchart of a fault detecting method for machine learning according to one or more possible embodiments of the present disclosure.

[0052] FIG. 2 is a schematic diagram showing an example of a binary representation of a number in brain floating point 16 bits (bfl 6).

[0053] FIG. 3 shows an example of lower boundary calculation process according to one or more embodiments of the present disclosure.

[0054] FIG. 4 shows a logical architecture of a fault detector according to one or more embodiment of the present disclosure.

[0055] FIG. 5 shows a fault detection module (FDM) inside computational system according to one or more embodiments of the present disclosure.

[0056] FIG. 6A shows a forward pass scheme of application of FDM within TP according to one or more embodiments of the present disclosure.

[0057] FIG. 6B shows a backward pass scheme of application of FDM within TP according to one or more embodiments of the present disclosure.

[0058] FIG. 7 shows a scheme of application of FDM within EP according to one or more embodiments of the preset disclosure.

[0059] FIG. 8 shows a scheme of application of FDM within DP and PP according to one or more embodiments of the preset disclosure.

[0060] FIG. 9 shows a schematic structural diagram of a fault detecting apparatus according to one or more possible embodiments of the present disclosure.

[0061] FIG. 10 shows a schematic diagram of an architecture of a computing device according to one or more possible embodiments of the present application.

[0062] FIG. 11 shows a schematic diagram of an architecture of a computing device cluster according to one or more possible embodiments of the present application.

[0063] FIG. 12 is a schematic diagram of a connection between computing devices over a network according to one or more embodiments of the present disclosure.DESCRIPTION OF EMBODIMENTS

[0064] In the following description, reference is made to the accompanying figures, which form part of the present disclosure, and which show, by way of illustration, specific aspects of embodiments of the present disclosure or specific aspects in which embodiments of the present disclosure may be used. It is understood that embodiments of the present disclosure may be used in other aspects and include structural or logical changes not depicted in the figures. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims.

[0065] As DNNs become more complex and are applied to a broader range of technical fields, a demand for computational and memory resources grows. The increased demand puts a strain on hardware, leading to a higher likelihood of technical failures, particularly bitflips. The bitflip is a type of error where a bit in the binary representation of a number is flipped, i.e., a 1 is changed to a 0 or vice versa. This can occur due to various reasons such as hardware malfunctions, interference, or environmental factors like radiation. These errors are often referred to as “soft errors” because they do not permanently damage the hardware and are not immediately apparent. They are also known as “silent data corruption” because they can silently alter data without raising any alarms. During neural network training, bitflips can lead toincorrect weight updates, which may result in convergence issues, where the network fails to learn correctly, or stability problems. The above leads to the need to develop methods that can detect such errors with a high rate, localize them and minimize their impact in the process of training. Some technologies primarily focus on adversarial attacks while neglecting silent data corruption caused by hardware failures during training.

[0066] One way is to monitor gradient history values in optimizers and moving variance values in normalization layers during the training process. However, there is still no effective ways to detect the silent data corruption caused by accidental technical failures during training process.

[0067] In view of the above, the present disclosure provides a fault detecting method in which whether a fault is presented in a current gradient from a current iteration of the machine learning is detected, based on the current gradient (or referred to as current gradient value) and at least one of an upper boundary for the current iteration or a lower boundary for the current iteration, and the statistical information is updated to obtain updated statistical information for an iteration after the current iteration. The at least one of the upper boundary for the current iteration or the lower boundary for the current iteration is obtained based on statistical information from an iteration before the current iteration, and the statistical information indicates a distribution characteristic of at least one previous gradient (or referred to as previous gradient value) from at least one previous iteration of the machine learning. The at least one of the upper boundary or the lower boundary can be adjusted dynamically to detect faults in the current gradients. By utilizing dynamically calculated upper and lower boundaries based on statistical information, the scheme can more accurately detect abnormal gradients caused by bitflip errors or other failures. In addition, the scheme reduces the impact of bitflip errors on the neural network training process by detecting and responding to them promptly, thereby enhancing the stability and robustness of training.

[0068] FIG. 1 is a schematic flowchart of a fault detecting method for machine learning according to one or more possible embodiments of the present disclosure. Thefault detecting method can be implemented by a fault detector. Optionally, the fault detector can be an independent module integrated within a machine learning framework or a chip, a standalone service that runs parallel to the main machine training process, a specific hardware device, which is not limited herein. As shown in FIG. 1, the fault detecting method for machine learning can include the following steps.

[0069] SI 10, detect whether a fault is presented in a current gradient from a current iteration of the machine learning, based on the current gradient and at least one of an upper boundary for the current iteration or a lower boundary for the current iteration, where the at least one of the upper boundary for the current iteration or the lower boundary for the current iteration is obtained based on statistical information from an iteration before the current iteration, and the statistical information indicates a distribution characteristic of at least one previous gradient from at least one previous iteration of the machine learning.

[0070] The machine learning training process generally lasts for multiple iterations, and it maybe, e.g., a training process for a model. For each current iteration, the fault detection would be carried out based on the current gradient obtained from the current iteration and the statistical information from an iteration (immediately) before the current iteration.

[0071] Specifically, the fault detector detects whether there is a fault, such as a bitflip error, present in the current gradient calculated during the current iteration of the machine learning training process. The detection is based on comparing the current gradient with at least one of an upper boundary or a lower boundary. The current gradient may represent a slope of loss functions with respect to the model’s parameters that is computed during the current iteration of training. The upper boundary and the lower boundary are thresholds that define the expected range of normal gradients. The upper boundary is set to detect significant increases in gradients that could indicate a fault, while the lower boundary is set to detect significant decreases. For example, a fault can be detected when the current gradient is larger than the upper boundary or smaller than the lower boundary. At least one of these boundaries (either the upper orthe lower) is used in the detection process for each iteration. The upper boundary or the lower boundary is obtained based on statistical information from an iteration before the current iteration, and the statistical information indicates a distribution characteristic of at least one previous gradient from at least one previous iteration of the machine learning. The at least one previous gradient may include at least one previous gradient where no fault is detected by the fault detector from the previous iteration of the machine learning.

[0072] In a possible implementation, the statistical information may include at least one of a mean of the at least one previous gradient, a variance of the at least one previous gradient, a mean of logarithms of the at least one previous gradient, a variance of a logarithm of the at least one previous gradient, or a standard deviation of the at least one previous gradient. The at least one previous gradient here includes one or more previous gradients for which the fault detection indicates that no fault is presented.

[0073] SI 20, update, based on a result of the detection, the statistical information to obtain updated statistical information for an iteration after the current iteration.

[0074] The statistical information is updated based on the result of the detection to obtain updated statistical information, and the updated statistical information can be used for an iteration after the current iteration. The statistical information is updated to reflect the latest distribution characteristics of the gradients.

[0075] In binary representations of numbers, especially in floating-point formats, a bit flip in the higher positions of the exponent part typically has a greater impact on the result. This is because the exponent part determines the magnitude of the number, and the higher bits have a more significant influence on the scale. FIG. 2 is a schematic diagram showing an example of a binary representation of a number in bfl6 (BFloatl6 or Brain Floatl6). As shown in FIG. 2, in the bfl6 format, a number is represented with a specific bit allocation: bit 0 for the sign, bits 1-7 for the exponent, and bits 8-15 for the fraction. 1stbitflip can be used to indicate a fault in the bit with index 1. The gradient is usually not extremely big, so the first bit in an exponent of a binary representation of the gradient is usually 0. If there is a fault on the first bit of the gradient, the first bitmay turn to 1 instead of 0, which leads to enormous numbers during calculations. 2ndand 3rdbitflips can be used to indicate faults in the bit with index 2 and in the bit with index 3 respectively. The second bit and / or the third bit in an exponent of a binary representation of the gradient is usually 1. If there is a fault on the second or third bit of the gradient, the bit may turn to 0 instead of 1 , which are too small in compare with real values of the gradient distribution, but bigger than 0. The 2ndand 3rdbitflips can subtly alter the exponent, leading to underflow in the gradient’s magnitude, which can still significantly affect calculations.

[0076] In a possible implementation, the result of the detection may indicate that at least one preset bit in an exponent of a binary representation of the cunent gradient is faulty, where the at least one preset bit may include one or more of highest three bits in the exponent. The fault detecting method can be used to identify errors in specific bits within the exponent part of the binary representation of the gradients. The fault detecting method may focus on the highest three bits in the exponent, which is particularly relevant because these bits often have a significant impact on the magnitude of the floating-point numbers used to represent the gradients. Errors in these bits can lead to substantial changes in the value.

[0077] In a possible implementation, when the detection of the presence of the fault is based on the upper boundary for the current iteration, the fault detector may detect whether the fault is presented in a current gradient from a current iteration of the machine learning by comparing the current gradient with the upper boundary for the current iteration; and when the current gradient is larger than the upper boundary for the current iteration, determining that the fault is presented on a first bit of the highest three bits in the exponent. The fault detector may compare the current gradient with the upper boundary to detect whether a fault is presented in a current gradient, and if the current gradient is larger than the upper boundary, the fault detector can determine that a fault is presented on a first bit of the highest three bits in the exponent. In this case, the result of the detection may indicate at least one of following information: the current gradient being larger than the upper boundary for the current iteration; or, the fault beingpresented on the first bit. The fault detector can not only detect whether a fault is present but also provide detailed information about the position of the fault, which can be crucial for diagnosing and addressing the error effectively. The result explicitly or implicitly indicating that the fault is presented on the first bit of the current gradient can pinpoint the fault to specific bits, so the system can take more targeted measures to address the issue, enhancing the robustness and reliability of the machine learning model training.

[0078] The upper boundary can be determined based on different approaches, e.g. Chebyshev Inequality. In a possible implementation, the statistical information may include a mean of the at least one previous gradient and a variance of the at least one previous gradient, the upper boundary for the current iteration is obtained based on the following formula:"where Bupperis the upper boundary for the current iteration, t is a number of iterations,is the mean of the at least one previous gradient, Vt_i is the variance of the at least one previous gradient, k is a hyperparameter corresponding to a number of a standard deviation of the at least one previous gradient.

[0079] In a possible implementation, k may be the number of values used to calculate the standard deviation (or the variance), it is usually an integer number. The hyperparameter can be preset at the beginning of training. A larger k can increase the range of values considered, thus increasing the probability that all of the at least one gradient will be within this boundary. For example, k = 100 can be set.

[0080] In a possible implementation, when the detection of the presence of the fault is based on the lower boundary for the current iteration, the fault detector may detect whether the fault is presented in a current gradient from a current iteration of the machine learning by comparing the current gradient with the lower boundary for the current iteration, and when the current gradient is smaller than the lower boundary for the current iteration, determining that the fault is presented on a second bit and / or athird bit of the highest three bits in the exponent. The fault detector may compare the current gradient with the lower boundary to detect whether a fault is presented in a current gradient, and if the current gradient is larger than the upper boundary, the fault detector can determine that a fault is presented on a first bit of the highest three bits in the exponent. In this case, the result of the detection may indicate at least one of following information: the current gradient being smaller than the lower boundary for the current iteration; or, the fault being presented on the second bit and / or the third bit. The fault detector can report the result of the detection explicitly or implicitly. The fault detector can not only detect whether a fault is present but also can achieve a more granular identification of faults in the gradient. The result explicitly or implicitly indicating that the fault is presented on the first bit of the current gradient can pinpoint the fault to specific bits, so the system can take more targeted measures to address the issue, enhancing the robustness and reliability of the machine learning model training.

[0081] In a possible implementation, the lower boundary for the current iteration can be obtained by flipping a second bit, or a second bit and a third bit of highest three bits in an exponent of a binary representation of the upper boundary for the current iteration. For example, the upper three bits of exponential part of the upper boundary is 011, and the lower boundary can be obtained by switching 1 at the second bit or the second and third bits of the upper boundary to 0. That is, the upper three bits of exponential part of the lower boundary can be 001 or 000. By using this method to obtain the lower boundary, the fault detector can dynamically adjust the lower boundary based on the upper boundary, and the fault detector can quickly estimate the lower boundary, which is useful especially for rapid fault detection in scenarios where computational resources are limited or real-time processing is required.

[0082] As bitflips result in either too big or too small values of the gradient, upper and lower boundaries can be determined to restrain the gradients. Upper boundary allows to detect 1stbitflip, and lower boundary allows to detect 2ndand 3rdbitflips.

[0083] In another possible implementation, the lower boundary can also be obtained independently. The statistical information may include a mean of the at least oneprevious gradient and a variance of the at least one previous gradient. The fault detector may employ a statistical algorithm to obtain a mean of logarithms of gradients for the current iteration and a variance of logarithms of gradients for the current iteration based on the statistical information and a gradient distribution of absolute values of the current gradient and the at least one previous gradient (the gradient distribution is subject to zeroes exclusion), and then the fault detector may obtain the lower boundary based on the mean of logarithms for the current iteration and the variance of logarithms for the current iteration. For example, The fault detector may obtain a mean of logarithm of the at least one previous gradient and a variance of the logarithm of the at least one previous gradient based on the statistical information, and employ a statistical algorithm to obtain a mean of logarithms of gradients for the current iteration and a variance of logarithms of gradients for the current iteration based on the mean of logarithm of the at least one previous gradient, the variance of the logarithm of the at least one previous gradient, and the gradient distribution, and then the fault detector may obtain the lower boundary based on the mean of logarithms for the current iteration and the variance of logarithms for the current iteration.

[0084] FIG. 3 shows an example of lower boundary calculation process according to one or more embodiments of the present disclosure. As shown in FIG. 3, the preset algorithm is Vysochanskij - Petunin inequality. A gradient distribution may be constructed using absolute values of the current gradient and the at least one previous gradient, zeroes are excluded in the gradient distribution. Logarithms of elements of the gradient distribution subject to zeroes exclusion can be obtained. A mean of logarithm of the at least one previous gradient and a variance of logarithm of the at least one previous gradient are obtained based on the statistical information. The Vysochanskij -Petunin inequality can be used to obtain a mean of logarithms of gradients for the cunent iteration and a variance of logarithms of gradients for the current iteration, and then a lower boundary can be obtained by returning the logarithm to an original scale of the gradient.

[0085] In a possible implementation, the statistical information may include a meanof the at least one previous gradient and a variance of the at least one previous gradient, and the lower boundary for the current iteration is obtained based on the following formula:>where Btoweris the lower boundary for the current iteration, t is the number of iterations, gtlJis an element at an i-th row and j-th column in the gradient distribution, and the gradient distribution is subject to zeroes exclusion, k is a hyperparameter corresponding to the number of a standard deviation of the at least one previous gradient, alta2are hyperparameters between 0 and 1 for adjusting weights for the at least one previous gradient and the current gradient,represents a mean of logarithm of the at least one previous gradient, and vt-i represents a variance of logarithm of the at least one previous gradient, mtis the mean of logarithms of gradients for the current iteration and vtis the variance of logarithms of gradients for the current iteration.

[0086] Among them,and a2represent contribution of the mean of logarithm of the at least one previous gradient (i.e. mt-x), and the variance of logarithm of the at least one previous gradient (i.e. Vf-i), so they should be from interval (0, 1). In a possible implementation, since the current gradient may be corrupted,and a2can be set as 0.99.

[0087] In a possible implementation, the fault detector may compare the current gradient with the upper boundary and the lower boundary parallelly. Alternatively, the fault detector may compare the current gradient with the upper boundary first. If the current gradient is larger than the upper boundary, the lower boundary would not be calculated, or the current gradient would not be compared with the lower boundary. If the current gradient is smaller than the upper boundary, the fault detector may comparethe current gradient with the lower boundary. Alternatively, the fault detector may compare the current gradient with the lower boundary first. If the current gradient is smaller than the lower boundary, the upper boundary would not be calculated, or the current gradient would not be compared with the upper boundary. If the current gradient is larger than the lower boundary, the fault detector may compare the current gradient with the upper boundary.

[0088] In a possible implementation, the fault detector may update the statistical information regardless of whether a fault is presented in the current gradient from the current iteration of the machine learning. The fault detector may report the result of the detection and the updated statistical information to the computational node or other modules in the system, and the computational node or other modules receiving the result of the detection and the updated statistical information may deal with the updated statistical information according to the result of the detection. If the result of the detection indicates that the fault is presented in the current gradient, the computational node or other modules may roll the updated statistical information back to previous values or replace corrupted data with zero values. Each computational node can handle errors independently, which can lead to better scalability since adding more computational nodes won’t impose additional burdens on the fault detector.

[0089] In another possible implementation, the fault detector may skip updating the statistical information upon detecting that the fault is presented in the current gradient. Upon detecting that the fault is not presented in the current gradient, the fault detector may update, based on the current gradient, the statistical information for the iteration after the cunent iteration. In this case, the fault detector may determine whether to update the statistical information based on the result of the detection, and if the result of the detection indicates that the fault is presented in the current gradient, the fault detector may skip updating the statistical information. If the result of the detection explicitly or implicitly indicates that the fault is not presented in the current gradient, the fault detector may update the statistical information based on the current gradient, and the updated statistical information can be used for the iteration after the currentiteration. For example, if the result of the detection indicates that the fault is presented in the current gradient, a mean of logarithms of gradients for the current iteration and a variance of logarithms of gradients for the current iteration respectively equal to the mean of logarithm of the at least one previous gradient and the variance of logarithm of the at least one previous gradient, i.e., mt=and vt= vt-i. For another example, if the result of the detection indicates that the fault is presented in the current gradient, a mean of gradients for the current iteration and a variance of gradients for the cunent iteration respectively equal to the mean of the at least one previous gradient and the variance of the at least one previous gradient, i.e. Mt= Mt- 1, and Vt= Vt- 1. In this case, if other modules determine that the statistical information output by the fault detector has been updated, other modules can determine that the fault is not presented in the current gradient. While, if other modules determine that the statistical information output by the fault detector has not been updated, other modules can determine that the fault is presented in the current gradient. It is considered that the gradient should not be taken into account if its value is corrupted, and static boundaries (i.e. boundaries with constant value) may lead to degradation of fault detector. In the embodiment, boundaries are calculated based on previous gradients, and both upper and lower boundaries should be updated during training process. The statistical information used for fault detection can be continuously updated based on the most recent data, ensuring that the fault detection system remains responsive to changes in the training dynamics. By ensuring that only accurate data is used in the training process, the reliability of the trained model can be improved.

[0090] FIG. 4 shows a logical architecture of a fault detector according to one or more embodiment of the present disclosure. As shown in FIG. 4, the fault detector may obtain input data including a current gradient, statistics from the previous step (that is, statistical information), and at least one hyperparameter for model training. The fault detector may calculate an upper boundary and a lower boundary based on the input data. The fault detector may compare the current gradient with the upper boundary first. If the current gradient is larger than the upper boundary, the current gradient would not becompared with the lower boundary and would not update the statistics. If the current gradient is smaller than the upper boundary, the fault detector may compare the current gradient with the lower boundary then. If the current gradient is smaller than the lower boundary, the statistics would not be updated. If the current gradient is larger than the lower boundary, the fault detector may update the statistics included in the input data based on the current gradient to obtain updated statistics. Then the fault detector may output statistics or updated statistics when they have been updated.

[0091] In a possible implementation, the statistical information may include a mean of the at least one previous gradient and a variance of the at least one previous gradient, and the updating of the mean of the at least one previous gradient and the updating of the variance of the at least one previous gradient may be based on the following formulas:> where Mtis an updated mean in the updated statistical information, Mt-1is the mean of the at least one previous gradient, Vtisanupdated variance in the updated statistical information, Vt-1is the variance of the at least one previous gradient, Halloo is a gradient norm of the current gradient,and ?2arehyperparameters between 0 and 1.

[0092] and ?2represent contribution of the mean of the at least one previous gradient and the variance of the at least one previous gradient (i.e. Mt- 1and Vt_i), so they should be from interval (0, 1). In a possible implementation,and / ?2can be set as 0.99. In a possible implementation, ||,gt|| = max gtl], where g1is an element iat an i-th row and j -th column in a gradient distribution of absolute values of the current gradient and the at least one previous gradient, and the gradient distribution is subject to zeroes exclusion.

[0093] In a possible implementation, the machine learning may include a training of a Deep Neural Network (DNN) model, where the DNN model may be a DNN modelwith parallelism. A type of the parallelism may include one of Tensor Parallelism (TP), Expert Parallelism (EP), Data Parallelism (DP), or Pipeline Parallelism (PP). The fault detector can be effectively used in different types of products based on DNN (e.g. virtual assistants, chat bots, self-driving vehicles, image recognition). The fault detector can prevent errors in training caused by silent data corruption with high detection rate and low overheads, which will allow to spend less resources on training, reduce the wear and tear of computing equipment and improve DNN-based product reliability.

[0094] In a possible implementation, the fault detector may be realized as an independent module. The training of the DNN model may involve multiple devices, and each time when the input data generated by each of at least one of the multiple devices is required to be communicated with one or more other devices in the multiple devices, the detection of the presence of the fault and the updating of the statistical information based on the result of the detection are performed. The fault detector may detect whether the fault is presented in the current gradient before the data exchange process. In this way, faults can be identified early in the data exchange process, reducing the propagation of erroneous data. This implementation can allow for real-time monitoring and fault management at critical points in the training process where data is exchanged between devices. By detecting and responding to faults promptly, the system can maintain the integrity and accuracy of the training process across all devices.

[0095] Examples of a computational system using the fault detector according to one or more embodiments of the present disclosure will be described below. The fault detector is realized as a special module (FDM) to detect and localize bitflips for illustration. The module can be used in any training process by inserting it before any cross-processor communications. The module is activated on each computational device.

[0096] FIG. 5 shows a FDM inside computational system according to one or more embodiments of the present disclosure. The position of the FDM in the general scheme of the computing process is shown in FIG. 5. As shown in FIG. 5, Devices 0-N denote computations on the nodes of a computational system (e.g., network processing units(NPUs) or graphics processing units (GPUs)), and “Communication operation” block denotes any type of communications between devices: AllReduce, AllToAll, Send, Recv etc. The computational system may include a plurality of FDMs, and each of the plurality of FDMs may correspond to a device of the Devices 0-N. The FDM is realized as an independent module, and runs separately on each computational node. In this way, the computational system can localize the device, where failure takes place.

[0097] FDM can be used in models with different types of parallelism, e.g. Tensor Parallelism (TP). FIG. 6A and FIG. 6B shows schemes of application of FDM within TP. FIG. 6A shows a forward pass scheme of application of FDM within TP, and FIG.6B shows a backward pass scheme of application of FDM within TP. As shown in FIG.6A and FIG. 6B, FDM is “hooked” before AllReduce operation, which occurs before Transformer and FeedForward blocks. In the forward pass scheme, the FDM is integrated into the process before the AllReduce operation. The AllReduce operation may involve a reduction phase where a certain operation (like sum, product, maximum, or minimum) is applied to the input data across all processes and the result is broadcast to the participating processes. If the FDM detects a fault present on the current gradient, the AllReduce module may discard the faulty data, and / or alter the system to the issue. The FDM is used to detect the faults in the gradients before they are combined in the AllReduce operation, which can ensure that erroneous data can be identified and handled before it affects the model’s update. In the backward pass scheme, similar to the forward pass, the FDM is also placed before the AllReduce operation. By positioning the FDM before the AllReduce operation in the backward pass, the system can identify and mitigate faults in the gradients that result from the backpropagation of errors.

[0098] The FDM may include submodules to execute the above-mentioned steps. For example, the FDM may include an upper boundary calculating submodule to calculate upper boundary, a lower boundary calculating submodule to calculate lower boundary, and a comparing and updating boundary module to compare gradients with boundaries and to update statistics if the gradient distribution lays within these boundaries.

[0099] FDM can also be used in models with EP. FIG. 7 shows a scheme of application of FDM within EP according to one or more embodiments of the preset disclosure. It should be noted that only forward pass scheme is shown in FIG. 7, but the principle also applies for backward pass scheme. As shown in FIG. 7, the FDM is “hooked” to AllToAll operation before every Expert Layer. In the forward pass scheme, the FDM is integrated into the process before the AllToAll operation. The AllToAll operation is a communication operation that exchanges data between devices (e.g. Device 1 and Device 2 shown in FIG. 7) in a way that each device receives data from other device. The FDM is placed before each expert layer to monitor the data that is being processed by each expert layer. By detecting and handling faults before the AllToAll operation, the FDM can help maintain the quality of the data that is used in the expert layers.

[0100] FDM can be also used in models with DP, PP and its combination. FIG. 8 shows a scheme of application of FDM within DP and PP according to one or more embodiments of the preset disclosure. It should be noted that only forward pass scheme is shown in FIG. 8, but the principle also applies for backward pass scheme. As shown in FIG. 8, the FDM is “hooked” to AllReduce and Send / Recv operations. Each expert layer processes its portion of the data. Before the results are communicated to other experts via an AllToAll operation, the FDM would detect whether a fault is presented in a current gradient for a current iteration. The FDM analyzes the data to detect any anomalies or faults. If a fault is detected by the FDM, the result of the detection would be transmitted to AllReduce module, and the AllReduce module trigger appropriate actions, such as discarding the faulty data, requesting re-computation, or alerting the system to the issue. If no faults are detected, the data may proceed to one or more devices. Each device would output a current gradient and statistical information to the FDM before exchanging the output data with other device via the Send / Recv module.

[0101] The fault detector according to one or more embodiments of the preset disclosure can also be realized as a fault injection tool within a training cloud service.

[0102] The present disclosure aims to detect silent data corruption on software level,it can also be adopted for use in other spheres, which include intensive mathematical computations on many devices and are vulnerable to soft errors. For example, the same approach can be used in direct numerical simulation of physical processes, where physical values are beneath some reasonable boundaries too.

[0103] In the embodiment, whether a fault is presented in a current gradient from a current iteration of the machine learning is detected, based on the current gradient and at least one of an upper boundary for the current iteration or a lower boundary for the current iteration, where the at least one of the upper boundary for the current iteration or the lower boundary for the current iteration is obtained based on statistical information from an iteration before the current iteration, and the statistical information indicates a distribution characteristic of at least one previous gradient from at least one previous iteration of the machine learning; the statistical information is updated based on a result of the detection, to obtain updated statistical information for an iteration after the current iteration. The at least one of the upper boundary or the lower boundary can be adjusted dynamically to detect faults in the current gradient. By utilizing dynamically calculated upper and lower boundaries based on statistical information, the scheme can more accurately detect abnormal gradients caused by bitflip errors or other failures. In addition, the scheme reduces the impact of bitflip errors on the neural network training process by detecting and responding to them promptly, thereby enhancing the stability and robustness of training. In some cases, 1stbitflip can be detected through the upper boundary, and 2ndand / or 3rdbitflip can be detected through the lower boundary, so more fault types can be detected. In some cases, the value which is detected to be faulty would not be taken into account in a next iteration, so the reliability of the trained model can be improved. In some cases, the fault detector is realized as an independent module, and the module runs separately on each computational node, so the device can be localized, where failure takes place.

[0104] Some examples have been described to facilitate understanding of the solutions of the fault detecting method for machine learning according to the present disclosure. It should be noted that the examples are for illustration purpose only, whichshould not be construed as limitations to the solution of the present disclosure.

[0105] Next, embodiments of products related to the fault detecting method for machine learning will be described.

[0106] FIG. 9 shows a schematic structural diagram of a fault detecting apparatus according to one or more possible embodiments of the present disclosure. The fault detecting apparatus may be applied to a machine learning framework or a chip, a standalone service that runs parallel to the main machine training process, a specific hardware device, or other component that has similar function, which is not limited herein. As shown in FIG. 9, the fault detecting apparatus 900 includes:a detecting module 901, configured to detect whether a fault is presented in a current gradient from a current iteration of the machine learning, based on the current gradient and at least one of an upper boundary for the current iteration or a lower boundary for the current iteration, where the at least one of the upper boundary for the current iteration or the lower boundary for the current iteration is obtained based on statistical information from an iteration before the current iteration, and the statistical information indicates a distribution characteristic of at least one previous gradient from at least one previous iteration of the machine learning; andan updating module 902, configured to update, based on a result of the detection, the statistical information to obtain updated statistical information for an iteration after the current iteration.

[0107] The updating module 902 may also be referred to as a dynamic boundary update module, and is configured to update statistics for both the upper boundary and the lower boundary based on current tensor value and fault check.

[0108] In a possible implementation, the result of the detection indicates that at least one preset bit in an exponent of a binary representation of the current gradient is faulty, where the at least one preset bit includes one or more of highest three bits in the exponent.

[0109] In a possible implementation, when the detection of the presence of the fault is based on the lower boundary for the current iteration, the detecting module 901 isconfigured to:compare the current gradient with the lower boundary for the current iteration;when the current gradient is smaller than the lower boundary for the current iteration, determine that the fault is presented on a second bit and / or a third bit of the highest three bits in the exponent.

[0110] In a possible implementation, the result of the detection indicates at least one of following information:the current gradient being smaller than the lower boundary for the current iteration; or,the fault being presented on the second bit and / or the third bit.[OHl] In a possible implementation, when the detection of the presence of the fault is based on the upper boundary for the current iteration, the detecting module 901 is configured to:compare the current gradient with the upper boundary for the current iteration;when the current gradient is larger than the upper boundary for the current iteration, determine that the fault is presented on a first bit of the highest three bits in the exponent.

[0112] In a possible implementation, the result of the detection indicates at least one of following information:the current gradient being larger than the upper boundary for the current iteration; or,the fault being presented on the first bit.

[0113] In a possible implementation, the lower boundary for the current iteration is obtained by flipping a second bit, or a second bit and a third bit of highest three bits in an exponent of a binary representation of the upper boundary for the current iteration.

[0114] In a possible implementation, the statistical information includes a mean of the at least one previous gradient and a variance of the at least one previous gradient, thedetecting module 901 is configured to:employ a statistical algorithm to obtain a mean of logarithms of gradients for the current iteration and a variance of logarithms of gradients for the current iteration, based on the statistical information and a gradient distribution of absolute values of the current gradient and the at least one previous gradient, where the gradient distribution is subject to zeroes exclusion;obtain the lower boundary based on the mean of logarithms of gradients for the current iteration and the variance of logarithms of gradients for the current iteration.

[0115] In a possible implementation, the lower boundary for the current iteration is obtained based on the following formula:>where Bioweris the lower boundary for the current iteration, t is a number of iterations, gtl}is an element at an i-th row and j-th column in the gradient distribution, k is a hyperparameter corresponding to a number of a standard deviation of the at least one previous gradient, alta2are hyperparameters between 0 and 1 for adjusting weights for the at least one previous gradient and the current gradient,represents a mean of logarithm of the at least one previous gradient, and vt_- represents a variance of logarithm of the at least one previous gradient, mtis the mean of logarithms of gradients for the current iteration and vtis the variance of logarithms of gradients for the current iteration .

[0116] In a possible implementation, the statistical information includes a mean of the at least one previous gradient and a variance of the at least one previous gradient;where the upper boundary for the current iteration is obtained based on the following formula:where Bupperis the upper boundary for the current iteration, t is a number of iterations, Mt-is the mean of the at least one previous gradient,is the variance of the at least one previous gradient, k is a hyperparameter corresponding to a number of a standard deviation of the at least one previous gradient.

[0117] In a possible implementation, the updating module 902 is configured to:upon detecting that the fault is presented in the current gradient, skip updating the statistical information.

[0118] In a possible implementation, the updating module 902 is configured to:upon detecting that the fault is not presented in the current gradient, update, based on the current gradient, the statistical information for the iteration after the current iteration.

[0119] In a possible implementation, the statistical information includes a mean of the at least one previous gradient and a variance of the at least one previous gradient;where the updating of the mean of the at least one previous gradient and the updating of the variance of the at least one previous gradient is based on the following formulas:Vt= max p2Vt-1+ (1 - / ?2)IW Vt-i), where Mtis an updated mean in the updated statistical information,is the mean of the at least one previous gradient, Vtisanupdated variance in the updated statistical information, Vt-is the variance of the at least one previous gradient, ||gt||oo is a gradient norm of the current gradient,and / 32are hyperparameters between 0 and 1.

[0120] In a possible implementation, the machine learning includes a training of a Deep Neural Network (DNN) model, where the DNN model is a DNN model with parallelism, and a type of the parallelism includes one of Tensor Parallelism (TP), Expert Parallelism (EP), Data Parallelism (DP), or Pipeline Parallelism (PP).

[0121] In a possible implementation, the training of the DNN model involves multiple devices, and each time when the input data generated by each of at least one of themultiple devices is required to be communicated with one or more other devices in the multiple devices, the detection of the presence of the fault and the updating of the statistical information based on the result of the detection are performed.

[0122] An embodiment of the present disclosure provides a fault detecting apparatus including processing circuitry for executing the fault detecting method for machine learning described above.

[0123] The fault detecting apparatus can be the fault detector described in the above embodiment. The working principle of the fault detecting apparatus is similar to that of the fault detector, and same technical effects will be achieved, which will not be described herein for brevity.

[0124] An embodiment of the present disclosure provides a machine learning system including the fault detecting apparatus as described with reference to FIG. 9 or the fault detecting apparatus including processing circuitry for executing the fault detecting method for machine learning described above.

[0125] FIG. 10 is a schematic diagram of an architecture of a computing device according to one or more possible embodiments of the present application. The computing device 1000 can be a server or a computer or other device with computing capabilities. The computing device 1000 shown in FIG. 10 includes: a processor 1020, a memory 1030 and a bus 1010. The processor 1020 and the memory 1030 are in communication with each other via the bus 1010. The computing device 1000 can be a server or a terminal device. The present application does not limit the number of processors, and memories in the computing device 1000.

[0126] The bus 1010 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one line is used in FIG. 10, but it does not mean that there is only one bus or one type of bus. The bus 1010 may include pathways for transferring information between various components of the computing device 1000 (e.g., the memory 1030, the processor 1020, or the transceiver 1040).

[0127] The processor 1020 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP) processor.

[0128] The memory 1030 can include volatile memory, such as RAM. The processor 1020 can also include non-volatile memory, such as read-only memory (ROM), flash memory, a mechanical hard disk (HDD), or a solid state drive (SSD).

[0129] An executable program code is stored in the memory 1030, and the processor 1020 executes the executable program code to implement the functions of the aforementioned detecting module 901, and the updating module 902, respectively, so as to implement the fault detecting method for machine learning described in the above embodiment. That is, the memory 1030 has instructions on it for executing the fault detecting method for machine learning described in the above embodiment.

[0130] The structure of the computing device 1000 enumerated above is only an exemplary illustration, and the present application is not limited thereto. The computing device 1000 of embodiments of the present application includes a variety of hardware found in computer systems in the prior art, e.g., the computing device 1000 also includes memory other than the memory 1030, such as, for example, a disk memory, and the like. It should be understood by those skilled in the art that the computing device 1000 can also include other devices necessary to achieve proper operation. In addition, depending on the specific needs, one of skill in the art should understand that the abovedescribed computing device 1000 may also include other hardware devices for realizing other additional functions. In addition, one of skill in the art should understand that the above-described computing device 1000 may also include only the devices necessary to realize the embodiments of the present application, and need not include all of the devices shown in FIG. 10.

[0131] FIG. 11 is a schematic diagram of an architecture of a computing device cluster according to one or more possible embodiments of the present application.

[0132] The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a localserver in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0133] As shown in FIG. 11, the cluster of computing devices described includes at least one computing device 1100. The memory 1130 in one or more of the computing devices 1100 in the cluster of computing devices can hold the same instructions for performing the fault detecting method for machine learning described in the above embodiment.

[0134] In some possible implementations, some of the instructions for performing the fault detecting method for machine learning described in the above embodiments can also be separately held in the memory 1130 of the one or more computing devices 1100 in the cluster of computing devices. In other words, a combination of the one or more computing devices 1100 can jointly execute instructions for performing the fault detecting method for machine learning described in the above embodiment.

[0135] It is noted that the memories 1130 in the different computing devices 1100 in the cluster of computing devices can store different instructions for performing some of the functions of the computing devices 1100, respectively. That is, the instructions stored in the memories 1130 in the different computing devices 1100 can realize the functions of one or more of the detecting module 901 and the updating module 902.

[0136] In some possible implementations, one or more computing devices in a cluster of computing devices can be connected via a network. Among other things, the network can be a wide area network or a local area network, etc. FIG. 12 illustrates one possible implementation. FIG. 12 is a schematic diagram of a connection between computing devices 1200 A and 1200B over a network according to one or more embodiments of the present disclosure. The two computing devices 1200 A and 1200B are connected to each other via a network. Specifically, the connection to said network is made through a communication interface in each computing device. In this class of possible implementations, the memory 1206 A in the computing device 1200 A holds instructions for performing a part of the method described in the above embodiments. At the same time, the memory 1206B in the computing device 1200B holds instructions forperforming other part(s) of the method described in the above embodiments.

[0137] The functions of computing device 1200 A illustrated in FIG. 12 can also be accomplished by multiple computing devices. Similarly, the functions of computing device 1200B can be accomplished by multiple computing devices.

[0138] An embodiment of the present disclosure provides a computer-readable storage medium storing computer execution instructions which, when executed by a processor, cause the processor to execute the fault detecting method for machine learning described above.

[0139] An embodiment of the present disclosure provides a computer program product including computer execution instructions which, when executed by a processor, cause the processor to execute the fault detecting method for machine learning described above.

[0140] An embodiment of the present disclosure provides a computer program which, when executed by a processor, cause the processor to execute the fault detecting method for machine learning described above.

[0141] The embodiments set forth herein represent information sufficient to practice and illustrate ways of practicing such subject matter. Upon reading the description in light of the accompanying figures, those of skill in the art will understand the concepts of the claimed subject matter and will recognize applications of these concepts not particularly addressed herein. It should be understood that these concepts and applications fall within the scope of the disclosure and the accompanying claims.

[0142] Moreover, it will be appreciated that any module, component, or device disclosed herein that executes instructions may include, or otherwise have access to, a non-transitory computer / processor readable storage medium or media for storage of information, such as computer / processor readable instructions, data structures, program modules and / or other data. A non-exhaustive list of examples of non-transitory computer / processor readable storage media includes magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, optical disks such as compact disc read-only memory (CD-ROM), digital video discs or digital versatile discs (i.e.,DVDs), Blu-ray Disc™, or other optical storage, volatile and non-volatile, removable and non-removable media implemented in any method or technology, random-access memory (RAM), read-only memory (ROM), electrically erasable programmable readonly memory (EEPROM), flash memory or other memory technology. Any such non-transitory computer / processor storage media may be part of a device / apparatus or accessible or connectable thereto. Computer / processor readable / executable instructions to implement a method, an application or a module described herein may be stored or otherwise held by such non-transitory computer / processor readable storage media.

[0143] In some aspects of the present disclosure, there is provided a non-transitory computer-readable medium has instructions stored therein, the instructions, when executed by a processor, may cause the processor to implement the method of the present disclosure.

[0144] In some aspects of the present disclosure, there is provided an apparatus / chipset system including at least one processor executing instructions stored in a computer-readable medium to implement the method of the present disclosure.

[0145] The present disclosure encompasses various embodiments, including not only method embodiments, but also other embodiments such as apparatus embodiments and embodiments related to non-transitory computer readable storage media. Embodiments may incorporate, individually or in combinations, the features disclosed herein.

[0146] Although this disclosure refers to illustrative embodiments, this is not intended to be construed in a limiting sense. Various modifications and combinations of the illustrative embodiments, as well as other embodiments of the disclosure, will be apparent to persons skilled in the art upon reference to the description.

[0147] Features disclosed herein in the context of any particular embodiments may also or instead be implemented in other embodiments. Method embodiments, for example, may also or instead be implemented in apparatus, system, and / or computer program product embodiments. In addition, although embodiments are described primarily in the context of methods and apparatus, other implementations are also contemplated, as instructions stored on one or more non-transitory computer-readablePC17RU2024 / 000346media, for example. Such media could store programming or instructions to perform any of various methods consistent with the present disclosure.

[0148] Further, in the disclosure, the word “a” or “an” when used in conjunction with the term “including” or “including” in the claims and / or the specification may mean “one”, but it is also consistent with the meaning of “one or more”, “at least one”, and “one or more than one” unless the content clearly dictates otherwise. Similarly, the word “another” may mean at least a second or more unless the content clearly dictates otherwise. The term “and / or” herein when used in association with a list of items means any one or more of the items including that list.

Claims

CLAIMS1. A fault detecting method for machine learning, comprising:detecting whether a fault is presented in a current gradient from a current iteration of the machine learning, based on the current gradient and at least one of an upper boundary for the current iteration or a lower boundary for the current iteration, wherein the at least one of the upper boundary for the current iteration or the lower boundary for the current iteration is obtained based on statistical information from an iteration before the current iteration, and the statistical information indicates a distribution characteristic of at least one previous gradient from at least one previous iteration of the machine learning; andupdating, based on a result of the detection, the statistical information to obtain updated statistical information for an iteration after the current iteration.

2. The method according to claim 1, wherein the result of the detection indicates that at least one preset bit in an exponent of a binary representation of the current gradient is faulty, wherein the at least one preset bit comprises one or more of highest three bits in the exponent.

3. The method according to claim 2, wherein when the detection of the presence of the fault is based on the lower boundary for the current iteration, the detection comprises:comparing the current gradient with the lower boundary for the current iteration; when the current gradient is smaller than the lower boundary for the current iteration, determining that the fault is presented on a second bit and / or a third bit of the highest three bits in the exponent.

4. The method according to claim 3, wherein the result of the detection indicates at least one of following information:the current gradient being smaller than the lower boundary for the current iteration; or,the fault being presented on the second bit and / or the third bit.

5. The method according to any one of claims 2 to 4, wherein when the detection of the presence of the fault is based on the upper boundary for the current iteration, the detection comprises:comparing the current gradient with the upper boundary for the current iteration; when the current gradient is larger than the upper boundary for the current iteration, determining that the fault is presented on a first bit of the highest three bits in the exponent.

6. The method according to claim 5, wherein the result of the detection indicates at least one of following information:the current gradient being larger than the upper boundary for the current iteration; or,the fault being presented on the first bit.

7. The method according to any one of claims 1 to 6, wherein the lower boundary for the current iteration is obtained by flipping a second bit, or a second bit and a third bit of highest three bits in an exponent of a binary representation of the upper boundary for the current iteration.

8. The method according to any one of claims 1 to 6, wherein the statistical information comprises a mean of the at least one previous gradient and a variance of the at least one previous gradient, the method further comprises:employing a statistical algorithm to obtain a mean of logarithms of gradients for the current iteration and a variance of logarithms of gradients for the current iteration, based on the statistical information and a gradient distribution of absolute values of the current gradient and the at least one previous gradient, wherein the gradient distribution is subject to zeroes exclusion;obtaining the lower boundary based on the mean of logarithms of gradients for the current iteration and the variance of logarithms of gradients for the current iteration.

9. The method according to claim 8, wherein the lower boundary for the current iteration is obtained based on the following formula:wherein Btoweris the lower boundary for the current iteration, Z is a number of iterations, gtlJis an element at an i-th row and j -th column in the gradient distribution, k is a hyperparameter corresponding to a number of a standard deviation of the at least one previous gradient,and a2are hyperparameters between 0 and 1 for adjusting weights for the at least one previous gradient and the current gradient,represents a mean of logarithm of the at least one previous gradient, and Vf-i represents a variance of logarithm of the at least one previous gradient, mtis the mean of logarithms of gradients for the current iteration and vtis the variance of logarithms of gradients for the current iteration.

10. The method according to any one of claims 1 to 9, wherein the statistical information comprises a mean of the at least one previous gradient and a variance of the at least one previous gradient;wherein the upper boundary for the current iteration is obtained based on the following formula:wherein Bupperis the upper boundary for the current iteration, t is a number of iterations, is the mean of the at least one previous gradient,is the variance of the at least one previous gradient, k is a hyperparameter corresponding to a number of a standard deviation of the at least one previous gradient.

11. The method according to any one of claims 1 to 10, wherein the updating of the statistical information comprises:upon detecting that the fault is presented in the current gradient, skipping updating the statistical information.

12. The method according to claim 11, wherein the updating of the statisticalPC17RU2024 / 000346information further comprises:upon detecting that the fault is not presented in the current gradient, updating, based on the current gradient, the statistical information for the iteration after the current iteration.

13. The method according to claim 11 or 12, wherein the statistical information comprises a mean of the at least one previous gradient and a variance of the at least one previous gradient;wherein the updating of the mean of the at least one previous gradient and the updating of the variance of the at least one previous gradient is based on the following formulas:&wherein Mtis an updated mean in the updated statistical information,is the mean of the at least one previous gradient, Vtis an updated variance in the updated statistical information, Vt- is the variance of the at least one previous gradient, Halloo is a gradient norm of the current gradient, p and / ?2arehyperparameters between 0 and 1.

14. The method according to any one of claims 1 to 13, wherein the machine learning comprises a training of a Deep Neural Network (DNN) model, wherein the DNN model is a DNN model with parallelism, and a type of the parallelism comprises one of Tensor Parallelism (TP), Expert Parallelism (EP), Data Parallelism (DP), or Pipeline Parallelism (PP).

15. The method according to claim 14, wherein the training of the DNN model involves multiple devices, and each time when the input data generated by each of at least one of the multiple devices is required to be communicated with one or more other devices in the multiple devices, the detection of the presence of the fault and the updating of the statistical information based on the result of the detection are performed.

16. A fault detecting apparatus for machine learning, comprising:a detecting module, configured to detect whether a fault is presented in a currentgradient from a current iteration of the machine learning, based on the current gradient and at least one of an upper boundary for the current iteration or a lower boundary for the current iteration, wherein the at least one of the upper boundary for the current iteration or the lower boundary for the current iteration is obtained based on statistical information from an iteration before the current iteration, and the statistical information indicates a distribution characteristic of at least one previous gradient from at least one previous iteration of the machine learning; andan updating module, configured to update, based on a result of the detection, the statistical information to obtain updated statistical information for an iteration after the current iteration.

17. A fault detecting apparatus, comprising a processing circuitry for executing the fault detecting method for machine learning according to any one of claims 1 to 15.

18. A machine learning system comprising a fault detecting component for executing the fault detecting method for machine learning according to any one of claims 1 to 15.

19. A computer-readable storage medium storing computer execution instructions which, when executed by a processor, cause the processor to execute the fault detecting method for machine learning according to any one of claims 1 to 15.

20. A computer program product including computer execution instructions which, when executed by a processor, cause the processor to execute the fault detecting method for machine learning according to any one of claims 1 to 15.

21. A computing device cluster, comprising at least one computing device, wherein the computing device comprises at least one processor and a memory coupled with the at least one processor, wherein the memory stores instructions which, when executed by the at least one processor, cause the computing device cluster to execute the method for machine learning according to any one of claims 1 to 15.

Images