Fault detection and recovery method, apparatus, and computing device

Abstract

Provided in the present application are a fault detection and recovery method, an apparatus and a computing device. The method comprises: acquiring a first tensor under test; performing precision fault detection on the first tensor to determine a precision fault detection result of the first tensor; when the precision fault detection result indicates that a precision fault is present in the first tensor, determining a target operator related to the first tensor, the first tensor being input data or output data of the target operator; and, on the basis of the target operator, recovering from the precision fault present in the first tensor. The solution can quickly detect precision faults of AI models, so as to recover from the precision faults of the AI models in a timely manner, such that the AI models can perform stable training and inference.

Description

Methods, apparatus, and computing devices for fault detection and recovery

[0001] This application claims priority to Russian Federation patent application No. 2025101033, filed on January 21, 2025, entitled "Method, Apparatus and Computing Device for Fault Detection and Recovery", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI), and more specifically, to a method, apparatus, and computing device for fault detection and recovery. Background Technology

[0003] AI models are a collective term for mathematical algorithms built upon the principles of artificial intelligence, and they form the foundation for solving specific problems using AI. In large-scale neural networks and high-precision computational tasks, the long-term stable training and inference of AI models has become a crucial research direction in the field of artificial intelligence. However, during the long-term stable training of AI models, accuracy issues can arise (e.g., overflow problems in forward computation, gradient explosion problems in backpropagation, tensor anomalous zeros, etc.), potentially leading to serious deviations in the AI ​​model's computational results and consequently affecting the overall training and inference performance of the AI ​​model.

[0004] One related technical solution still relies primarily on manual methods to locate accuracy faults in AI models. For example, technicians need to analyze the AI ​​model based on computation logs and experience to determine if accuracy faults exist. This not only requires high manpower and time costs, but also results in slow accuracy problem location and an inability to achieve real-time monitoring, thus hindering timely recovery from AI model accuracy faults.

[0005] Therefore, how to quickly detect and restore the accuracy of AI models has become a pressing technical problem that needs to be solved. Summary of the Invention

[0006] This application provides a method, apparatus, and computing device for fault detection and recovery, which can quickly detect accuracy faults in AI models, thereby timely recovering the accuracy faults of AI models and enabling AI models to perform stable training and inference.

[0007] In a first aspect, a method for fault detection and recovery is provided, comprising: acquiring a first tensor to be detected; performing precision fault detection on the first tensor to obtain a precision fault detection result for the first tensor; if the precision fault detection result indicates that the first tensor has a precision fault, determining a target operator related to the first tensor, wherein the input data or output data of the target operator is the first tensor, and the target operator is an operator in an artificial intelligence (AI) model; and recovering the precision fault existing in the first tensor based on the target operator.

[0008] The aforementioned technical solution determines whether an AI model has precision faults by detecting the presence of precision faults in the tensors generated during training. This eliminates the need for extensive offline audits of log data based on experience and cumbersome manual intervention, allowing for rapid detection of precision faults. Furthermore, after detecting tensors with precision faults, the AI ​​model can be promptly restored based on the target operator associated with these tensors, enabling stable training and inference.

[0009] In conjunction with the first aspect, in certain implementations of the first aspect, if outliers and / or isolated values ​​exist among the elements contained in the first tensor, it is determined that the first tensor has a precision fault.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: detecting whether there are outliers and / or isolated values ​​among the elements contained in the first tensor.

[0011] For example, the above outliers may include, but are not limited to: infinity (INF), not a number (NaN), and all zeros.

[0012] For example, the outliers mentioned above refer to values ​​that are close to or about to exceed the range that a computer can represent.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, if the first tensor does not meet the data configuration of the target operator, the first tensor is adjusted according to the data configuration of the target operator to obtain the second tensor.

[0014] In the above technical solution, the precision faults of the first tensor can be recovered according to the data configuration of the target operator, preventing precision faults caused by improper data configuration. At the same time, since the overhead required to adjust the first tensor according to the data configuration of the target operator is small, this precision fault recovery method can also reduce the recovery overhead.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, the attributes of the first tensor are adjusted according to the data configuration of the target operator to obtain the second tensor, wherein the attributes of the first tensor may include at least one of the following attributes: the shape of the first tensor, the numerical type of the first tensor, and the data range of the first tensor.

[0016] For example, the shape of a tensor refers to its size in each dimension. The shape is the basis for defining the structure and dimensions of a tensor and is usually represented as a tuple of integers. For example, a tensor with shape (3,4) means that it has 3 axes, with 3 elements on the first axis and 4 elements on the second axis.

[0017] For example, the numeric type of a tensor refers to the data type of the elements in the tensor, such as integers, floating-point numbers, etc. For instance, PyTorch provides various numeric types, such as FloatTensor (32-bit floating-point), DoubleTensor (64-bit floating-point), IntTensor (32-bit signed integer), etc.

[0018] For example, the data range of a tensor refers to the range of values ​​that its elements can take. This typically depends on the numeric type and the specific application scenario. For instance, for an integer tensor, its data range depends on the specific integer type (e.g., the range of an 8-bit signed integer is [-128, 127]).

[0019] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: obtaining the first tensor from a first register; obtaining the second tensor from a second register; comparing the first tensor and the second tensor to determine whether the second tensor has the precision fault.

[0020] In the above technical solution, since the data transmission speed of the register is relatively fast, the first tensor and the second tensor are stored in the first register and the second register respectively. When it is necessary to compare the first tensor and the second tensor, the first tensor and the second tensor can be directly obtained from the first register and the second register. By comparing the first tensor and the second tensor, it can be determined whether the fault has been recovered, thereby improving the recovery efficiency of accuracy faults.

[0021] In conjunction with the first aspect, in some implementations of the first aspect, the precision of the first tensor is improved by increasing the precision of the target operator, thereby obtaining the third tensor.

[0022] In the above technical solution, the accuracy of the target operator is improved, which means the accuracy of all input and output data related to the target operator is also improved. Since the first tensor serves as the input or output data of the target operator, the accuracy of the first tensor is also improved, thereby avoiding the overflow problem caused by the low accuracy of the first tensor.

[0023] In conjunction with the first aspect, in some implementations of the first aspect, when the second tensor has this precision fault, the precision of the second tensor is improved by improving the precision of the target operator, thereby obtaining the third tensor.

[0024] In the above technical solution, since adjusting the attributes of the first tensor based on the data configuration of the target operator requires relatively little overhead, the attributes of the first tensor can be adjusted first to obtain the second tensor. If the second tensor does not have precision faults, it means that the precision fault problem has been solved, and there is no need to further improve the precision of the second tensor. If the second tensor still has precision faults, it means that the precision fault problem cannot be solved by the above low-cost method. In this case, the precision of the second tensor can be further improved by increasing the precision of the target operator, avoiding the overflow problem caused by the low precision of the second tensor, thereby reducing the overall overhead of precision fault recovery.

[0025] In conjunction with the first aspect, in some implementations of the first aspect, the first tensor is the output data of the target operator, and the target operator outputs the fourth tensor by calling the target function, which is used to achieve the continuity of the input data of the target operator in memory.

[0026] In the above technical solution, since the objective function is used to ensure the continuity of the input data of the objective operator in memory, the objective operator can be correctly loaded into the input variables by calling the objective function, thereby ensuring the correctness of the output tensor generated by the objective operator.

[0027] In conjunction with the first aspect, in some implementations of the first aspect, the above objective function is the contiguous() function.

[0028] In the above technical solution, since the `contiguous()` function rearranges the elements of a tensor to ensure its contiguous layout in memory, the target `contiguous()` function can be called before the target operator is executed. If the `contiguous()` function checks that the elements of the target operator's input tensor are not contiguous in memory, it can allocate a new memory block and copy the elements of the input tensor into this new memory block. During the copying process, the elements of the input tensor will be rearranged in a contiguous manner. Finally, the `contiguous()` function returns a contiguous input variable to the target operator, allowing the target operator to correctly load the input variable, thereby ensuring the correctness of the output tensor generated by the target operator.

[0029] In conjunction with the first aspect, in some implementations of the first aspect, the first tensor is the output data of the target operator, and in the case that the third tensor has the precision fault, the target operator outputs the fourth tensor by calling the target function.

[0030] In the above technical solution, in order to reduce the overhead of precision faults, the first tensor can be recovered to obtain the third tensor in a way with lower overhead. If the third tensor still has precision faults, it means that the precision fault problem cannot be solved by the method with lower overhead. Then, the fourth tensor is output by calling the target function through the target operator, avoiding precision faults caused by memory overload, thereby reducing the overhead of precision fault recovery as a whole.

[0031] In conjunction with the first aspect, in some implementations of the first aspect, the first tensor is the output data of the target operator, and the CPU is switched to execute the target operator, which outputs the fifth tensor.

[0032] In the aforementioned technical solutions, on the one hand, GPUs (or NPUs) may reduce computational precision to optimize performance. Typically, GPUs (or NPUs) use single-precision floating-point numbers (float) for operations, while CPUs may default to using double-precision floating-point numbers (double). Double-precision floating-point numbers use 64 bits to represent a number, providing higher precision and a larger numerical range compared to the 32-bit representation of single-precision floating-point numbers. Therefore, the target operator can be switched from the GPU (or NPU) to the CPU for recalculation to avoid precision failures caused by the target operator being executed on the GPU (or NPU). On the other hand, the hardware architecture of GPUs (or NPUs) contains thousands of small cores. Many simple tasks are executed simultaneously by these small cores. In some cases, these tasks may have data dependencies or error propagation effects. When these tasks of the target operator are executed in parallel on the GPU, errors may accumulate and lead to precision failures. While CPUs have a relatively small number of cores, each core possesses strong computational power. This avoids accuracy failures caused by the accumulation of errors due to data dependencies. Therefore, the target operator can be recalculated from the GPU (or NPU) to the CPU, thus mitigating the accuracy failures that occur when the target operator is executed on the GPU (or NPU). This ensures the stability and reliability of the target operator's computation process.

[0033] In conjunction with the first aspect, in some implementations of the first aspect, the first tensor is the output data of the target operator. In the event of a precision fault in the fourth tensor, the CPU is switched to execute the target operator, which outputs the fifth tensor.

[0034] In the above technical solution, since the overhead of switching to the CPU to execute the target operator is large, in order to reduce the overhead of precision faults, the first tensor can be recovered to obtain the fourth tensor in a way with lower overhead. If the fourth tensor still has precision faults, it means that the precision fault problem cannot be solved by the method with lower overhead. Then, the fifth tensor is output by switching to the CPU to execute the target operator, avoiding precision faults caused by the GPU (or NPU) itself, thereby reducing the overhead of precision fault recovery as a whole.

[0035] In conjunction with the first aspect, in some implementations of the first aspect, before performing precision fault detection on the first tensor, the method further includes: downsampling the first tensor to obtain a sampled first tensor; the precision fault detection on the first tensor includes: performing precision fault detection on the sampled first tensor to determine the precision fault detection result of the first tensor.

[0036] In the above technical solution, the first tensor can be downsampled and the sampled first tensor can be used for accuracy fault detection, thereby reducing the computational load of the accuracy fault detection process.

[0037] In conjunction with the first aspect, in some implementations of the first aspect, the first tensor is sampled diagonally to obtain the sampled first tensor.

[0038] In the above technical solution, sampling is performed along the double diagonal of the first tensor to obtain the sampled first tensor. Outliers and / or unusual values ​​are generally more likely to appear along the diagonal of a tensor. By sampling the first tensor along its double diagonal, the sampled first tensor retains the outliers and / or unusual values, thereby reducing the computational load of the accuracy fault detection process while improving the accuracy of fault detection.

[0039] Secondly, a fault detection and recovery apparatus is provided, comprising: an acquisition module, a detection module, a determination module, and a recovery module. The acquisition module acquires a first tensor to be detected; the detection module performs precision fault detection on the first tensor to obtain a precision fault detection result; the determination module, when the precision fault detection result indicates that the first tensor has a precision fault, determines a target operator related to the first tensor, wherein the input data or output data of the target operator is the first tensor, and the target operator is an operator in an artificial intelligence (AI) model; the recovery module recovers the precision fault in the first tensor based on the target operator.

[0040] In conjunction with the second aspect, in some implementations of the second aspect, the detection module is specifically used to: determine that the first tensor has a precision fault when there are outliers and / or irregularities in the elements contained in the first tensor.

[0041] In conjunction with the second aspect, in some implementations of the second aspect, the detection module is specifically used to: detect whether there are outliers and / or isolated values ​​among the elements contained in the first tensor.

[0042] In conjunction with the second aspect, in some implementations of the second aspect, the recovery module is specifically used to: adjust the first tensor according to the data configuration of the target operator when the first tensor does not meet the data configuration of the target operator, so as to obtain the second tensor.

[0043] In conjunction with the second aspect, in some implementations of the second aspect, the acquisition module is further configured to acquire the first tensor from the first register and the second tensor from the second register; the determination module is further configured to compare the first tensor and the second tensor to determine whether the second tensor has the precision fault.

[0044] In conjunction with the second aspect, in some implementations of the second aspect, the recovery module is specifically used to: improve the precision of the first tensor by improving the precision of the target operator, and obtain the third tensor.

[0045] In conjunction with the second aspect, in some implementations of the second aspect, the recovery module is specifically used to: improve the accuracy of the second tensor by increasing the accuracy of the target operator in the case of the accuracy fault of the second tensor, and obtain the third tensor.

[0046] In conjunction with the second aspect, in some implementations of the second aspect, the first tensor is the output data of the target operator, and the recovery module is specifically used for: the target operator outputs the fourth tensor by calling the target function, which is used to achieve the continuity of the input data of the target operator in memory.

[0047] In conjunction with the second aspect, in some implementations of the second aspect, the first tensor is the output data of the target operator, and the recovery module is specifically used to: in the case that the third tensor has the precision fault, the target operator outputs the fourth tensor by calling the target function.

[0048] In conjunction with the second aspect, in some implementations of the second aspect, the first tensor is the output data of the target operator, and the recovery module is specifically used to: switch to the CPU to execute the target operator, and the target operator outputs the fifth tensor.

[0049] In conjunction with the second aspect, in some implementations of the second aspect, the first tensor is the output data of the target operator, and the recovery module is specifically used to: switch to the CPU to execute the target operator when the fourth tensor has the precision fault, and the target operator outputs the fifth tensor.

[0050] In conjunction with the second aspect, in some implementations of the second aspect, the device further includes: a sampling module, used to downsample the first tensor before performing accuracy fault detection on the first tensor to obtain a sampled first tensor; the detection module is specifically used to: perform accuracy fault detection on the sampled first tensor and determine the accuracy fault detection result of the first tensor.

[0051] In conjunction with the second aspect, in some implementations of the second aspect, the sampling module is specifically used to: perform bi-diagonal sampling on the first tensor to obtain the sampled first tensor.

[0052] It should be understood that for the beneficial effects of the second aspect and its various implementations, please refer to the first aspect and its various implementations; they will not be repeated here.

[0053] Thirdly, a computing device is provided, including a processor and a memory, and optionally, an input / output interface. The processor controls the input / output interface to send and receive information, the memory stores a computer program, and the processor retrieves and runs the computer program from the memory, causing the computing device to execute the method of the first aspect or any possible implementation thereof.

[0054] Optionally, the processor can be a general-purpose processor, which can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, integrated circuit, etc.; when implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. This memory can be integrated into the processor or located outside the processor and exist independently.

[0055] Fourthly, a chip is provided that acquires and executes instructions to implement the methods described in the first aspect and any implementation thereof.

[0056] Optionally, as one implementation, the chip includes a processor and a data interface, through which the processor reads instructions stored in the memory and executes the methods in the first aspect and any implementation thereof.

[0057] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to perform the method in the first aspect and any implementation thereof.

[0058] Fifthly, a computer program product containing instructions is provided, which, when executed by a computing device, causes the computing device to perform the methods described in the first aspect and any implementation thereof.

[0059] In a sixth aspect, a computer-readable storage medium is provided, including computer program instructions that, when executed by a computing device, perform the method as described in the first aspect and any implementation thereof.

[0060] As examples, these computer-readable storage devices include, but are not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), flash memory, electrically EPROM (EEPROM), and hard drive.

[0061] Alternatively, as one implementation method, the aforementioned storage medium can specifically be a non-volatile storage medium. Attached Figure Description

[0062] Figure 1 is a schematic flowchart of a fault detection and recovery method provided in an embodiment of this application.

[0063] Figure 2 is a schematic block diagram of a fault detection and recovery system architecture provided in an embodiment of this application.

[0064] Figure 3 is a schematic diagram of bi-diagonal sampling of a first tensor provided in an embodiment of this application.

[0065] Figure 4 is a schematic diagram of comparing a first tensor and a second tensor provided in an embodiment of this application.

[0066] Figure 5 is a schematic diagram of comparing the first tensor and the fourth tensor provided in an embodiment of this application.

[0067] Figure 6 is a schematic block diagram of a fault detection and recovery device 600 provided in an embodiment of this application.

[0068] Figure 7 is a schematic diagram of the architecture of a computing device 1500 provided in an embodiment of this application. Detailed Implementation

[0069] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0070] This application will present various aspects, embodiments, or features relating to systems comprising multiple devices, components, modules, etc. It should be understood and appreciated that individual systems may include additional devices, components, modules, etc., and / or may not include all devices, components, modules, etc. discussed in conjunction with the accompanying drawings. Furthermore, combinations of these approaches are also possible.

[0071] Furthermore, in the embodiments of this application, the words "exemplary," "for example," etc., are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the term "exemplary" is intended to present the concept in a concrete manner.

[0072] In the embodiments of this application, "corresponding" and "corresponding" can sometimes be used interchangeably. It should be noted that when the distinction is not emphasized, their intended meanings are consistent.

[0073] The business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0074] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0075] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0076] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0077] The basic principle of AI is to combine massive amounts of data with powerful computing capabilities and intelligent algorithms to build an AI model that solves specific problems. This AI model can automatically summarize and learn potential patterns or features from the data, thereby achieving a way of thinking that is close to that of humans.

[0078] AI models are a general term for mathematical algorithms built upon the principles of artificial intelligence, and they form the foundation for using AI to solve specific problems. Depending on the specific methods and / or technologies used to implement artificial intelligence, AI models can also be called machine learning models, deep learning models, or reinforcement learning models.

[0079] Before any AI model can be used to solve a specific technical problem, it needs to be trained. AI model training refers to using a specified initial model to compute on training data, and then adjusting the parameters of the initial model based on the computation results, so that the model gradually learns certain patterns and acquires specific functions. Once trained and possessing stable functionality, the AI ​​model can be used for inference. AI model inference is the process of using the trained AI model to compute on input data and obtain predicted inference results.

[0080] Typically, the training process of an AI model includes forward propagation (FP) and back propagation (BP) computations. In FP, training data, such as pixel information from an input image, is acquired and passes through the input and hidden layers of the neural network, ultimately outputting a prediction (predicted value) from the output layer. In BP, the current predicted value is compared with prior knowledge (also known as the ground truth), and the weights of the parameter matrix of each layer in the neural network are updated based on the difference between the two, thus minimizing the error between the predicted value and the prior knowledge.

[0081] In large-scale neural networks and high-precision computational tasks, the long-term stable training and inference of AI models has become an important research direction in the field of artificial intelligence. However, during the long-term stable training of AI models, accuracy problems (also known as accuracy faults) may occur, which may lead to serious deviations in the calculation results of the AI ​​model, thereby affecting the overall training and inference performance of the AI ​​model.

[0082] For example, common accuracy problems in AI models mainly fall into the following categories:

[0083] 1. Overflow problem in forward computation

[0084] Overflow occurs when a computer performs arithmetic operations and the result exceeds the range that the computer can represent. Specifically, numbers in computers are typically stored in binary form, and each data type has a fixed number of bits and a fixed range of representation. When the result of an operation exceeds this range, overflow occurs.

[0085] Overflow has a significant impact on the results of forward computations. For example, when the result exceeds the representation range, the computer may return an incorrect value or cause the program to crash, leading to incorrect calculation results. Overflow can also reduce data precision, affecting the accuracy of the calculation results and resulting in a loss of data precision. Furthermore, in some security-sensitive applications, overflows can be exploited to execute malicious code or damage the system, thus creating security problems.

[0086] In forward computation, overflow is mainly divided into two types: overflow and underflow. Overflow occurs when the result of an operation exceeds the maximum value in the positive direction. For example, in an 8-bit binary integer, the maximum value is 127. If the result exceeds this maximum value, overflow occurs. Underflow occurs when the result of an operation falls below the minimum value in the negative direction or is close to zero. For example, in an 8-bit binary integer, the minimum value is -128. If the result is less than this minimum value, due to the limitations of the computer's internal representation, the actual result may not be accurately represented, resulting in underflow.

[0087] 2. Gradient explosion problem in backpropagation

[0088] Gradient explosion refers to the phenomenon in which gradient values ​​become extremely large during the backpropagation process of a neural network, even exceeding the network's processing range, leading to unstable weight updates and even preventing the network from converging.

[0089] Gradient explosion during backpropagation can cause excessively large weight updates, leading the network to skip the optimal weight values ​​during training and preventing convergence. Furthermore, extremely large gradients can cause numerical overflow, resulting in non-numeric (NaN) or infinitely large values, making model training impossible.

[0090] 3. The vanishing gradient problem in backpropagation

[0091] Vanishing gradient refers to the phenomenon in neural networks where the gradient gradually decreases and eventually approaches zero during backpropagation, as it is passed layer by layer. This causes earlier layers (those closer to the input) to not receive enough gradient information to update their weights, resulting in them learning very little or nothing at all. Consequently, the neural network cannot effectively learn the feature representations of the input data, thus affecting the overall learning ability of the network and making its training unstable.

[0092] 4. Tensor anomaly with all zeros

[0093] The tensor anomalous all-zero problem refers to the situation where all elements of a tensor unexpectedly become zero. For example, the input data may not have been loaded or preprocessed correctly, or it may have undergone inappropriate normalization or scaling, causing the tensor to be initialized with all zeros. Another example is in a neural network where improper configuration of certain layers or parameters can lead to gradient vanishing during backpropagation, rendering weight updates ineffective and causing the tensor to remain in an all-zero state.

[0094] If the accuracy issues of the aforementioned AI models are not detected and corrected in a timely manner, they often reduce the accuracy of the AI ​​model's calculation results and may even interrupt the execution of the entire task. Therefore, how to detect and correct the accuracy issues of AI models in a timely manner has become a major challenge in the industry.

[0095] In one related technical solution, on the one hand, the identification of accuracy issues in AI models still relies heavily on manual methods. For example, technicians need to analyze the fault causes of each operator based on calculation logs and experience to determine whether each operator has accuracy overflow or other accuracy problems. This not only requires high manpower and time costs, but also results in slow accuracy problem location and makes real-time monitoring impossible. On the other hand, the repair of accuracy problems also mainly relies on manual repair. For example, technicians manually repair the identified accuracy problems, leading to slow recovery speeds and an inability to resolve accuracy faults promptly.

[0096] In view of this, embodiments of this application provide a method for fault detection and recovery, which can quickly locate accuracy faults in AI models and recover from them in a timely manner, thereby improving the training and inference effects of AI models.

[0097] Figure 1 is a schematic flowchart of a fault detection and recovery method provided in an embodiment of this application. For example, this method can be executed by a graphics processing unit (GPU) or a neural processing unit (NPU).

[0098] As shown in Figure 1, the method includes steps 110-140, which are described in detail below.

[0099] Step 110: Obtain the first tensor to be detected.

[0100] In this embodiment of the application, a first tensor to be detected can be obtained. This first tensor is any one of multiple tensors generated during the training of the artificial intelligence (AI) model.

[0101] This application does not specifically limit the implementation method of obtaining the first tensor to be detected from multiple stored tensors. For example, the first tensor to be detected can be obtained by traversing the multiple stored tensors. Or, the first tensor to be detected can be obtained by sampling the multiple stored tensors.

[0102] For example, as shown in Figure 2, the GPU or NPU may include a storage unit 210, which includes a tensor storage module 211 that stores multiple tensors generated by the AI ​​model during training. The fault location unit 220 includes an instruction and data dispatch module 221, which can retrieve the first tensor from the multiple tensors stored in the tensor storage module 211.

[0103] It should be understood that tensors are a generalization of multidimensional arrays, capable of representing arrays ranging from scalars (zero-dimensional arrays), vectors (one-dimensional arrays), matrices (two-dimensional arrays) to higher-dimensional arrays. During the training of AI models, most of the data (e.g., weights) is stored in the form of tensors.

[0104] Optionally, in some embodiments, the first tensor to be detected can be sparsified (also known as downsampling) to obtain the first tensor after sparsification, so as to reduce the amount of computation required for subsequent accuracy fault detection.

[0105] For example, as shown in Figure 2, after the instruction and data dispatch module 221 obtains the first tensor from the tensor storage module 211, it can also perform sparse sampling on the first tensor. For example, for the two-dimensional first tensor, sampling is performed on the double diagonal of the first tensor according to the double diagonal direction to obtain the sampled first tensor. The sampled first tensor retains the elements in the double diagonal direction of the first tensor.

[0106] It should be understood that the above double diagonals refer to the main diagonal and the secondary diagonal (i.e., the diagonal from the upper right corner to the lower left corner of the matrix).

[0107] In the above technical solution, outliers and / or isolated values ​​are more likely to appear on the diagonal of the tensor. Therefore, the elements in the double diagonal direction of the first tensor are retained in the first tensor after sampling, so that the first tensor after sampling can retain outliers and / or isolated values ​​in the first tensor. This reduces the computational load of the precision fault detection process and improves the accuracy of precision fault detection.

[0108] For example, as shown in Figure 3, assuming the number of elements in the first tensor to be detected is bs*sl*hd, after performing double diagonal sampling on this first tensor, the number of elements in the obtained double diagonal sampled first tensor is 2*min(bs*sl*hd). That is, after performing double diagonal sampling on the first tensor, the number of elements in the first tensor is reduced from bs*sl*hd to 2*min(bs*sl*hd).

[0109] Step 120: Perform precision fault detection on the first tensor and determine the precision fault detection result of the first tensor.

[0110] In this embodiment of the application, after obtaining the first tensor to be detected, precision fault detection can be performed on the first tensor to determine whether there is a precision fault in the first tensor.

[0111] In one possible implementation, outliers and / or isolated values ​​can be detected among the elements contained in the first tensor. For example, each element in the first tensor can be traversed to determine whether outliers and / or isolated values ​​exist. If outliers and / or isolated values ​​are found among the elements contained in the first tensor, the accuracy fault detection result of the first tensor is used to indicate that the first tensor has an accuracy fault.

[0112] For example, as shown in Figure 2, the GPU or NPU may include a fault location unit 220, which includes an outlier and outlier calculation module 222. The outlier and outlier calculation module 222 can detect whether there are outliers and / or outliers in the elements contained in the first tensor.

[0113] For example, the above outliers may include, but are not limited to: infinity (INF), not a number (NaN), and all zeros.

[0114] The INF value mentioned above is a numerical value that exceeds the range that a computer can represent. For example, when a value exceeds the range of floating-point numbers, the data will be represented as infinity INF. Similarly, when a non-zero number is divided by zero, the result is infinity INF.

[0115] The term NaN refers to a result or operation that cannot be represented as a valid floating-point number. For example, taking the square root of a negative number or performing a logarithmic operation on a negative number results in a NaN because the result cannot be represented as a valid floating-point number. Additionally, certain special cases of operations (such as dividing 0.0 by 0.0) can also result in NaN.

[0116] The term "all zeros" refers to the tensor having all elements with values ​​of 0.

[0117] For example, the outliers mentioned above refer to values ​​that are close to or about to exceed the range that a computer can represent.

[0118] Optionally, in some embodiments, the outlier and outlier calculation module 222 can also determine the accuracy fault detection result of the first tensor by detecting whether there is an accuracy fault in the tensor after sparse sampling. For example, if the outlier and outlier calculation module 222 detects outliers and / or outliers in the tensor after sparse sampling, it can determine that the first tensor has an accuracy fault.

[0119] Step 130: If the accuracy fault detection result of the first tensor indicates that the first tensor has an accuracy fault, determine the target operator related to the first tensor.

[0120] In this embodiment of the application, when the accuracy fault detection result of the first tensor indicates that the first tensor has an accuracy fault, the target operator related to the first tensor is determined.

[0121] For example, as shown in Figure 2, the outlier and outlier calculation module 222 can determine the target operator associated with the first tensor and store the information of the target operator in the storage unit 210. For instance, the outlier and outlier calculation module 222 can store the name of the target operator in the tensor outlier storage module 212 of the storage unit 210.

[0122] It should be understood that operators are the basic operational units for computation in AI models. They are responsible for performing various mathematical and logical operations, such as addition, multiplication, convolution, and activation functions. In AI models, operators are typically implemented as independent functions or modules that can accept one or more tensors as input and produce a new tensor as output. These operators can be repeatedly called during the training and inference processes of the AI ​​model to implement complex computational logic.

[0123] The first tensor and the target operator can have a one-to-one relationship. For example, the first tensor can be used as the input data of the target operator, or the first tensor can be used as the output data of the target operator. This application does not specifically limit this.

[0124] For example, suppose the input data of the first operator includes the first tensor mentioned above; this first operator can serve as the target operator. In another example, suppose the output data of the second operator includes the first tensor mentioned above; this second operator can serve as the target operator.

[0125] The aforementioned target operator can also be called a high-risk operator.

[0126] Step 140: Recover the precision faults in the first tensor based on the target operator.

[0127] In this embodiment of the application, the precision faults existing in the first tensor can be recovered based on the target operator, thereby eliminating the precision faults existing in the first tensor.

[0128] For example, as shown in Figure 2, a fault recovery unit 240 may be included in the GPU or NPU, which can recover precision faults in the first tensor based on the target operator.

[0129] The aforementioned technical solution determines whether an AI model has precision faults by detecting the presence of precision faults in the tensors generated during training. This eliminates the need for extensive offline audits of log data based on experience and cumbersome manual intervention, allowing for rapid detection of precision faults. Furthermore, after detecting tensors with precision faults, the AI ​​model can be promptly restored based on the target operator associated with these tensors, enabling stable training and inference.

[0130] The following examples illustrate different implementation methods for recovering the precision faults of the first tensor based on the target operator.

[0131] Implementation method 1 can check whether the first tensor meets the data configuration criteria of the target operator. If the first tensor does not meet the data configuration criteria of the target operator, it can be adjusted according to the data configuration criteria of the target operator to obtain the second tensor.

[0132] For example, as shown in Figure 2, the GPU or NPU may include a fault recovery unit 240, which includes an instruction and data dispatch module 241 and a parameter configuration checking module 242. The instruction and data dispatch module 241 is used to obtain information about a first tensor and a target operator from the storage unit 210, and the parameter configuration checking module 242 is used to check whether the first tensor meets the data configuration criteria of the target operator.

[0133] For example, assuming the first tensor is the input data of the target operator, we can check whether the first tensor meets the input data configuration criteria of the target operator. If the first tensor does not meet the input data configuration criteria of the target operator, we can adjust the first tensor according to the input data configuration criteria of the target operator to obtain the second tensor.

[0134] In another example, assuming the first tensor is the output data of the target operator, we can check whether the first tensor meets the output data configuration criteria of the target operator. If the first tensor does not meet the output data configuration criteria of the target operator, we can adjust the first tensor according to the output data configuration criteria of the target operator to obtain the second tensor.

[0135] For example, taking the first tensor as the input data of the target operator, the attributes of the first tensor can be checked and adjusted according to the input data configuration standard of the target operator. The attributes of the first tensor can include at least one of the following: the shape of the first tensor, the numerical type of the first tensor, the numerical range of the first tensor, etc., thereby preventing accuracy failures caused by improper input data.

[0136] For example, the shape of a tensor refers to its size in each dimension. The shape is the basis for defining the structure and dimensions of a tensor and is usually represented as a tuple of integers. For example, a tensor with shape (3,4) means that it has 3 axes, with 3 elements on the first axis and 4 elements on the second axis.

[0137] For example, the numeric type of a tensor refers to the data type of the elements in the tensor, such as integers, floating-point numbers, etc. For instance, PyTorch provides various numeric types, such as FloatTensor (32-bit floating-point), DoubleTensor (64-bit floating-point), IntTensor (32-bit signed integer), etc.

[0138] For example, the data range of a tensor refers to the range of values ​​that its elements can take. This typically depends on the numeric type and the specific application scenario. For instance, for an integer tensor, its data range depends on the specific integer type (e.g., the range of an 8-bit signed integer is [-128, 127]).

[0139] For example, the shape of the first tensor can be checked according to the shape standard of the input data configured by the target operator. If the shape of the first tensor does not conform to the shape standard of the input data configured by the target operator, the shape of the first tensor can be adjusted based on the shape standard of the input data to obtain the second tensor mentioned above.

[0140] For example, the value type of the first tensor can be checked according to the value type standard of the input data configured by the target operator. If the value type of the first tensor does not conform to the value type standard of the input data configured by the target operator, the value type of the first tensor can be adjusted based on the value type standard of the input data to obtain the aforementioned second tensor.

[0141] For example, the numerical range of the first tensor can be checked according to the numerical range standard of the input data configured by the target operator. If the numerical range of the first tensor does not conform to the numerical range standard of the input data configured by the target operator, the numerical range of the first tensor can be adjusted based on the numerical range standard of the input data to obtain the aforementioned second tensor.

[0142] In this embodiment, after obtaining the second tensor, the first and second tensors can be compared to determine whether the second tensor still contains outliers and / or isolated values ​​from the first tensor. If the comparison determines that the second tensor still contains outliers and / or isolated values ​​from the first tensor, it can be determined that the second tensor still has precision issues. If the comparison determines that the second tensor does not contain outliers and / or isolated values ​​from the first tensor, it can be determined that the second tensor does not have precision issues, thus eliminating the precision issues present in the first tensor.

[0143] The following describes in detail how to determine whether outliers and / or isolated values ​​from the first tensor still exist in the second tensor.

[0144] In one possible implementation, since the data transfer speed of registers is relatively fast, the first tensor and the second tensor are stored in the first register and the second register respectively. When it is necessary to compare the first tensor and the second tensor, the first tensor and the second tensor can be directly obtained from the first register and the second register. By comparing the first tensor and the second tensor, it can be determined whether the fault has been recovered, thereby improving the recovery efficiency of accuracy faults.

[0145] For example, as shown in Figure 2, a GPU or NPU may include a tensor computation unit 230, which includes multiple registers (e.g., register 231, register 232) and a matrix operation core 233. Register 231 stores a first tensor, register 232 stores a second tensor, and the matrix operation core 233 retrieves the stored first tensor from register 231 and the stored second tensor from register 232, comparing the first and second tensors. If the matrix operation core 233 determines that the second tensor does not contain outliers and / or isolated values ​​found in the first tensor, it can be determined that the precision fault in the first tensor has been resolved. If the matrix operation core 233 determines that the second tensor still contains outliers and / or isolated values ​​found in the first tensor, it can be determined that the precision fault in the first tensor has not been resolved, and further fault recovery is required.

[0146] For example, as shown in Figure 4, suppose the first tensor stored in register 231 contains the following elements: [1, 3, INF, 7, 8], and suppose the second tensor stored in register 232 contains the following elements: [1, 3, 5, 7, 8]. The matrix operation core 233 determines that the second tensor does not contain the outlier INF found in the first tensor by comparing the first tensor and the second tensor. Therefore, the matrix operation core 233 can determine that the second tensor does not contain the outlier and / or outlier found in the first tensor, and the precision fault problem of the first tensor has disappeared or been resolved.

[0147] In another possible implementation, the second tensor can also be stored in the aforementioned storage unit 210. When it is necessary to compare the first tensor and the second tensor, the second tensor can be retrieved from storage unit 210, and the fault recovery can be determined by comparing the first tensor and the second tensor. For example, matrix operation core 233 is used to retrieve the first tensor and the second tensor from storage unit 210 respectively, and compare the first tensor and the second tensor.

[0148] Implementation method 2 can improve the precision of the target operator, thereby improving the precision of the first tensor to obtain the third tensor, which can avoid the overflow problem caused by the low precision of the tensor.

[0149] For example, the precision of the target operator can be improved, which means improving the precision of all input and output data related to the target operator. Since the first tensor serves as either input or output data for the target operator, its precision is also improved, resulting in the third tensor. Thus, by improving the precision of the target operator, the precision of the first tensor can be improved, thereby avoiding overflow problems caused by the low precision of the first tensor.

[0150] It should be understood that the third tensor mentioned above was obtained by improving the accuracy of the first tensor.

[0151] For example, as shown in FIG2, the fault recovery unit 240 includes a high-precision switching module 243, which is used to improve the precision of the target operator, thereby improving the precision of the first tensor and obtaining the third tensor.

[0152] Implementation method 3 can improve the precision of the target operator, thereby improving the precision of the second tensor to obtain the third tensor, which can avoid the overflow problem caused by the low precision of the tensor.

[0153] For example, after obtaining the second tensor, if the second tensor still has an accuracy fault, the accuracy of the target operator can be improved, thereby improving the accuracy of the second tensor and obtaining the third tensor.

[0154] It should be understood that since adjusting the first tensor to obtain the second tensor according to implementation method 1 requires less overhead, the properties of the first tensor can be adjusted first to obtain the second tensor. If the second tensor does not have a precision fault, it means the precision fault problem has been solved, and there is no need to further improve the precision of the second tensor. If the second tensor still has a precision fault, it means that the precision fault problem cannot be solved by the above-mentioned low-overhead implementation method 1. In this case, the precision fault problem can be solved by further improving the precision of the target operator, thereby avoiding the overflow problem caused by the low precision of the second tensor and reducing the overall overhead of precision fault recovery.

[0155] For example, as shown in Figure 2, the high-precision switching module 243 is used to improve the precision of the target operator, thereby improving the precision of the second tensor and obtaining the third tensor.

[0156] In implementation methods 2 and 3 described above, the first tensor can be either the input data of the target operator or the output data of the target operator. This application embodiment does not specifically limit this.

[0157] In implementation methods 2 and 3 described above, after obtaining the third tensor by improving the precision of the second tensor, the tensor calculation unit 230 can compare the first and third tensors to determine whether the third tensor still contains outliers and / or isolated values ​​from the first tensor. The process of comparing the first and third tensors is similar to the process of comparing the first and second tensors; please refer to the description above for details, which will not be repeated here.

[0158] Implementation method 4: When the first tensor is the output data of the target operator, the target function can be called during the execution of the target operator so that the target operator outputs a new tensor relative to the first tensor (e.g., the fourth tensor), thereby avoiding precision failures caused by memory overload.

[0159] For example, as shown in Figure 2, the fault recovery unit 240 includes a memory tamper recovery module 244, which is used to call the target function before the target operator is executed, so that the target operator outputs the fourth tensor mentioned above.

[0160] It should be understood that if the target operator encounters a memory overflow issue when accessing memory, the output tensor of the target operator may result in an abnormal situation where all zeros are displayed because the target operator cannot correctly load the input data from that memory. This memory overflow includes, but is not limited to, memory out-of-bounds access.

[0161] The aforementioned memory out-of-bounds access refers to a situation where the target operator accesses memory beyond its allocated or permitted memory range. That is, the target operator accesses memory that was not allocated to it, or memory that is not permitted to it. Assume the input tensor of the target operator is non-contiguous, with elements not stored contiguously, and there may be free memory regions or other data between these elements. If the target operator attempts to load elements from the input tensor into memory, due to the non-contiguous memory layout, accessing these elements requires complex index calculations. If the memory location of an element in the input tensor is incorrectly calculated (e.g., due to an index error or a misunderstanding of the memory layout), it may access a memory region outside the tensor, leading to a memory out-of-bounds access situation. This results in the target operator being unable to load the correct input tensor, causing the tensor output by the target operator to contain an abnormal set of all zeros.

[0162] The objective function described above is used to ensure the continuity of the input tensor of the objective operator in memory, so that the objective operator can obtain a correct input variable from memory.

[0163] As an example, the objective function mentioned above is the `contiguous()` function. Since the `contiguous()` function rearranges the elements of a tensor to ensure its contiguous layout in memory, it can be called before the objective operator is executed. If the `contiguous()` function checks that the elements of the input tensor of the objective operator are not contiguous in memory, it can allocate a new memory block and copy the elements of the input tensor into this new memory block. During the copying process, the elements of the input tensor will be rearranged in a contiguous manner. Finally, the `contiguous()` function returns a contiguous input variable to the objective operator, allowing the objective operator to correctly load the input variable and thus ensuring the correctness of the output tensor generated by the objective operator.

[0164] Optionally, in some embodiments, if the output tensor of the target operator needs to be used as input data for other operators, and the other operators require their input data to be continuous, the above-mentioned target function can be called after the target operator is executed, thereby ensuring the continuity of the elements of the output tensor of the target operator in memory.

[0165] Implementation method 5: If the third tensor still has a precision fault, and the first tensor is the output data of the target operator, then the target function can be called before the target operator is executed, so that the target operator outputs the fourth tensor.

[0166] In the above technical solution, in order to reduce the overhead of precision faults, the first tensor can be recovered to obtain the third tensor in a way with lower overhead (such as the above implementation method 2 or implementation method 3). If the third tensor still has precision faults, it means that the precision fault problem cannot be solved by the method with lower overhead. Then, the fourth tensor is output by calling the target function through the target operator to avoid precision faults caused by memory overload, thereby reducing the overhead of precision fault recovery as a whole.

[0167] In this embodiment, after obtaining the fourth tensor, the first and fourth tensors can be compared to determine whether the fourth tensor still contains outliers and / or isolated values ​​from the first tensor. If the comparison determines that the fourth tensor still contains outliers and / or isolated values ​​from the first tensor, it can be determined that the fourth tensor still has a precision fault. If the comparison determines that the fourth tensor does not contain outliers and / or isolated values ​​from the first tensor, it can be determined that the fourth tensor does not have a precision fault, that is, the precision fault present in the first tensor has been eliminated.

[0168] For example, as shown in Figure 5, suppose the first tensor stored in register 231 contains the following elements: [0, 0, 0, 0, 0], and suppose the fourth tensor stored in register 232 contains the following elements: [1, 2, 3, 4, 5]. The matrix operation core 233 determines that the fourth tensor does not contain the abnormal all-zero problem found in the first tensor by comparing the first tensor and the fourth tensor. Therefore, the matrix operation core 233 can determine that the fourth tensor does not contain the abnormal value found in the first tensor, and the precision fault problem of the first tensor has disappeared or been resolved.

[0169] Implementation method 6: When the first tensor is the output data of the target operator, switch to the central processing unit (CPU) to execute the target operator, so that the target operator outputs a new tensor relative to the first tensor (e.g., the fifth tensor).

[0170] For example, as shown in Figure 2, the fault recovery unit 240 includes a CPU switching module 245, which is used to switch the target operator from the GPU (or NPU) to the CPU for recalculation, so that the target operator outputs a fifth tensor, thereby ensuring the stability and reliability of the calculation process of the target operator.

[0171] On the one hand, GPUs (or NPUs) may reduce computational precision to optimize performance. Typically, GPUs (or NPUs) use single-precision floating-point numbers (float) for operations, while CPUs may default to double-precision floating-point numbers (double). Double-precision floating-point numbers use 64 bits to represent a number, offering higher precision and a wider numerical range compared to the 32-bit representation of single-precision floating-point numbers. Therefore, the target operator can be switched from the GPU (or NPU) to the CPU for recalculation to avoid precision failures caused by the target operator being executed on the GPU (or NPU). On the other hand, the hardware architecture of GPUs (or NPUs) contains thousands of small cores. Many simple tasks are executed simultaneously by these cores. In some cases, these tasks may have data dependencies or error propagation effects. When these tasks of the target operator are executed in parallel on the GPU, errors may accumulate and lead to precision failures. While the CPU hardware architecture contains a relatively small number of cores, each core has strong computing power, which can avoid the accuracy failure caused by the accumulation of errors due to data dependencies. Therefore, the target operator can be switched from the GPU (or NPU) to the CPU for recalculation to avoid the accuracy failure caused by the target operator when it is executed on the GPU (or NPU).

[0172] In implementation method 7, if the fourth tensor still has a precision fault, and the first tensor is the output data of the target operator, the target operator is switched from GPU (or NPU) to CPU for execution, so that the target operator outputs the fifth tensor.

[0173] In the above technical solution, since the overhead of switching to the CPU to execute the target operator is large, in order to reduce the overhead of precision faults, the first tensor can be recovered to obtain the fourth tensor in a way with lower overhead (such as implementation method 5). If the fourth tensor still has precision faults, it means that the precision fault problem cannot be solved by the way with lower overhead. Then, the fifth tensor is output by switching to the CPU to execute the target operator, avoiding precision faults caused by the GPU (or NPU) itself, thereby reducing the overhead of precision fault recovery as a whole.

[0174] In implementation method 6 or 7 above, after executing the target operator on the CPU and obtaining the fifth tensor output by the target operator, the tensor calculation unit 230 can compare the first tensor and the fifth tensor to determine whether the fifth tensor still contains outliers and / or isolated values ​​from the first tensor. The process of comparing the first tensor and the fifth tensor is similar to the process of comparing the first tensor and the second tensor; please refer to the description above for details, which will not be repeated here.

[0175] It should be noted that the fault recovery unit 240 includes a parameter configuration check module 242, a high-precision switching module 243, a memory failure recovery module 244, and a CPU switching module 245. In this embodiment, at least one module in the fault recovery unit 240 can be used to recover the precision fault existing in the first tensor. Furthermore, this embodiment does not specifically limit the execution order of these at least one module.

[0176] For example, the precision fault of the first tensor can be recovered sequentially, from lowest to highest cost, through parameter configuration check module 242, high-precision switching module 243, memory collapse recovery module 244, and CPU switching module 245. After each fault recovery module in the fault recovery unit 240, the tensor calculation unit 230 compares the tensor before and after recovery to determine whether the precision fault has been recovered. If the precision fault has not been recovered after the current fault recovery module, the next precision fault recovery module is entered to continue the recovery process. If the precision fault has been recovered after the next precision fault recovery module, no further precision recovery modules are needed; the precision fault location and recovery log is output to a register, and the normal training process of the subsequent AI model continues.

[0177] The above technical solution forms a systematic recovery framework for positioning accuracy faults, which can adaptively solve various types of accuracy fault problems, has high scalability and adaptability, and thus can quickly recover from accuracy faults, enabling AI models to be trained and inferred stably.

[0178] This application does not specifically limit the application scenarios of the above-described fault detection and recovery methods. These application scenarios may include, but are not limited to, AI framework tools and server products that include various AI computing accelerator card architectures (for example, the server products may include, but are not limited to, mobile phones, tablets, smart cars, smart homes, etc.).

[0179] As an example, taking the application scenario of an AI framework as an example, the AI ​​framework enables the AI ​​framework to have fault detection and recovery functions by executing the above-mentioned fault detection and recovery methods.

[0180] For example, the aforementioned AI framework can provide users with an application programming interface (API), through which users can provide input information. After obtaining the user's input information through the API, the AI ​​framework determines the target operator (also known as a high-risk operator) in the AI ​​model running within the AI ​​framework by executing the aforementioned fault detection and recovery methods, and then recovers the accuracy faults existing in the AI ​​model through the target operator according to the aforementioned fault recovery methods.

[0181] The input information mentioned above can be all the operators of the AI ​​model uploaded by the user, or suspicious operators in the AI ​​model (suspected operators in the user's opinion). This application embodiment does not specifically limit this.

[0182] Optionally, after recovering from accuracy faults in the AI ​​model, the aforementioned AI framework can also display a fault detection and recovery report to the user via the API. This report may include, but is not limited to, information about the target operator, the type of accuracy fault, and the recovery method employed.

[0183] For example, suppose the AI ​​framework detects a precision fault in the first tensor of the AI ​​model and uses implementation method 1 described above to recover from the precision fault. Then, the target operator in the report returned by the AI ​​framework to the user can be an operator whose input data or output data is the first tensor; the type of precision fault in the report can be a parameter configuration problem; and the recovery method adopted in the report can be adjusting the first tensor according to the data configuration standard of the target operator.

[0184] For example, suppose the AI ​​framework detects a precision fault in the first tensor of the AI ​​model and uses implementation method 2 described above to recover from the precision fault. Then, the target operator in the report returned by the AI ​​framework to the user can be an operator whose input data or output data is the first tensor; the type of precision fault in the report can be a precision problem of the tensor or operator; and the recovery method adopted in the report can be to improve the precision of the target operator, thereby improving the precision of the second tensor.

[0185] For example, suppose the AI ​​framework detects a precision fault in the first tensor of the AI ​​model and uses implementation method 3 described above to recover from the precision fault. Then, the target operator in the report returned by the AI ​​framework to the user can be an operator whose input data or output data is the first tensor; the type of precision fault in the report can be a memory scramble fault; and the recovery method adopted in the report can be to call the contiguous() function during the execution of the target operator, so that the target operator outputs a new tensor.

[0186] For example, suppose the AI ​​framework detects a precision fault in the first tensor of the AI ​​model and uses implementation method 4 described above to recover from the precision fault. Then, the target operator in the report returned by the AI ​​framework to the user can be an operator whose input data or output data is the first tensor; the type of precision fault in the report can be a hardware fault; and the recovery method adopted in the report can be to switch the target operator to the CPU for recalculation, so that the target operator outputs a new tensor.

[0187] The methods provided by the embodiments of this application have been described in detail above with reference to Figures 1 to 5. The embodiments of the apparatus of this application will be described in detail below with reference to Figures 6 and 7. It should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments; therefore, any parts not described in detail can be referred to the preceding method embodiments.

[0188] Figure 6 is a schematic block diagram of a fault detection and recovery device 600 provided in an embodiment of this application. The device 600 can be implemented by software, hardware, or a combination of both. The device 600 provided in this embodiment can implement the method flow shown in Figure 1 of this embodiment. The device 600 includes: an acquisition module 610, a detection module 620, a determination module 630, and a recovery module 640.

[0189] It should be understood that the acquisition module 610, the detection module 620, and the determination module 630 can correspond to the fault location unit 220 mentioned above; the recovery module 640 can correspond to the fault recovery unit 240 mentioned above.

[0190] For example, the acquisition module 610 is used to acquire a first tensor to be detected; the detection module 620 is used to perform precision fault detection on the first tensor to obtain a precision fault detection result for the first tensor; the determination module 630 is used to determine a target operator related to the first tensor when the precision fault detection result indicates that the first tensor has a precision fault, wherein the input data or output data of the target operator is the first tensor, and the target operator is an operator in an artificial intelligence (AI) model; the recovery module 640 is used to recover the precision fault existing in the first tensor based on the target operator.

[0191] Optionally, the detection module 620 is specifically used to: determine that the first tensor has a precision fault if there are outliers and / or irregularities in the elements contained in the first tensor.

[0192] Optionally, the detection module 620 is specifically used to: detect whether there are outliers and / or isolated values ​​among the elements contained in the first tensor.

[0193] Optionally, the recovery module 640 is specifically used to: adjust the first tensor according to the data configuration of the target operator when the first tensor does not meet the data configuration of the target operator, so as to obtain a second tensor.

[0194] Optionally, the acquisition module 610 is further configured to acquire the first tensor from the first register and the second tensor from the second register; the determination module 630 is further configured to compare the first tensor and the second tensor to determine whether the second tensor has the precision fault.

[0195] Optionally, the recovery module 640 is specifically used to: improve the accuracy of the first tensor by improving the accuracy of the target operator, and obtain the third tensor.

[0196] Optionally, the recovery module 640 is specifically used to: improve the accuracy of the second tensor by improving the accuracy of the target operator in the event of the accuracy fault in the second tensor, thereby obtaining a third tensor.

[0197] Optionally, the first tensor is the output data of the target operator, and the recovery module 640 is specifically used for: the target operator outputs a fourth tensor by calling a target function, which is used to achieve the continuity of the input data of the target operator in memory.

[0198] Optionally, the first tensor is the output data of the target operator, and the recovery module 640 is specifically used to: in the case that the third tensor has the precision fault, the target operator outputs the fourth tensor by calling the target function.

[0199] Optionally, the first tensor is the output data of the target operator, and the recovery module 640 is specifically used to: switch to the CPU to execute the target operator, and the target operator outputs the fifth tensor.

[0200] Optionally, the first tensor is the output data of the target operator, and the recovery module 640 is specifically used to: switch to the CPU to execute the target operator in the case that the fourth tensor has the precision fault, and the target operator outputs the fifth tensor.

[0201] Optionally, the device 600 further includes: a sampling module, used to downsample the first tensor before performing accuracy fault detection on the first tensor to obtain a sampled first tensor; the detection module is specifically used to: perform accuracy fault detection on the sampled first tensor and determine the accuracy fault detection result of the first tensor.

[0202] Optionally, the sampling module is specifically used to: perform bi-diagonal sampling on the first tensor to obtain the sampled first tensor.

[0203] The device 600 here can be embodied in the form of a functional module. The term "module" here can be implemented in software and / or hardware, without specific limitations.

[0204] For example, a "module" can be a software program, a hardware circuit, or a combination of both that implements the above functions. For instance, the implementation of module 610 will be described below using module 610 as an example. Similarly, the implementation of other modules, such as detection module 620, determination module 630, recovery module 640, and sampling module, can refer to the implementation of module 610.

[0205] As an example of a software functional unit, the acquisition module 610 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the acquisition module 610 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0206] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0207] As an example of a hardware functional unit, the acquisition module 610 may include at least one computing device, such as a server. Alternatively, the acquisition module 610 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0208] The multiple computing devices included in the acquisition module 610 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the acquisition module 610 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the acquisition module 610 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0209] Therefore, the modules of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0210] It should be noted that the device provided in the above embodiments is only illustrated by the division of the functional modules described above when executing the above methods. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. For example, the acquisition module 610 can be used to execute any step in the above methods, the detection module 620 can be used to execute any step in the above methods, the determination module 630 can be used to execute any step in the above methods, the recovery module 640 can be used to execute any step in the above methods, and the sampling module can be used to execute any step in the above methods. The steps implemented by the acquisition module 610, detection module 620, determination module 630, recovery module 640, and sampling module can be specified as needed. By implementing different steps in the above methods through the acquisition module 610, detection module 620, determination module 630, recovery module 640, and sampling module, all the functions of the above device can be realized.

[0211] Furthermore, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments above. Their beneficial effects are the same as those of the method embodiments. For details, please refer to the relevant descriptions of the above method embodiments, which will not be repeated here.

[0212] The method provided in this application can be executed by a computing device, which can also be referred to as a computer system. It includes a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as processing units, memory, and memory control units; the functions and structure of this hardware will be described in detail later. The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software. Optionally, the computer system can be a handheld device such as a smartphone, or a terminal device such as a personal computer; this application does not particularly limit this, as long as the method provided in this application can be used. The executing entity of the method provided in this application can be a computing device, or a functional module within the computing device capable of calling and executing programs.

[0213] The following describes in detail, with reference to Figure 7, a computing device provided in an embodiment of this application.

[0214] Figure 7 is a schematic diagram of the architecture of a computing device 1500 provided in an embodiment of this application. The computing device 1500 can be a server, a computer, or other device with computing capabilities. The computing device 1500 shown in Figure 7 includes at least one processor 1510 and a memory 1520.

[0215] It should be understood that this application does not limit the number of processors and memories in the computing device 1500.

[0216] The processor 1510 executes instructions in the memory 1520, causing the computing device 1500 to implement the method provided in this application. Alternatively, the processor 1510 executes instructions in the memory 1520, causing the computing device 1500 to implement the various functional modules provided in this application, thereby implementing the method provided in this application.

[0217] Optionally, the computing device 1500 also includes a communication interface 1530. The communication interface 1530 uses a transceiver module, such as, but not limited to, a network interface card or a transceiver, to enable communication between the computing device 1500 and other devices or communication networks.

[0218] Optionally, the computing device 1500 also includes a system bus 1540, wherein the processor 1510, memory 1520, and communication interface 1530 are respectively connected to the system bus 1540. The processor 1510 can access the memory 1520 through the system bus 1540; for example, the processor 1510 can perform data read / write or code execution in the memory 1520 through the system bus 1540. The system bus 1540 is a peripheral component interconnect express (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus 1540 is divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in Figure 7, but this does not mean that there is only one bus or one type of bus.

[0219] In one possible implementation, the processor 1510 primarily functions to interpret the instructions (or code) of a computer program and process data within the computer software. The instructions of the computer program and the data within the computer software can be stored in memory 1520 or cache 1516.

[0220] Optionally, processor 1510 may be an integrated circuit chip with signal processing capabilities. By way of example and not limitation, processor 1510 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Among these, a general-purpose processor is a microprocessor, etc. For example, processor 1510 may be a central processing unit (CPU).

[0221] Optionally, each processor 1510 includes at least one processing unit 1512 and a memory control unit 1514.

[0222] Optionally, the processing unit 1512, also known as the core, is the most important component of the processor. The processing unit 1512 is manufactured from single-crystal silicon using a specific production process. All calculations, command reception, command storage, and data processing are performed by the core. Each processing unit independently executes program instructions, utilizing parallel computing capabilities to accelerate program execution. Various processing units have fixed logical structures; for example, a processing unit includes logical units such as a Level 1 cache, a Level 2 cache, an execution unit, an instruction-level unit, and a bus interface.

[0223] In one implementation example, the memory control unit 1514 controls the data interaction between the memory 1520 and the processing unit 1512. Specifically, the memory control unit 1514 receives memory access requests from the processing unit 1512 and controls access to memory based on the memory access requests. By way of example and not limitation, the memory control unit is a device such as a memory management unit (MMU).

[0224] In one implementation example, each memory control unit 1514 addresses the memory 1520 via the system bus. An arbitrator (not shown in Figure 7) is configured on the system bus to handle and coordinate contention for access by the multiple processing units 1512.

[0225] In one implementation example, the processing unit 1512 and the memory control unit 1514 are connected via internal chip connection lines, such as address lines, thereby enabling communication between the processing unit 1512 and the memory control unit 1514.

[0226] Optionally, each processor 1510 also includes a cache 1516, which is a buffer for data exchange (called a cache). When the processing unit 1512 needs to read data, it first looks for the required data in the cache. If the data is found, it is executed directly; otherwise, it looks for the data in memory. Since the cache operates much faster than memory, its purpose is to help the processing unit 1512 run faster.

[0227] The memory 1520 provides runtime space for processes in the computing device 1500. For example, the memory 1520 stores the computer program (specifically, the program code) used to generate the process. After the computer program is run by the processor to generate a process, the processor allocates corresponding storage space for the process in the memory 1520. Furthermore, the aforementioned storage space further includes text segments, initialized data segments, bit initialized data segments, stack segments, heap segments, etc. The memory 1520 stores data generated during the process's execution, such as intermediate data or process data, in the aforementioned process-specific storage space.

[0228] Optionally, the memory, also known as RAM, is used to temporarily store the data processed by the processor 1510, as well as data exchanged with external storage devices such as hard disks. As long as the computer is running, the processor 1510 will load the data that needs to be processed into RAM for processing, and after the processing is completed, the processing unit 1512 will send the result out.

[0229] By way of example and not limitation, memory 1520 is volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory is read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory is random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory 1520 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0230] The structure of the computing device 1500 listed above is merely illustrative and is not limited thereto. The computing device 1500 in this application includes various hardware components in existing computer systems. For example, the computing device 1500 also includes other memories besides memory 1520, such as disk storage. Those skilled in the art should understand that the computing device 1500 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the computing device 1500 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the computing device 1500 may only include the devices necessary for implementing the embodiments of this application, and not necessarily all the devices shown in FIG. 7.

[0231] In this embodiment, a computer program product containing instructions is also provided. The computer program product may be a software or program product containing instructions capable of running on a computing device or stored on any usable medium. When run on a computing device, it causes the computing device to perform the methods provided above, or causes the computing device to perform the functions of the apparatus provided above.

[0232] In this embodiment, a computer-readable storage medium is also provided. This computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that, when executed on a computing device, cause the computing device to perform the method described above.

[0233] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0234] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0235] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0236] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0237] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0238] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0239] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0240] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for fault detection and recovery, characterized in that, The method includes: Obtain the first tensor to be inspected; Perform precision fault detection on the first tensor to obtain the precision fault detection result of the first tensor; When the accuracy fault detection result indicates that the first tensor has an accuracy fault, a target operator related to the first tensor is determined, wherein the input data or output data of the target operator is the first tensor, and the target operator is an operator in an artificial intelligence (AI) model. The precision fault existing in the first tensor is recovered based on the target operator.

2. The method according to claim 1, characterized in that, Perform precision fault detection on the first tensor and determine the precision fault detection result of the first tensor, including: If outliers and / or isolated values ​​exist in the elements contained in the first tensor, it is determined that the first tensor has the precision fault.

3. The method according to claim 1 or 2, characterized in that, The recovery of the precision fault in the first tensor based on the target operator includes: If the first tensor does not meet the data configuration of the target operator, the first tensor is adjusted according to the data configuration of the target operator to obtain the second tensor.

4. The method according to claim 3, characterized in that, The method further includes: Retrieve the first tensor from the first register; Retrieve the second tensor from the second register; The first tensor and the second tensor are compared to determine whether the second tensor has the aforementioned precision fault.

5. The method according to claim 1 or 2, characterized in that, The recovery of the precision fault in the first tensor based on the target operator includes: The precision of the first tensor is improved by increasing the precision of the target operator, thereby obtaining the third tensor.

6. The method according to claim 3 or 4, characterized in that, The recovery of the precision fault in the first tensor based on the target operator includes: In the case of the accuracy fault in the second tensor, the accuracy of the second tensor is improved by improving the accuracy of the target operator, thereby obtaining a third tensor.

7. The method according to claim 1 or 2, characterized in that, The first tensor is the output data of the target operator. The recovery of the precision fault in the first tensor based on the target operator includes: The target operator outputs a fourth tensor by calling a target function, which is used to ensure the continuity of the input data of the target operator in memory.

8. The method according to claim 5 or 6, characterized in that, The first tensor is the output data of the target operator. The recovery of the precision fault in the first tensor based on the target operator includes: In the event of the precision fault in the third tensor, the target operator outputs the fourth tensor by calling the target function.

9. The method according to claim 1 or 2, characterized in that, The first tensor is the output data of the target operator. The recovery of the precision fault in the first tensor based on the target operator includes: Switch to the CPU to execute the target operator, which outputs the fifth tensor.

10. The method according to claim 7 or 8, characterized in that, The recovery of the precision fault in the first tensor based on the target operator includes: If the fourth tensor has the aforementioned precision fault, the CPU switches to execute the target operator, which outputs the fifth tensor.

11. The method according to any one of claims 1 to 10, characterized in that, Before performing precision fault detection on the first tensor, the method further includes: The first tensor is downsampled to obtain the sampled first tensor; The precision fault detection of the first tensor includes: Precision fault detection is performed on the first tensor after sampling, and the precision fault detection result of the first tensor is determined.

12. A fault detection and recovery device, characterized in that, The device includes: The acquisition module is used to acquire the first tensor to be detected; The detection module is used to perform precision fault detection on the first tensor to obtain the precision fault detection result of the first tensor; The determination module is used to determine a target operator related to the first tensor when the accuracy fault detection result indicates that the first tensor has an accuracy fault, wherein the input data or output data of the target operator is the first tensor, and the target operator is an operator in an artificial intelligence (AI) model. The recovery module is used to recover the precision fault existing in the first tensor based on the target operator.

13. The apparatus according to claim 12, characterized in that, The detection module is specifically used for: If outliers and / or isolated values ​​exist in the elements contained in the first tensor, it is determined that the first tensor has the precision fault.

14. The apparatus according to claim 12 or 13, characterized in that, The recovery module is specifically used for: If the first tensor does not meet the data configuration of the target operator, the first tensor is adjusted according to the data configuration of the target operator to obtain the second tensor.

15. The apparatus according to claim 14, characterized in that, The acquisition module is further configured to acquire the first tensor from the first register; The acquisition module is further configured to acquire the second tensor from the second register; The determining module is further configured to compare the first tensor and the second tensor to determine whether the second tensor has the precision fault.

16. The apparatus according to claim 12 or 13, characterized in that, The recovery module is specifically used for: The precision of the first tensor is improved by increasing the precision of the target operator, thereby obtaining the third tensor.

17. The apparatus according to claim 14 or 15, characterized in that, The recovery module is specifically used for: In the case of the accuracy fault in the second tensor, the accuracy of the second tensor is improved by increasing the accuracy of the target operator, and a third tensor is obtained.

18. The apparatus according to claim 12 or 13, characterized in that, The first tensor is the output data of the target operator. The recovery module is specifically used for: The target operator outputs a fourth tensor by calling a target function, which is used to ensure the continuity of the input data of the target operator in memory.

19. The apparatus according to claim 16 or 17, characterized in that, The first tensor is the output data of the target operator. The recovery module is specifically used for: In the event of the precision fault in the third tensor, the target operator outputs the fourth tensor by calling the target function.

20. The apparatus according to claim 12 or 13, characterized in that, The first tensor is the output data of the target operator. The recovery module is specifically used for: Switch to the CPU to execute the target operator, which outputs the fifth tensor.

21. The apparatus according to claim 18 or 19, characterized in that, The first tensor is the output data of the target operator. The recovery module is specifically used for: If the fourth tensor has the aforementioned precision fault, the CPU switches to execute the target operator, which outputs the fifth tensor.

22. The apparatus according to any one of claims 12 to 21, characterized in that, The device further includes: The sampling module is used to downsample the first tensor before performing precision fault detection on the first tensor to obtain the sampled first tensor; The detection module is specifically used for: Precision fault detection is performed on the first tensor after sampling, and the precision fault detection result of the first tensor is determined.

23. A computing device, characterized in that, The computing device includes a processor and a memory, the processor being configured to execute instructions stored in the memory to cause the computing device to perform the method as described in any one of claims 1 to 11.

24. A computer program product containing instructions, characterized in that, When the instructions are executed by the computing device, the computing device performs the method as described in any one of claims 1 to 11.

Images