Data extraction method and computer storage medium

Abstract

A data extraction method, the data being first object data of a specific type, and first byte stream data being obtained by means of serializing the first object data. The method comprises: acquiring a specific deserialization scheme associated with the first byte stream data; and on the basis of the specific deserialization scheme, deserializing the first byte stream data so as to extract the first object data.

Description

Data extraction methods and computer storage media Cross-reference to related applications

[0001] This application claims priority to application number 2024118340622, filed on December 12, 2024, entitled "Method for extracting data and computer storage medium" and application number 2024118340872, filed on December 12, 2024, entitled "Method, interface and medium for determining anomaly operators in a ported neural network model". Technical Field

[0002] This disclosure relates to the field of data processing technology. More specifically, this disclosure relates to a method for extracting data and a computer storage medium. Background Technology

[0003] In the field of data processing technology, serialization is the process of converting object data into byte stream data and using it for storage or transmission; deserialization is the process of converting byte stream data into corresponding object data.

[0004] Generally speaking, object data consists of one or more types of data arranged in complex data structures. Since the specific data structure of the object data is unknown before searching, and the data structure of the object data is often quite complex, searching for specific types of data in the object data requires complex traversal, which is tedious and time-consuming.

[0005] In view of this, there is an urgent need to provide a method for extracting data in order to simplify the process of finding specific types of data from object data and improve the efficiency of data extraction.

[0006] In the field of machine learning, a neural network model trained on one device is usually transferred to another device, and then trained again according to the requirements of the new task on the other device.

[0007] However, since the specific logic of operators is different on different devices, when training or running the neural network model on another device, some operators in the neural network model may exhibit abnormal precision, resulting in the neural network model's inference accuracy being substandard or unable to be trained.

[0008] To pinpoint which operator(s) in a neural network model exhibits anomalies in precision, the model is typically run on both the original and current devices. Next, the input and output data for each operator are acquired. Then, the input and output data for each operator on the different devices are compared, and the operator exhibiting the anomaly is identified based on the comparison results. This method requires transferring the input and output data for each operator in the neural network model, resulting in a large storage space requirement. Furthermore, the need to compare the input and output data for each operator leads to a significant time commitment in identifying the operator exhibiting anomalies, severely impacting the efficiency of locating such operators.

[0009] In view of this, there is an urgent need to provide a scheme for identifying anomalous operators in a transplanted neural network model, so as to improve the efficiency of locating anomalous operators in the neural network model. Summary of the Invention

[0010] In order to at least address one or more of the technical problems mentioned above, this disclosure provides methods for extracting data and computer-readable storage media in several aspects.

[0011] In a first aspect, this disclosure provides a method for extracting data, wherein the data is first object data of a specific type, and the first byte stream data is obtained by serializing the first object data. The method includes: obtaining a specific deserialization method associated with the first byte stream data; and performing deserialization processing on the first byte stream data according to the specific deserialization method to extract the first object data.

[0012] In a second aspect, this disclosure provides a computer storage medium, wherein a computer program is stored on the computer storage medium, and the program, when executed by a processor, implements the method described in the first aspect above.

[0013] The data extraction method described above first obtains the specific deserialization method associated with the first byte stream data. Then, based on this specific deserialization method, the first byte stream data is deserialized to extract the first object data. This allows for direct extraction of specific type first object data after specific deserialization of the first byte stream data. Thus, without complex traversal, specific type first object data can be quickly extracted from complex object data, simplifying the extraction process and improving efficiency. 2024118340622

[0014] In order to at least address one or more of the technical problems mentioned above, this disclosure proposes a method, interface, and medium for determining anomalous operators in a ported neural network model in several aspects.

[0015] In a first aspect, this disclosure provides a method for determining anomalous operators in a ported neural network model, comprising:

[0016] The sampling parameters are set, including the granularity of sampling the input and output data of the operators in the neural network model; based on the sampling parameters, at least a portion of the input and / or output data of the operators of the neural network model are obtained when running on a first device to obtain a first result; and the at least a portion of the input and / or output data of the operators of the neural network model are obtained when running on a second device to obtain a second result, wherein the input data of the neural network model is the same when running on the first device and the second device respectively; the first result and the second result are compared correspondingly to determine the comparison difference between the first result and the second result; and based on whether the comparison difference reaches a threshold, abnormal operators in the neural network model are determined.

[0017] In a second aspect, this disclosure provides an application programming interface, including: a parameter setting component for setting sampling parameters, wherein the sampling parameters include the granularity of sampling input and output data of operators in a neural network;

[0018] A first sampling component is configured to acquire, based on the sampling parameters, input and / or output data of at least a portion of the operators of the neural network model running on a first device to obtain a first result; and a second sampling component is configured to acquire, based on the input and / or output data of the at least a portion of the operators of the neural network model running on a second device to obtain a second result, wherein the input data of the neural network model is the same when the neural network model is run on the first device and the second device respectively; a comparison component is configured to compare the first result and the second result accordingly to determine the comparison difference between the first result and the second result; and an anomaly operator determination component is configured to determine anomaly operators in the neural network model based on whether the comparison difference reaches a threshold.

[0019] In a third aspect, this disclosure provides a machine-readable medium having an application programming interface (API) call stored thereon, which, if executed by one or more processors, causes the one or more processors to perform the following operations: setting sampling parameters, wherein the sampling parameters include the granularity of sampling input and output data of operators in a neural network model; acquiring, based on the sampling parameters, input and / or output data of at least a portion of the operators of the neural network model when running on a first device to obtain a first result; and acquiring, based on the sampling parameters, input and / or output data of the at least a portion of the operators of the neural network model when running on a second device to obtain a second result, wherein the input data of the neural network model is the same when the neural network model is run on the first device and the second device respectively;

[0020] The first result and the second result are compared accordingly to determine the difference between the first result and the second result; and based on whether the difference reaches a threshold, the abnormal operator in the neural network model is determined.

[0021] The method for determining anomalous operators in a ported neural network model, as provided above, is used to migrate a neural network model running from a first device to a second device. The method includes: first, setting sampling parameters; then, acquiring input and / or output data of some operators of the neural network model running on the first device, based on the sampling parameters, to obtain a first result; and acquiring input and / or output data of the aforementioned partial operators of the neural network model running on the second device, to obtain a second result; then, comparing the first result and the second result accordingly to determine the difference between them; and determining anomalous operators in the neural network model based on whether the difference reaches a threshold. This eliminates the need to compare the input and output data of all operators in the neural network model, but only compares the input and / or output data of some operators running on different devices. Based on the comparison results, anomalous operators in the neural network model can be determined, thereby effectively shortening the time for determining anomalous operators and improving the efficiency of anomalous operator determination. 2024118340872 Attached Figure Description

[0022] The above and other objects, features, and advantages of exemplary embodiments of this disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding portions wherein:

[0023] Figure 1-1 shows an exemplary flowchart of a data extraction method according to an embodiment of this disclosure;

[0024] Figures 1-2 illustrate a schematic diagram of serializing first object data and a specific deserialization method in one embodiment of this disclosure;

[0025] Figures 1-3 illustrate a schematic diagram of extracting a specific type of data according to an embodiment of this disclosure;

[0026] Figures 1-4 show exemplary flowcharts of another method for extracting data according to one embodiment of this disclosure;

[0027] Figures 1-5 show exemplary flowcharts of a method for aggregating third object data and fourth object data on a single device according to an embodiment of this disclosure;

[0028] Figures 1-6 show exemplary flowcharts of a method for determining anomaly operators in a neural network according to an embodiment of this disclosure;

[0029] Figure 2-1 illustrates an exemplary scenario in which embodiments of this disclosure can be applied;

[0030] Figure 2-2 is a schematic diagram of existing schemes for determining anomaly operators in neural network models;

[0031] Figure 2-3 shows an exemplary flowchart 2-300 of a method for determining anomalous operators in a ported neural network model according to some embodiments of this disclosure;

[0032] Figure 2-4 shows an exemplary flowchart 2-320 of a method for obtaining operator intervals according to an embodiment of this disclosure;

[0033] Figure 2-5a shows a schematic diagram of a partitioned neural network model according to an embodiment of this disclosure;

[0034] Figure 2-5b shows a schematic diagram of a partitioned neural network model for another embodiment of this disclosure;

[0035] Figure 2-6 shows an exemplary flowchart 2-321 of a method for obtaining operator intervals according to another embodiment of this disclosure;

[0036] Figure 2-6a shows a schematic diagram of a partitioned neural network model according to an embodiment of this disclosure;

[0037] Figure 2-6b shows a schematic diagram of a partitioned neural network model for another embodiment of this disclosure;

[0038] Figure 2-6c shows a schematic diagram of a partitioned neural network model for yet another embodiment of this disclosure;

[0039] Figures 2-7 show schematic diagrams of a block diagram of an application programming interface provided by some embodiments of this disclosure. Detailed Implementation

[0040] 2024118340622 The technical solutions of the embodiments disclosed herein will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0041] It should be understood that the terms "first," "second," "third," and "fourth," etc., in the claims, specification, and drawings of this disclosure are used to distinguish different objects, not to describe a specific order. "First," "second," "third," and "fourth," etc., also do not necessarily mean only one, but can also mean multiple. The terms "comprising" and "including" used in the specification and claims of this first disclosure indicate the presence of the described feature, integral, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or sets thereof.

[0042] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0043] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0044] The specific embodiments disclosed herein will now be described in detail with reference to the accompanying drawings.

[0045] Exemplary application scenarios

[0046] In the field of deep learning technology, if a neural network model is trained on one device and then transferred to another device for training or use, some operators in the neural network model often malfunction, resulting in the neural network model failing to complete training or experiencing a decrease in inference accuracy.

[0047] The "anomaly" referred to here means that during the training or inference phases, the predicted value obtained by the operator deviates significantly from the actual value, resulting in excessive computational error. For example, because different devices may implement the operator logic differently, if a neural network model trained on one device is migrated to another and run on that device, it may cause some operators to obtain predicted values ​​that differ significantly from the actual values, i.e., the operator has malfunctioned.

[0048] To determine which operator(s) in a neural network model are malfunctioning, the following steps can be taken: First, serialize and save the input and / or output data of each operator in the neural network model as corresponding byte streams when running on the two devices mentioned above. This input and / or output data consists of one or more Tensor type objects and other types of object data arranged in complex data structures; for example, the input and / or output data are often complex Python objects. Next, transmit and aggregate the serialized byte streams obtained from different devices to compare the Tensor type objects in the input and / or output data corresponding to different devices. Then, determine the malfunctioning operator in the neural network model based on the comparison results of the Tensor type objects.

[0049] When comparing Tensor type object data across different devices, the byte stream data must first be deserialized to recover the input and / or output data of the neural network model running on different devices. Then, Tensor type object data must be identified from the recovered input and / or output data. In this process, the search for Tensor type objects must begin after the unserialized data has been recovered. Furthermore, since the data structure of the input and / or output data is unknown beforehand, and this structure is often complex, typically with multiple nested layers, a complex traversal of the input and / or output data is required to extract all Tensor type object data. This makes the extraction of specific data types complex and time-consuming.

[0050] For example, when the input and / or output data is [A, {B1: T1, B2: T2}], the data structure consists of a list storing a basic data A and a nested dictionary {B1: T1, B2: T2}, where T1 and T2 are objects of type Tensor, and A, B1, and B2 are objects of other types. Therefore, without knowing the specific data structure, complex traversal is required to extract the Tensor objects T1 and T2.

[0051] Exemplary application scheme

[0052] In view of this, the present disclosure provides a data extraction scheme. In this scheme, a specific deserialization method associated with the first byte stream data is first obtained; then, the first byte stream data is deserialized according to the specific deserialization method to extract the first object data. In this way, after performing specific deserialization processing on the first byte stream data, the first object data of a specific type can be directly extracted, thus enabling rapid extraction of the first object data of a specific type without the need for complex traversal. This process simplifies the extraction of specific type data and improves the efficiency of extracting specific type data. The specific embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings.

[0053] Figure 1-1 shows an exemplary flowchart 1-100 of a data extraction method according to an embodiment of this disclosure.

[0054] As shown in Figure 1-1, this disclosure provides a method for extracting data, wherein the data is a first object data of a specific type, and the first byte stream data is obtained by serializing the first object data. The method includes: in step S1-110, obtaining a specific deserialization method associated with the first byte stream data; and in step S1-120, performing deserialization processing on the first byte stream data according to the specific deserialization method to extract the first object data.

[0055] In the embodiments disclosed herein, object data refers to structured data. The term "structured data" refers to data with a certain data structure. This object data is composed of one or more data items arranged according to a specific data structure, and it is stored in storage spaces such as memory, files, or databases in the form of a corresponding storage structure. For example, in Python, a list can store multiple consecutive data items; data using a list as its data structure can be considered object data. In this embodiment, different types of object data exist. A specific type of object data is referred to as "first object data." "Specific type" refers to a designated data type among all data types. Object data specified with a particular data type is called "first object data." For example, in Python, a dictionary can store objects of any type. A dictionary stores one or more key-value pairs; data stored using a dictionary as its data structure can be considered first object data.

[0056] The byte stream data mentioned above refers to data stored in binary digital form. During serialization, the object data described above is converted into corresponding byte stream data. In this disclosure, "first byte stream data" is obtained after serialization of the first object data.

[0057] In some embodiments, the first object data of the specific type mentioned above is object data of the tensor type, where the specific type refers to the tensor type.

[0058] In other embodiments, the first object data of a specific type includes, but is not limited to, custom structures, data of custom data types, and data of any existing data type. Those skilled in the art will understand that there exists complex object data that includes object data of any data type, and the solution provided in this disclosure embodiment can extract object data of any data type from the aforementioned complex object data.

[0059] The "specific deserialization method associated with the first byte stream data" in step S1-110 is a method for performing specific deserialization processing on the first byte stream data.

[0060] In some embodiments, the function of a specific deserialization method is to perform specific deserialization processing on the first byte stream data to obtain the first object data and save the first object data.

[0061] As described above, in this embodiment, the first object data is object data with a certain data structure. This first object data may include one or more sub-data. During the specific deserialization process of the first byte stream data according to the aforementioned specific deserialization method, the first object data is obtained directly, rather than first obtaining one or more sub-data from the first object data and then combining the one or more sub-data into the first object data according to the corresponding data structure.

[0062] For example, when the first object data is a tensor, the first object data can be a multidimensional array, and the one or more sub-data included in the multidimensional array are one or more integer data. In the above-mentioned specific deserialization process, the first object data is the basic unit extracted by the above-mentioned specific deserialization method. That is, in the process of performing specific deserialization processing on the first byte stream data according to the above-mentioned specific deserialization method, one or more integer data will not be obtained first, and then the one or more integer data will be composed into a tensor according to the corresponding multidimensional data. Instead, the tensor, i.e., the first object data, will be obtained directly.

[0063] In some embodiments, the second byte stream data is obtained by serializing it using a specific deserialization method. In steps S1-110, obtaining the specific deserialization method associated with the first byte stream data includes: performing deserialization processing on the second byte stream data to obtain the specific deserialization method.

[0064] In the embodiments of this application, a specific deserialization method is pre-serialized to obtain byte stream data corresponding to the specific deserialization method. In order to distinguish it from the first byte stream data mentioned above, the "byte stream data corresponding to the specific deserialization method" is referred to as the "second byte stream data".

[0065] In some embodiments, the first object data is a portion of the fifth object data, wherein the fifth byte stream data is obtained by serializing the fifth object data.

[0066] According to one embodiment of this disclosure, the fifth object data is the input and / or output data of at least some operators in the neural network model when running on the first device and / or the second device. As can be seen from the description of the above embodiments, the fifth object data may include the first object data and non-first object data of a specific type, and the first object data and non-first object data are composed of the fifth object data according to a certain data structure.

[0067] In this embodiment, the first object data can be part of another data. For ease of description, the "other data" mentioned here is referred to as the "fifth object data," meaning that the first object data can be a portion of the fifth object data. The aforementioned "fifth byte stream data" refers to the byte stream data obtained after serializing the fifth object data.

[0068] It should be noted that the fifth object data includes not only object data of a specific type, i.e., the first object data, but also object data of other types that are different from the specific type. Here, "object data of other types that are different from the specific type" is referred to as non-first object data, and the byte stream data obtained after serializing the non-first object data is referred to as "non-first byte stream data". Since the fifth object data includes both first and non-first object data, the fifth byte stream data includes both first and non-first byte stream data.

[0069] In some embodiments, the process of deserializing the first byte stream data according to the specific deserialization method in steps S1-120 to extract the first object data specifically includes:

[0070] In response to the first byte stream data being the target during the deserialization process of the fifth byte stream data, the first byte stream data is deserialized using the specific deserialization method to extract the first object data.

[0071] Specifically, after deserializing the first byte stream data to extract the first object data, the extracted first object data also needs to be saved in a predetermined order.

[0072] If it is necessary to extract the first object data from the fifth object data, then during the deserialization process of the fifth byte stream data, when it is necessary to deserialize the first byte stream data, the first byte stream data should be deserialized using the specific deserialization method described above to obtain the first object data, and the first object data should be saved. In this way, there is no need to perform complex traversal of the complete fifth object data, and the specific type of object data can be extracted from the fifth object data, thereby improving the extraction efficiency of specific type of object data.

[0073] In some embodiments, steps S1-120 further include: in response to the target being non-first byte stream data during the deserialization process of the fifth byte stream data, performing deserialization processing on the non-first byte stream data using a conventional deserialization method.

[0074] In this embodiment, during the deserialization of the fifth byte stream data, if it is necessary to deserialize non-first byte stream data, then the non-first byte stream data is deserialized using a conventional deserialization method. The "conventional deserialization method" mentioned above refers to a deserialization method commonly used in the art, and is different from the specific deserialization method described above.

[0075] For example, the first object data of a specific type is tensor type object data, and the non-first object data of a non-specific type is a dictionary composed of strings and integers. The aforementioned fifth object data is formed by multiple instances of the first object data and multiple instances of non-first object data in a multi-level nested structure. As described in the above embodiment, during the deserialization process of the fifth byte stream data, whenever deserialization of the first byte stream data is required, a specific deserialization process is performed on the first byte stream data according to the specific deserialization method associated with it, to extract the aforementioned tensor type object data, i.e., the first object data, and then save the first object data.

[0076] In some embodiments, the fifth object data includes third object data of at least some operators in the neural network model running on a first device, and fourth object data running on a second device. The third object data contains a specific type of first object data, and the fourth object data also contains a specific type of first object data. Furthermore, the first object data in the third object data and the first object data in the fourth object data can be the same or different. In the foregoing, the third object data refers to the input data and / or output data of at least some operators in the neural network model running on the first device. The first device can be a GPU (Graphics Processing Unit) or a non-GPU device. Non-GPU devices include, but are not limited to, at least one of an NPU (Neural Network Processing Unit) and a CPU (Central Processing Unit). The fourth object data refers to the input data and / or output data of the at least some operators in the same neural network model running on the second device. The second device can also be a GPU or a non-GPU device, but the second device is a different device from the first device. In this embodiment, the first device and the second device cannot both be GPUs, and the first device and the second device cannot both be non-GPU devices. For example, if the first device is a GPU, then the second device is a non-GPU device; if the first device is a non-GPU device, then the second device is a GPU.

[0077] To compare the first object data in the third object data and the first object data in the fourth object data, the third and fourth object data need to be serialized separately to obtain the third-byte stream data corresponding to the third object data and the fourth-byte stream data corresponding to the fourth object data. Then, the third-byte stream data and the fourth-byte stream data are transmitted and combined. In the above text, "third-byte stream data" refers to the data obtained after serializing the third object data, and "fourth-byte stream data" refers to the data obtained after serializing the fourth object data.

[0078] Here, "transfer aggregation" refers to combining the transfer of third-byte and fourth-byte stream data together; that is, combining the transfer of third-object data and fourth-object data in byte stream form. For example, the third-byte and fourth-byte stream data can be stored in the same file, on the same hard drive, or transferred to the same processing device.

[0079] In some embodiments, before serializing the third object data and the fourth object data respectively, it is necessary to improve the conventional deserialization methods in the art to obtain a specific deserialization method. The conventional deserialization method mentioned above refers to the method of deserializing byte stream data to obtain the original object data. A conventional deserialization method can be the deserialization method in the pickle module of the PyTorch framework, which is a deep learning framework.

[0080] Specifically, improving conventional deserialization methods in this field to obtain a specific deserialization method can include: first, intercepting the conventional deserialization method in Python's API (Application Programming Interface) before it runs, for example, before transferring the input and output data of the operator; then, adding a new processing logic after the processing logic of the conventional deserialization method, and encapsulating the conventional deserialization method and the newly added processing logic to obtain a new deserialization method, i.e., a specific deserialization method. In the above, the "newly added processing logic" refers to saving the specific type of object data recovered through the conventional deserialization method.

[0081] As described in the above embodiments, the process of deserializing the first byte stream data using this specific deserialization method specifically includes: deserializing the first byte stream data according to a conventional deserialization method to recover object data of a specific type, and saving the recovered object data of the specific type. In one embodiment disclosed herein, the location where the Tensor type object data is saved includes, but is not limited to, Python lists and files. However, it should be understood that the above-mentioned saving location is merely an example, and the location where the Tensor type object data is saved is not limited in any way.

[0082] Moreover, the difference between this specific deserialization method and the existing conventional deserialization method is that it saves the recovered object data of a specific type, thereby enabling the extraction of object data of a specific type while deserializing byte stream data.

[0083] Once a specific deserialization method is obtained, it can be used to replace the conventional deserialization method described above. That is, the processing logic corresponding to this specific deserialization method can be used to directly extract the first object data, i.e., the Tensor type object data, during the deserialization process of the complete third-byte stream data and / or fourth-byte stream data.

[0084] In existing technologies, to extract Tensor-type object data from the third and / or fourth object data mentioned above, it is necessary to first deserialize the third byte stream data corresponding to the third object data and / or the fourth byte stream data corresponding to the fourth object data to obtain the third and / or fourth object data. Then, complex traversal is performed on the third and / or fourth object data to extract the Tensor-type object data. However, with the method described in this application, complex traversal is not required. The first object data can be extracted from the third and / or fourth object data simultaneously with the deserialization of the third and / or fourth byte stream data. This simplifies the logic and process of extracting specific type object data from complex object data and improves the efficiency of extracting specific type object data.

[0085] Specifically, this particular deserialization method is a set of instructions, which defines the specific behavior and operations of the deserialization method. It should be noted that the set of instructions mentioned above can also be considered as "structured data," that is, data with a certain data structure, and this set of instructions will be stored in memory, files, or databases with a corresponding storage structure. In this application, for ease of description, the aforementioned specific deserialization method with a certain data structure is referred to as a "structured specific deserialization method."

[0086] Figure 1-2 illustrates a schematic diagram of serializing the first object data and a specific deserialization method of the structure in one embodiment of this disclosure. As shown in Figure 1-2, the first object data is data C of type {A1: [T3, T4, {B2: T5}]}. This data C is a dictionary, where the key is character data A1 and the value is a list storing Tensor data T3 and T4. Furthermore, this list also contains another dictionary {B2: T5}, where the key is integer data B2 and the value is Tensor data T5.

[0087] The specific deserialization method shown in Figure 1-2 is the structured specific deserialization method described above. In the embodiments disclosed herein, data C is serialized to obtain data c; and the structured specific deserialization method is then serialized to obtain a specific deserialization method in byte stream form.

[0088] In this embodiment, the serialized data of data C is data c, i.e., the first byte stream data, which is a byte stream data in binary digital form. Data c includes fields c1 and c2. Field c1 is the byte stream data corresponding to data A1 after serialization, and data c2 is the byte stream data corresponding to the list [T3, T4, {B2: T5}] after serialization. Data c also includes binary instructions and operators, where x represents any type of instruction and operator, which indicates how to deserialize data c.

[0089] In Figure 1-2, data c2 includes not only instructions and operators, but also fields t3, t4, and d1. Field t3 represents the byte stream data corresponding to data T3 after serialization, field t4 represents the byte stream data corresponding to data T4 after serialization, and field d1 is the byte stream data corresponding to dictionary {B2:T5} after serialization.

[0090] In addition to instructions and operators, field d1 also includes fields b2 and t5. Field b2 represents the byte stream data corresponding to data B2 after serialization, and field t5 represents the byte stream data corresponding to data T5 after serialization.

[0091] It should be noted that, in the technical solution of this disclosed embodiment, the order in which different object data in data C are serialized can be set by the user.

[0092] In this embodiment, data c and a specific deserialization method in byte stream form can be stored simultaneously on a hard disk or other file. Alternatively, data c and the specific deserialization method in byte stream form can be stored in different locations.

[0093] As described in the above embodiments, both the first object data and the structured specific deserialization method are serialized to convert them into byte stream data that is easy to save and transmit. It should be noted that the structured specific deserialization method also needs to be serialized, that is, converted into a corresponding byte stream format. This is so that when any device needs to extract the first object data, the specific deserialization method can be easily sent to the corresponding device, allowing that device to obtain the specific deserialization method associated with the first byte stream data. The structured specific deserialization method can be serialized into byte stream data along with the object data to be transferred, and then stored. The byte stream data corresponding to the structured specific deserialization method can be stored in the same location as the byte stream data corresponding to the object data, or it can be stored in a different location.

[0094] It should be noted that, in order to determine which operator(s) or specific operators malfunctioned during the neural network model transfer process, during the serialization of the input and / or output data of the operators in the neural network model, whenever the data requiring serialization includes Tensor type object data, a specific structured deserialization method is used to obtain a specific deserialization method in byte stream form. Furthermore, the Tensor type object data is also serialized to obtain the corresponding first byte stream data, and the first byte stream data and the specific deserialization method in byte stream form are stored in hard disk or other storage space. In addition, after the input and / or output data of an operator is transferred, the specific deserialization method is restored to the conventional deserialization method described above to avoid affecting the logic of deserializing subsequent byte stream data. In this application, for ease of description, the "specific deserialization method in byte stream form" obtained through serialization processing above is referred to as "second byte stream data".

[0095] Figure 1-3 illustrates a schematic diagram of extracting a specific type of data according to an embodiment of this disclosure. As shown in Figure 1-3, the byte stream data is data c obtained through serialization processing. This data c is the same as the data c in Figure 1-2, both obtained by serializing data C. Further details regarding data C and data c will not be provided here.

[0096] Specifically, one embodiment of this disclosure describes a process for extracting specific types of data, which includes: first, performing deserialization processing on a specific deserialization method in the form of a byte stream to obtain a structured specific deserialization method associated with the specific type of data in the form of a byte stream; then, performing specific deserialization processing on the specific type of data in the form of a byte stream in data c according to the structured specific deserialization method to extract Tensor type data from data C.

[0097] The specific process described above for performing specific deserialization processing on specific types of data in the byte stream form of data c to extract Tensor type data from data c can include the following steps, based on a specific deserialization method of the structured data.

[0098] First, deserialization is performed starting with the basic units of data in the corresponding data structure of data c. That is, the innermost data in the nested structure of data c is deserialized first. Field b2 is deserialized using a conventional deserialization method to obtain data B2. Field t5 is deserialized using a specific deserialization method to obtain data T5 of type Tensor, and this data T5 is stored in a list. After obtaining data B2 and data T5, data B2 and data T5 are combined into a dictionary {B2:T5} according to the corresponding instructions and characters.

[0099] Next, the byte stream data corresponding to the outermost structure in the dictionary {B2:T5} is deserialized. Specifically, fields t3 and t4 are deserialized using a specific deserialization method to obtain Tensor type data T3 and data T4, which are then stored in a list. Then, according to the corresponding instructions and characters, the data T3, data T4, and the dictionary {B2:T5} obtained in the previous step are combined into a list [T3, T4, {B2:T5}.

[0100] Finally, after deserializing field c1, data A1 is obtained. Based on the corresponding instructions and characters, data A1 and the list [T3, T4, {B2: T5}] above are combined into a dictionary {A1: [T3, T4, {B2: T5}]}, which is data C above.

[0101] It should be noted that the above deserialization process for data c also includes moving or processing the obtained data according to the instructions and characters in data c to obtain the complete data C.

[0102] In some embodiments, deserializing the first byte stream data to extract the first object data includes: deserializing the first byte stream data to obtain the first object data and saving the first object data to a list in a predetermined order.

[0103] In the embodiment described above, after deserializing the third-byte stream data and the fourth-byte stream data to obtain the corresponding first object data, the first object data in the third object data and the fourth object data can be saved to the corresponding lists in the same preset order. This facilitates the comparison of the first object data in the third object data and the first object data in the fourth object data. The corresponding first object data from different devices can be compared one by one according to the storage order, eliminating the need to search for the corresponding first object data from different devices, thus quickly obtaining the comparison result. Furthermore, this avoids the probability of comparing non-corresponding first object data, improving the accuracy of the comparison.

[0104] In some embodiments, the predetermined order includes the order in which the first object data is extracted.

[0105] In other embodiments, the predetermined order also includes the order in which the neural network operators operate or output the first object data. Those skilled in the art will understand that the order in which multiple sets of first object data are stored can be set independently.

[0106] Figures 1-4 illustrate exemplary flowcharts 1-200 of another method for extracting data according to an embodiment of this disclosure.

[0107] As shown in Figures 1-4, this disclosure provides a method for extracting data, wherein the fifth object data includes third object data and fourth object data. The third object data is data from at least some operators in a neural network model running on a first device, and the fourth object data is data from at least some operators in a neural network model running on a second device. The method includes: in step S1-210, summarizing the third object data and the fourth object data on the same device to compare the first object data in the third object data and the first object data in the fourth object data to obtain a comparison result; in step S1-220, determining the abnormal operators in the neural network model based on the comparison result. The first object data in the third object data and the first object data in the fourth object data mentioned above are extracted using the data extraction method described in any of the above embodiments, and will not be elaborated further here.

[0108] In some embodiments, the same device is a first device, a second device, or a new device other than the first device and the second device.

[0109] If, after training a neural network model on the first device, the model is then transferred to the second device for inference, a decrease in inference accuracy or the inability to train the model for inference may occur. This is because anomalies have appeared in the operators within the neural network model. To determine which operators in the neural network model are abnormal, the model needs to be run on both the first and second devices to obtain the aforementioned third and fourth object data. Next, the third and fourth object data are aggregated on the same device. Then, the first object data within the third object data and the first object data within the fourth object data are compared to obtain the comparison results. Finally, based on these comparison results, the anomalous operators in the neural network model can be identified.

[0110] In the embodiments disclosed herein, aggregating the third object data and the fourth object data on the same device further includes: first, serializing the third object data and the fourth object data to obtain the third byte stream data corresponding to the third object data and the fourth byte stream data corresponding to the fourth object data; then, transmitting and aggregating the third byte stream data and the fourth byte stream data on the same device to achieve the aggregation of the third object data and the fourth object data on the same device. Next, comparing the first object data in the third object data and the first object data in the fourth object data further includes: extracting the first object data in the third object data during the deserialization of the third byte stream data, and extracting the first object data in the fourth object data during the deserialization of the fourth byte stream data, thereby achieving the comparison of the first object data in the third object data and the first object data in the fourth object data. The above process will now be described with reference to schematic diagrams.

[0111] Figures 1-5 illustrate exemplary flowcharts 1-210a of a method for summarizing third object data and fourth object data on the same device according to an embodiment of this disclosure.

[0112] As shown in Figures 1-5, step S1-210 disclosed herein specifically includes steps S1-211-S1-212: In step S1-211, the third object data is serialized to obtain the third byte stream data, and the fourth object data is serialized to obtain the fourth byte stream data; in step S1-212, the third byte stream data and the fourth byte stream data are transmitted and aggregated on the same device.

[0113] The terms “third object data”, “third byte stream data”, “fourth object data” and “fourth byte stream data” in steps S1-211-S1-212 have been described above and will not be repeated here.

[0114] The third-byte stream data and the fourth-byte stream data can be aggregated using the methods described in the above embodiments, thereby enabling the third object data and the fourth object data to be aggregated on the same device or in the same file.

[0115] Figures 1-6 illustrate an exemplary flowchart 1-2 of a method for determining anomaly operators in a neural network according to an embodiment of this disclosure. Following steps S1-211-S1-212, steps S1-213 and S1-214 are further included in step S1-210 to determine anomaly operators in the neural network.

[0116] As shown in Figure 1-6, specifically, step S1-210 further includes the following steps: in step S1-213, the third byte stream data and the fourth byte stream data are deserialized to extract the first object data in the third object data and the first object data in the fourth object data; in step S1-214, the first object data in the third object data and the first object data in the fourth object data are compared to determine the comparison result.

[0117] In steps S1-213, the specific method of "deserializing the third byte stream data to extract the first object data" is the same as the specific method of "deserializing the fourth byte stream data to extract the first object data," and both are the same as steps S1-110-S1-120 in the data extraction method described in the above embodiment. That is, during the deserialization of the third and fourth byte stream data, if it is necessary to deserialize the first byte stream data in the third and fourth byte stream data, the specific deserialization method associated with the first byte stream data is first obtained, and then the specific deserialization process is performed on the first byte stream data in the third and fourth byte stream data respectively according to the specific deserialization method, so as to extract the first object data from the third and fourth object data during the deserialization process of the third and fourth byte stream data. This will not be elaborated here.

[0118] Specifically, step S1-214 above includes comparing the first object data in the third object data and the first object data in the fourth object data at corresponding positions one by one to obtain the comparison result.

[0119] Further, in steps S1-220, the specific operator in the neural network model that has become abnormal is determined based on the comparison results, i.e., the abnormal operator.

[0120] In summary, the data extraction method described in the above embodiments can first serialize the object data to obtain the corresponding byte stream data. Then, during the deserialization process of the byte stream data, specific type of first object data is extracted. This method obtains a specific deserialization method associated with the first byte stream data and performs deserialization processing on the first byte stream data according to the specific deserialization method to extract the first object data. In this way, after deserializing the first byte stream data, the specific first object data can be directly extracted. Thus, without the need for complex traversal, specific type of first object data can be quickly extracted from complex object data, simplifying the process of extracting specific type of data and improving the efficiency of extracting specific type of data.

[0121] Secondly, this disclosure provides a computer storage medium. The storage medium stores a computer program for retrieving data, which, when executed by a processor, implements the data retrieval method as described in any of the foregoing embodiments.

[0122] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this disclosure. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this disclosure, and on the specific implementation methods and application scope of this disclosure, are all within the scope of protection of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure. 2024118340622

[0123] Figure 2-1 illustrates an exemplary scenario in which embodiments of this disclosure can be applied. As shown in Figure 2-1, after a neural network model is trained on a GPU (Graphics Processing Unit), the neural network model is transferred to a non-GPU, such as an NPU (Neural Network Processing Unit), to train or run the neural network model on a non-GPU.

[0124] Figure 2-2 is a schematic diagram of an existing scheme for determining the anomaly operators in a neural network model. As shown in Figure 2-2, the neural network model consists of n cascaded operators, where n is a positive integer. The input data of the neural network model is the input data of the first operator. Except for the first operator, the output data of each operator is the input data of the next associated operator, and the output data of the last operator is the output data of the entire neural network model. That is, the input data of operator 1 is the same as the input data of the entire neural network model, the input data of operator 2 is the same as the output data of operator 1, and so on, until the output data of operator n is the same as the output data of the entire neural network model.

[0125] Existing methods for identifying anomalous operators in a ported neural network model include: First, running the neural network model on both a GPU and an NPU, where the input data is identical on both platforms (data A). Next, acquiring the input and output data of each operator in the neural network model on different devices. Specifically, when the neural network model is running on a GPU, acquiring the output data of operator 1 (data a1), the input data of operator 2 (data a1), the output data of operator 2 (data a2), and so on, acquiring the output data of operator (n-1) (data a(n-1)) and the output data of operator n (data B1).

[0126] Similarly, when the neural network model is running on the NPU, the output data of operator 1, i.e., data b1, the input data of operator 2, i.e., data b1, the output data of operator 2, i.e., data b2, and so on, the output data of operator (n-1), i.e. data b(n-1), and the output data of operator n, i.e. data B2, are obtained.

[0127] Then, the input and output data of each operator on different devices are compared separately, that is, data a1 is compared with data b1, data a2 is compared with data b2, and so on, data a(n-1) is compared with data b(n-1) and data B1 is compared with data B2; finally, based on the comparison results, the operators that have precision anomalies are determined.

[0128] In the above method, the input and output data of each operator in the neural network model need to be collected and transferred, which results in a large storage space required for transferring the input and output data of the operators. Moreover, this method requires comparison of the input and output data of each operator, which results in a lot of time spent in the process of identifying operators with abnormal accuracy, which seriously affects the efficiency of operators with abnormal positioning accuracy.

[0129] Exemplary application scheme

[0130] In view of this, the present disclosure provides a scheme for determining anomalous operators in a ported neural network model. This scheme determines anomalous operators in the neural network model by comparing only the input and / or output data of some operators running on different devices, and based on the comparison results. This effectively shortens the time required to determine anomalous operators and improves the efficiency of the determination process. The specific embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings.

[0131] Figure 2-3 illustrates an exemplary flowchart 2-300 of a method for determining anomalous operators in a ported neural network model according to some embodiments of this disclosure.

[0132] As shown in Figure 2-3, in step S2-310, sampling parameters are set, including the granularity of sampling the input and output data of operators in the neural network model; in step S2-320, based on the sampling parameters, input and / or output data of at least a portion of operators of the neural network model are obtained when running on the first device to obtain a first result; in step S2-330, based on the sampling parameters, input and / or output data of at least a portion of operators of the neural network model are obtained when running on the second device to obtain a second result; wherein, when the neural network model is run on the first device and the second device respectively, the input data of the neural network model is the same; in step S2-340, the first result and the second result are compared correspondingly to determine the comparison difference between the first result and the second result; in step S2-350, based on whether the comparison difference reaches a threshold, abnormal operators in the neural network model are determined.

[0133] In the embodiments disclosed herein, "porting" refers to the process of transferring a neural network model, after it has been trained on a first device, to a second device for further training and / or inference. The first device refers to a GPU, and the second device refers to at least one of a CPU (Central Processing Unit) and an NPU. Here, an operator refers to one or more computational units within a neural network model in a deep learning algorithm. In this embodiment, an operator represents the underlying operations or computational logic in the neural network model structure; specifically, operators are used to perform operations such as data computation, transformation, and transfer. For example, operators can be convolution operators, pooling operators, etc.

[0134] The sampling parameters in steps S2-310 above include the granularity of sampling the input and output data of operators in the neural network model. In some embodiments disclosed herein, "granularity" refers to the interval between sampling the input and / or output data of operators in the neural network model. A larger granularity indicates a greater interval between two consecutive samples of input or output data. This means that after sampling a set of input or output data, a larger interval of input and / or output data is required before sampling another set of input or output data to complete the next sampling. Conversely, a smaller granularity indicates a smaller interval between two consecutive samples of input or output data. This means that after sampling a set of input or output data, a smaller interval of input and / or output data is required before sampling another set of input or output data to complete the next sampling.

[0135] In some embodiments, during steps S2-310, the input and output data of all operators in the neural network model need to be arranged in a predetermined order to obtain the sequential order of the input and output data of all operators. The phrase "interval of how many operators' input and / or output data" refers to sampling a set of data from the input and output data after obtaining the sequential order of all operators' input and output data, skipping a certain number of input or output data points. As can be seen from the sampling method described above, "interval of how many operators' input and / or output data" represents the number of input and / or output data points between two consecutive sampled input or output data points. Furthermore, the number of operators skipped or intervened can be set independently and changed at any time, meaning the granularity can be flexibly set.

[0136] The aforementioned "predetermined order" can be at least one of the following: the execution order of operators in a neural network model and the topological order of operators in a neural network model. Here, "execution order of operators" refers to the order in which each operator in the neural network model is executed during its operation; "topological order" refers to the logical connection order of each operator in the neural network model. For example, in a convolutional neural network model including one or more convolution operators, one or more pooling operators, and activation function operators, the topological order of the convolutional neural network model is as follows: one convolution operator is followed by one pooling operator, one pooling operator is followed by another convolution operator, another convolution operator is followed by another pooling operator, and another pooling operator is followed by an activation function operator.

[0137] Specifically, given that the predetermined order is "the order in which operators are executed in the neural network model", for the first and second operators executed in the neural network model, the input data of the first operator is taken as the first data, and the output data of the first operator is taken as the second data; the input data of the second operator is taken as the third data, and the output data of the second operator is taken as the fourth data, and so on, until the input and output data of all operators are arranged to obtain the order in which the input and output data of all operators are ordered.

[0138] For example, if there exists a neural network model N1, which includes operators o1 and o2, and the input and output data of all operators in N1 are arranged according to the order in which they are executed, with operator o1 executed first and operator o2 executed later, and operator o1's input data is input1 and its output data is out1; and operator o2's input data is input2 and its output data is out2, then the order of the input and output data of all operators in N1 from front to back is input1, out1, input2, and out2. With a granularity of 1, this means that sampling is performed every time a data point is skipped or spaced out. Based on this granularity, starting from out1, the input or output data of the operators in the neural network model N1 are sampled, with the first sampled data being out1 and the second sampled data being out2. The first sampled data out1 and the second sampled data out2 are data obtained from two consecutive samples. The number of data between these two consecutive samples is 1, which is input2.

[0139] In other embodiments, the predetermined order is the topological order of the operators in the neural network model.

[0140] Furthermore, when sampling the input data of operators in a neural network model, the number of input data points between each sampling interval can vary depending on the type of operator; that is, the granularity can be flexibly adjusted. For example, at the beginning of the sampling process, the initial granularity is 10, meaning that the input data of operators in the neural network model is sampled every 10 operator input data points. Then, the granularity can be set to 5, meaning that the input data of operators in the neural network model is sampled every 5 operator input data points.

[0141] In other embodiments disclosed herein, the granularity can be a number of operator output data intervals at which the output data of the operators in the neural network model is sampled. For example, with a granularity of 3, the output data of the corresponding operator is sampled every three operator output data intervals.

[0142] In some further embodiments disclosed herein, the granularity can be a number of input and output data intervals at which the input or output data of the operators in the neural network model are sampled. For example, given the sequential order of the input and output data of all the aforementioned operators, if the granularity is 4, then the total number of input and output data intervals between each previously sampled data and the next sampled data is four.

[0143] It should be noted that, in the embodiments disclosed herein, the granularity can be at least one of the granularities described in the above embodiments.

[0144] In some embodiments, steps S2-310 above specifically include: setting sampling parameters according to the risk level of operators in the neural network model. It should be noted that the risk level of operators in the neural network model is inversely proportional to the granularity.

[0145] Here, "risk level" refers to the probability of an operator experiencing precision anomalies. A higher risk level means a higher probability of an operator experiencing precision anomalies; conversely, a lower risk level means a lower probability of an operator experiencing precision anomalies. This risk level is related to the type of operator; generally speaking, the higher the computational complexity of the operator, the higher its risk level. "Precision anomaly" here refers to a situation where an operator, when running on a first device, receives the same input data as when running on a second device, and the difference between the output data from the first device and the second device is too large.

[0146] Specifically, in the process of setting sampling parameters based on the risk level of operators in the neural network model, if the complexity of each operator in the neural network model is low, the granularity value is set to a higher value. That is, when sampling the data of operators in the neural network model according to the sampling parameters, there are more input data and / or output data of operators between two adjacent samplings. If the complexity of each operator in the neural network model is high, the granularity value is set to a lower value. That is, when sampling the data of operators in the neural network model according to the sampling parameters, there are fewer input data and / or output data of operators between two adjacent samplings.

[0147] In this embodiment, since operators with higher risk levels also have a higher probability of precision anomalies, the number of operators with precision anomalies among the multiple operators with higher risk levels is relatively large. In steps S2-310, the granularity corresponding to the multiple operators with higher risk levels is set to a lower level. That is, during the sampling of the input data and / or output data of the multiple operators with higher risk levels, the data between two adjacent samplings includes a smaller number of operators' input data and / or output data. Thus, during the sampling of the multiple operators with higher risk levels, the input data and / or output data of more operators among the multiple operators can be sampled. Multiple comparisons are then performed based on the sampled data to obtain comparison results. Based on the comparison results, it can be determined more precisely whether a larger number of operators among the multiple operators are operators with precision anomalies, which can reduce the probability of missing operators with precision anomalies and thus improve the accuracy of identifying operators with precision anomalies.

[0148] Furthermore, since operators with lower risk levels have a lower probability of precision anomalies, the number of operators with precision anomalies among multiple operators with lower risk levels is relatively small. In steps S2-310, the granularity corresponding to multiple operators with lower risk levels is set to a higher level. That is, during the sampling of the input data and / or output data of the aforementioned multiple operators with lower risk levels, the data between two adjacent samplings includes input data and / or output data of a larger number of operators. Thus, during the sampling of multiple operators with lower risk levels, the input data and / or output data of a smaller number of operators can be sampled. This allows for fewer comparisons based on the sampled data, thereby improving the efficiency of identifying operators with precision anomalies.

[0149] Figure 2-4 shows an exemplary flowchart 2-320 of a method for obtaining operator intervals according to an embodiment of this disclosure.

[0150] As shown in Figures 2-4, in some embodiments, step S2-320 specifically includes: in step S2-321, the neural network model is divided into one or more operator intervals according to the granularity, and the operator interval includes one or more operators; in step S2-322, the left boundary data and right boundary data of each operator interval are obtained when running on the first device to obtain the first result corresponding to each operator interval.

[0151] In step S2-321, the "operator interval" refers to a set of operators consisting of at least one operator in the neural network model. It should be noted that if multiple operators exist within this operator interval, the connection method for these multiple operators can be arbitrary. For example, the connection method can be based on the execution order of the operators, linking the first executed operator to the last executed operator pairwise; or any two operators can be connected to each other. Specifically, multiple operators can be connected according to a certain logical connection topology, such as connecting multiple operators in a tree structure. This disclosure does not limit the connection method for multiple operators within this operator interval.

[0152] As described in the above embodiments, granularity refers to the sampling of a set of data from the input and / or output data of an operator in the neural network model at intervals of a predetermined number of operator inputs and / or outputs, given that the input and output data of all operators in the neural network model are arranged in a predetermined order. If the input and / or output data are sampled according to this granularity, then, given the order of the input and output data of all operators, there will be not only one or more input or output data intervals between two consecutive sampled input or output data, but also one or more operators. Therefore, in steps S2-321, dividing one or more operators in the neural network model into different operator sets according to the "granularity" means that the one or more operators between two consecutive sampled input or output data are considered as one operator set, i.e., an operator interval.

[0153] As described in steps S2-320 above, the "left boundary data" and "right boundary data" are two sets of data sampled consecutively. In step S2-322 above, one or more operators that separate the "left boundary data" and "right boundary data" form the "operator interval" mentioned above.

[0154] For example, if there exists a neural network model N2, which includes operators o3 and o4, and the input and output data of all operators in N2 are arranged according to the order in which they are executed, with operator o3 executed first and operator o4 executed later, and the input data of operator o3 is input3 and the output data of operator o3 is out3; the input data of operator o4 is input4 and the output data of operator o4 is out4, then the order of the input and output data of all operators in the neural network model N2 from front to back is input3, out3, input4, and out4. With a granularity of 1, this means that sampling is performed every time a data point is skipped or spaced out. Based on this granularity, starting from out3, the input or output data of the operators in the neural network model N2 are sampled, with the first sampled data being out3 and the second sampled data being out4. The first sampled data out3 and the second sampled data out4 are data obtained from two consecutive samples. The operator between these two consecutive samples is operator o4.

[0155] In some embodiments, the neural network model is formed by one or more operators in a certain topological order, and the granularity includes structural granularity. Steps S2-321 specifically include: dividing one or more operators according to the structural granularity and the topological order of the neural network model to obtain one or more operator intervals. In this embodiment, the left boundary data is the input data of the leftmost operator in the operator interval, and the right boundary data is the output data of the rightmost operator in the operator interval.

[0156] In this embodiment, "structural granularity" as mentioned above refers to how many operators are divided into an operator interval each time in the above process; that is, "structural granularity" represents the number of operators in the resulting operator interval. "Topological order" here refers to the logical connection order of each operator in the neural network model, which has the same specific meaning as "topological order" mentioned above and will not be elaborated upon here.

[0157] In this embodiment, one or more operators in the neural network model are divided into an operator interval according to the above-mentioned structural granularity and the topological order of the neural network model. Specifically, this includes: according to the order of logical connections of all operators in the neural network model, each time a corresponding number of operators in the above-mentioned structural granularity are taken as an operator interval, and each time one or more operators represented by the structural granularity are selected as a set of operators, i.e., operator intervals.

[0158] It should be noted that the "predetermined order" in this embodiment refers to the topological order of the operators in the neural network model described above in steps S2-321. Specifically, steps S2-321 in this embodiment involve grouping a corresponding number of operators into a set of operators, i.e., an operator interval, according to the order in which all operators are connected, based on the structural granularity. Furthermore, if an operator c connects to two or more operators simultaneously, the multiple operators connected to that operator c will be considered as the same operator interval.

[0159] After step S2-321, one or more operator intervals are obtained; then, in step S2-322, the left boundary data and / or right boundary data of the operator interval are obtained. Here, "left boundary data" and "right boundary data" refer to the input or output data of the previous sample in two consecutive samples; "right boundary data" refers to the input or output data of the next sample in the two consecutive samples. It should be noted that the operator interval includes one or more operators. Further, in the case of dividing operators to obtain operator intervals, the left boundary data of the operator interval is the input data of the leftmost operator in that operator interval, that is, the input data of the first operator in that operator interval according to the topological order of the neural network model; the right boundary data of the operator interval is the output data of the rightmost operator in that operator interval, that is, the output data of the last operator in that operator interval according to the topological order of the neural network model.

[0160] Figure 2-5a shows a schematic diagram of a partitioned neural network model for one embodiment of this disclosure.

[0161] As shown in Figure 2-5a, there exists a neural network model N5a, which is run on the first device. This neural network model N5a includes multiple operators, specifically operators 5ap1, 5ap2, 5ap3, and 5ap4. The input data for operator 5ap1 is 5ap1-in, and its output data is 5ap1-out; the input data for operator 5ap2 is 5ap2-in, and its output data is 5ap2-out; the input data for operator 5ap3 is 5ap3-in, and its output data is 5ap3-out; the input data for operator 5ap4 is 5ap4-in, and its output data is 5ap4-out.

[0162] In this embodiment, the topological order of the neural network model N5a, i.e., the connection order of all operators, is as follows: each operator is connected to the next operator in sequence, i.e., operator 5ap1 is connected to operator 5ap2, and the output data of operator 5ap1 is the same as the input data of operator 5ap2, i.e., 5ap1-out is the same as 5ap2-in; operator 5ap2 is connected to operator 5ap3, and the output data of operator 5ap2 is the same as the input data of operator 5ap3, i.e., 5ap2-out is the same as 5ap3-in; operator 5ap3 is connected to operator 5ap4, and the output data of operator 5ap3 is the same as the input data of operator 5ap4, i.e., 5ap3-out is the same as 5ap4-in.

[0163] If the structural granularity is 2, then in steps S2-321 above, based on this structural granularity and the topological order of the neural network model N4a (i.e., the connection order of all operators), two operators are grouped into one operator interval each time. Specifically, operators 5ap1 and 5ap2 are grouped into one operator interval 5ai1, and operators 5ap3 and 5ap4 are grouped into one operator interval 5ai2. The left boundary data of operator interval 5ai1 is the input data of the leftmost operator in operator interval 5ai1, that is, the input data of the first operator 5ap1 in operator interval 5ai1, i.e., 5ap1-in, under the topological order of the neural network model described above. The right boundary data of operator interval 5ai1 is the output data of the rightmost operator in operator interval 5ai1, that is, the output data of the last operator 5ap2 in operator interval 5ai1, i.e., 5ap2-out, in accordance with the topological order of the above neural network model; similarly, the left boundary data of operator interval 5ai2 is 5ap3-in, and the right boundary data of operator interval 5ai2 is 5ap4-out. Figure 2-5b shows a schematic diagram of a partitioned neural network model for another embodiment of this disclosure.

[0164] As shown in 2-5b, there exists a neural network model N5b, which includes multiple operators, specifically 5bp1, 5bp2, 5bp3, 5bp4, and 5bp5. The input data of operator 5bp1 is 5bp1-in, and the output data of operator 5bp1 is 5bp1-out; the input data of operator 5bp2 is 5bp2-in, and the output data of operator 5bp2 is 5bp2-out; the input data of operator 5bp3 is 5bp3-in, and the output data of operator 5bp3 is 5bp3-out; the input data of operator 5bp4 is 5bp4-in, and the output data of operator 5bp4 is 5bp4-out; the input data of operator 5bp5 is 5bp5-in, and the output data of operator 5bp5 is 5bp5-out.

[0165] In this embodiment, the topological order of the neural network model N5b in Figure 2-5b, i.e., the connection order of all operators, is as follows: operator 5bp1 is connected to operator 5bp2, and the output data of operator 5bp1 is the same as the input data of operator 5bp2, i.e., 5bp1-out and 5bp2-in are the same; operator 5bp1 is connected to operator 5bp3, and the output data of operator 5bp1 is the same as the input data of operator 5bp3, i.e., 5bp1-out and 5bp3-in are the same; operator 5bp2 is connected to operator 5bp4, and the output data of operator 5bp2 is the same as the input data of operator 5bp4, i.e., 5bp2-out and 5bp4-in are the same; operator 5bp4 is connected to operator 5bp5, and the output data of operator 5bp4 is the same as the input data of operator 5bp5, i.e., 5bp4-out and 5bp5-in are the same.

[0166] If the granularity is 2, then in steps S2-321 above, based on the structural granularity and the topological order of the neural network model N5b (i.e., the connection order of all operators), two operators are grouped into one operator interval each time. Specifically, operators 5bp1 and 5bp2 are grouped into one operator interval, and operators 5bp4 and 5bp5 are grouped into one operator interval, i.e., operator interval 5bi2. Moreover, since operators 5bp2 and 5bp3 are both connected to operator 5bp1, and the input data of operators 5bp2 and 5bp3 are the output data of operator 5bp1, from a topological perspective, operators 5bp2 and 5bp3 belong to the same level of operators. Therefore, operators 5bp3 and 5bp2 belong to the same operator interval, i.e., operators 5bp1, 5bp2, and 5bp3 are grouped into one operator interval, i.e., operator interval 5bi1.

[0167] In the above embodiment, the left boundary data of operator interval 5bi1 is the input data of the leftmost operator in operator interval 5bi1, that is, the input data of the first operator 5bp1 in operator interval 5bi1, i.e., 5bp1-in, in accordance with the topological order of the above neural network model. The right boundary data of operator interval 5bi1 is the output data of the rightmost operator in operator interval 5bi1, that is, the output data of the last operator 5bp2 or 5bp3 in operator interval 5bi1, i.e., 5bp2-out or 5bp3-out, in accordance with the topological order of the above neural network model.

[0168] Similarly, the left boundary data of the operator interval 5bi2 is the input data of the operator 5bp4, i.e., 5bp4-in. The right boundary data of the operator interval 5bi2 is the output data of the operator 5bp5, i.e., 5bp5-out.

[0169] As described in the above embodiments, based on the structural granularity and the topological order of the neural network model, the neural network model can be divided into multiple operator intervals from the perspective of its topological structure.

[0170] Figure 2-6 shows an exemplary flowchart 2-321 of a method for obtaining operator intervals according to another embodiment of this disclosure.

[0171] As shown in Figures 2-6, in some other embodiments, the granularity also includes data granularity. Step S2-321 may include: in step S2-3211, obtaining the sampling nodes of the neural network model according to the data granularity and the execution order of each operator in the neural network model; in step S2-3212, based on the execution order, taking one or more operators contained in each pair of adjacent sampling nodes as an operator interval to obtain one or more operator intervals.

[0172] The "data granularity" mentioned above refers to the order in which the operators in the neural network model are executed, and after collecting one piece of input and output data from all operators, how many data points are skipped or intermittently collected before collecting the next input or output data.

[0173] In this embodiment, the predetermined order is the execution order of each operator. The input or output data sampled by an operator in the neural network model is called a "sampling node". The specific process of step S2-3211 is as follows: according to the execution order of each operator in the neural network model and the specific number of input and output data skipped or intermittently in each data granularity, the corresponding input or output data, i.e., the sampling node, is obtained.

[0174] In this embodiment, the specific process in step S2-3212 is as follows: according to the execution order of each operator, the data sampled twice consecutively, i.e., the sampling points, include one or more operators as a set of operators, i.e., the operator interval.

[0175] Figure 2-6a shows a schematic diagram of a partitioned neural network model for one embodiment of this disclosure.

[0176] As shown in Figure 2-6a, there exists a neural network model N6a, which includes multiple operators, specifically operators 6ap1, 6ap2, 6ap3, 6ap4, and 6ap5. The input data of operator 6ap1 is 6ap1-in, and the output data of operator 6ap1 is 6ap1-out; the input data of operator 6ap2 is 6ap2-in, and the output data of operator 6ap2 is 6ap2-out; the input data of operator 6ap3 is 6ap3-in, and the output data of operator 6ap3 is 6ap3-out; the input data of operator 6ap4 is 6ap4-in, and the output data of operator 6ap4 is 6ap4-out; the input data of operator 6ap5 is 6ap5-in, and the output data of operator 6ap5 is 6ap5-out.

[0177] In this embodiment, the topology of the neural network model N6a, i.e., the connection method of all operators, is as follows: operator 6ap1 is connected to operator 6ap2, and the output data of operator 6ap1 is the same as the input data of operator 6ap2, i.e., 6ap1-out and 6ap2-in are the same; operator 6ap2 is connected to operator 6ap3, and the output data of operator 6ap2 is the same as the input data of operator 6ap3, i.e., 6ap2-out and 6ap3-in are the same; operator 6ap3 is connected to operator 6ap4, and the output data of operator 6ap3 is the same as the input data of operator 6ap4, i.e., 6ap3-out and 6ap4-in are the same; operator 6ap4 is connected to operator 6ap5, and the output data of operator 6ap4 is the same as the input data of operator 6ap5.

[0178] Moreover, the order in which the operators in the neural network model N6a are executed is as follows: first, operator 6ap1 is executed; then, operator 6ap2 is executed; next, operator 6ap3 is executed; then, operator 6ap4 is executed; and finally, operator 6ap5 is executed.

[0179] If the data granularity is 3, and the first sampled data is the output data of operator 6ap1, i.e., 6ap1-out, then in the above step S2-3211, it means that according to the data granularity and the execution order of each operator in the neural network model, starting from 6ap1-out, every three sets of data correspond to a set of data as a sampling node. As can be seen from the above description, the sampling nodes in this embodiment include: 6ap1-out, 6ap3-out, and 6ap5-out.

[0180] Since there are 3 sets of data between 6ap1-out and 6ap3-out, and the operators included between 6ap1-out and 6ap3-out are operators 6ap2 and 6ap3, then operators 6ap2 and 6ap3 are considered as one operator interval 6ai1.

[0181] Similarly, since there are 3 sets of data between 6ap3-out and 6ap5-out, and the operators included between 6ap3-out and 6ap5-out are operators 6ap4 and 6ap5, then operators 6ap4 and 6ap5 are considered as one operator interval 6ai2.

[0182] It should be noted that in this embodiment, 6ap1-out is the left boundary data of operator interval 6ai1, 6ap3-out is the right boundary data of operator interval 6ai1; 6ap3-out is the left boundary data of operator interval 6ai2, and 6ap5-out is the right boundary data of operator interval 6ai2.

[0183] In the above embodiment, the left boundary data of operator interval 6ai1 is the input data of the leftmost operator in operator interval 6ai1, that is, the input data of the first operator 6ap2 in operator interval 6ai1, i.e., 6ap2-in, which is 6ap1-out, given the execution order of the operators in the above neural network model. The right boundary data of operator interval 6ai1 is the output data of the rightmost operator in operator interval 6ai1, that is, the output data of the last operator 6ap3 in operator interval 6ai1, i.e., 6ap3-out, given the execution order of the operators in the above neural network model.

[0184] Similarly, the left boundary data of operator interval 6ai2 is the input data of operator 6ap4, i.e., 6ap4-in, which is also 6ap3-out. The right boundary data of operator interval 6ai2 is the output data of operator 6ap5, i.e., 6ap5-out.

[0185] Figure 2-6b shows a schematic diagram of a partitioned neural network model for another embodiment of this disclosure.

[0186] As shown in the figure, there exists a neural network model N6b, which includes multiple operators, specifically operators 6bp1, 6bp2, and 6bp3. The input data of operator 6bp1 is 6bp1-in, and the output data of operator 6bp1 is 6bp1-out; the input data of operator 6bp2 is 6bp2-in, and the output data of operator 6bp2 is 6bp2-out; the input data of operator 6bp3 is 6bp3-in.

[0187] In this embodiment, the topology of the neural network model N6b, i.e., the connection method of all operators, is as follows: operator 6bp1 is connected to operator 6bp2, and the output data of operator 6bp1 is the same as the input data of operator 6bp2, i.e., 6bp1-out and 6bp2-in are the same; operator 6bp1 is connected to operator 6bp3, and the output data of operator 6bp1 is the same as the input data of operator 6bp3, i.e., 6bp1-out and 6bp3-in are the same.

[0188] Moreover, the order in which the operators in the neural network model N6b are executed is as follows: first, operator 6bp1 is executed; then, operator 6bp2 is executed; and finally, operator 6bp3 is executed.

[0189] If the data granularity is 1, and the first sampled data is the output data of operator 6bp1, i.e., 6bp1-out, then in the above step S2-3211, it means that according to the data granularity and the execution order of each operator in the neural network model N6b, starting from 6bp1-out, each set of data corresponding to every other set of input or output data is a sampling node. As can be seen from the above description, the sampling nodes in this embodiment include: 6bp1-out, 6ap2-out, and 6ap3-out.

[0190] Since there is a data interval of 6bp1-out and 6bp2-out, the data interval is 6bp2-in, and the operator included between 6bp1-out and 6bp2-out is operator 6bp2, then operator 6bp2 is regarded as an operator interval 6bi1.

[0191] Similarly, since there is a data interval of 1 set between 6bp2-out and 6bp3-out, the data interval is 6bp3-in, and the operator included between 6bp2-out and 6bp3-out is operator 6bp3, then operator 6bp3 is regarded as an operator interval 6bi2.

[0192] It should be noted that in this embodiment, 6bp1-out is the left boundary data of operator interval 6bi1, 6bp2-out is the right boundary data of operator interval 6bi1; 6bp2-out is the left boundary data of operator interval 6bi2, and 6bp3-out is the right boundary data of operator interval 6bi2.

[0193] Figure 2-6c shows a schematic diagram of a partitioned neural network model for yet another embodiment of this disclosure.

[0194] As shown in the figure, there exists a neural network model N6c, which includes multiple operators: 6cp1, 6cp2, 6cp3, 6cp4, and 6cp5. The input data for operator 6cp1 is 6cp1-in, and the output data is 6cp1-out; the input data for operator 6cp2 is 6cp2-in, and the output data is 6cp2-out; the input data for operator 6cp3 is 6cp3-in, and the output data is 6cp3-out; the input data for operator 6cp4 is 6cp4-in, and the output data is 6cp4-out; and the input data for operator 6cp5 is 6cp5-in, and the output data is 6cp5-out.

[0195] In this embodiment, the topology of the neural network model N6c, i.e., the connection method of all operators, is as follows: operator 6cp1 is connected to operator 6cp2, and the output data of operator 6cp1 is the same as the input data of operator 6cp2, i.e., 6cp1-out and 6cp2-in are the same; operator 6cp1 is connected to operator 6cp3, and the output data of operator 6cp1 is the same as the input data of operator 6cp3, i.e., 6cp1-out and 6cp3-in are the same; operator 6cp2 is connected to operator 6cp4, and the output data of operator 6cp2 is the same as the input data of operator 6cp4, i.e., 6cp2-out and 6cp4-in are the same; operator 6cp4 is connected to operator 6cp5, and the output data of operator 6cp4 is the same as the input data of operator 6cp5.

[0196] Moreover, the order in which the operators in the neural network model N6c are executed is as follows: first, operator 6cp1 is executed; then, operator 6cp2 is executed; next, operator 6cp3 is executed; then, operator 6cp4 is executed; and finally, operator 6cp5 is executed.

[0197] If the data granularity is 2, and the first sampled data is the input data of operator 6cp1, i.e., 6cp1-in, then in the above steps S2-3211, it means that according to the data granularity and the execution order of each operator in the neural network model N6c, starting from 6cp1-in, every two sets of data correspond to a set of data as sampling nodes. As can be seen from the above description, the sampling nodes in this embodiment include: 6cp1-in, 6cp2-out, 6cp4-in, and 6cp5-out.

[0198] Since there are two sets of data between 6cp1-in and 6cp2-out, and the operators between 6cp1-in and 6cp2-out are operators 6cp1 and 6cp2, then operators 6cp1 and 6cp2 are considered as one operator interval 6ci1.

[0199] Similarly, since there are two sets of data between 6cp2-out and 6cp4-in, and the two sets of data are 6cp3-in and 6cp3-out, and the operator included between 6cp2-out and 6cp4-in is operator 6cp3, then operator 6cp3 is taken as an operator interval 6ci2.

[0200] Since there are two sets of data between 6cp4-in and 6cp5-out, and the operators between 6cp4-in and 6cp5-out are operators 6cp4 and 6cp5, then operators 6cp4 and 6cp5 are considered as one operator interval 6ci3.

[0201] It should be noted that in this embodiment, 6cp1-in is the left boundary data of operator interval 6ci1, 6cp2-out is the right boundary data of operator interval 6ci1; 6cp2-out is the left boundary data of operator interval 6ci2, and 6cp4-in is the right boundary data of operator interval 6ci2. Furthermore, the left boundary data 6cp1-in of operator interval 6ci1 is also the input data of operator 6cp1 in operator interval 6ci1, and the right boundary data 6cp2-out of operator interval 6ci1 is the output data of operator 6cp2 in operator interval 6ci1; the left boundary data 6cp2-out of operator interval 6ci2 does not belong to the input data of operator in operator interval 6ci2, and the right boundary data 6cp4-in of operator interval 6ci2 does not belong to the output data of operator in operator interval 6ci2; 6cp4-in is the left boundary data of operator interval 6ci3, 6cp5-out is the right boundary data of operator interval 6ci3, and the left boundary data 6cp4-in of operator interval 6ci3 is also the input data of operator 6cp4 in operator interval 6ci3, and the right boundary data 6cp5-out of operator interval 6ci3 is also the output data of operator 6cp5 in operator interval 6ci3.

[0202] In some embodiments, step S2-330 is specifically implemented by: acquiring the left boundary data and right boundary data of each operator interval when it is run on the second device, so as to obtain a second result corresponding to each operator interval. In this embodiment, the "second result" refers to the left boundary data and right boundary data sampled by each operator interval when it is run on the second device.

[0203] It should be noted that the principle for obtaining the second result in step S2-330 is the same as the principle for obtaining the first result in step S2-320. The difference between steps S2-330 and S2-320 is that the second result in step S2-330 refers to the data sampled by the neural network model when running on the second device, while the first result in step S2-320 refers to the data sampled by the neural network model when running on the first device. Step S2-330 will not be elaborated upon here. It should also be noted that in this embodiment, the order in which steps S2-320 and S2-330 are executed is not limited. Specifically, step S2-320 can be executed first, followed by step S2-330; or step S2-330 can be executed first, followed by step S2-320; or steps S2-320 and S2-330 can be executed simultaneously.

[0204] In some embodiments, the first result corresponding to each operator interval includes the first left boundary data and the first right boundary data of the operator interval, and the second result corresponding to each operator interval includes the second left boundary data and the second right boundary data of the operator interval. In this embodiment, steps S2-340 specifically include: comparing the first left boundary data and the second left boundary data, and comparing the first right boundary data and the second right boundary data corresponding to each operator interval, to determine the comparison difference corresponding to each operator interval.

[0205] Here, "first left boundary data" refers to the left boundary data collected by the neural network model when running on the first device; "second left boundary data" refers to the left boundary data collected by the neural network model when running on the second device; "first right boundary data" refers to the right boundary data collected by the neural network model when running on the first device; and "second right boundary data" refers to the right boundary data collected by the neural network model when running on the second device. For ease of distinction, the left and right boundary data continuously sampled by the neural network model when running on the first device are referred to as "first left boundary data" and "first right boundary data," and the left and right boundary data continuously sampled by the neural network model when running on the second device are referred to as "second left boundary data" and "second right boundary data."

[0206] It is important to understand that, depending on actual needs, the entire neural network model can be run or only a portion of the neural network model (e.g., operator intervals) can be run to sample operator input and output data.

[0207] In some embodiments, the threshold includes a first threshold and a second threshold. Step S2-350 specifically includes: for any operator interval, performing the following steps: if the difference between the first left boundary data and the second left boundary data does not reach the first threshold, and the difference between the first right boundary data and the second right boundary data reaches the second threshold, then the operator interval is determined to be a potential abnormal operator interval.

[0208] Here, the "first threshold" refers to the boundary value used to determine whether the difference between the first left boundary data and the second left boundary data is too large. If the difference between the first left boundary data and the second left boundary data is greater than the first threshold, it means that the difference between the first left boundary data and the second left boundary data is too large. Specifically, this can be manifested as the absolute value of the difference between the first left boundary data and the second left boundary data being too large. The "second threshold" refers to the boundary value used to determine whether the difference between the first right boundary data and the second right boundary data is too large. If the difference between the first right boundary data and the second right boundary data is greater than the second threshold, it means that the difference between the first right boundary data and the second right boundary data is too large. Specifically, this can be manifested as the absolute value of the difference between the first right boundary data and the second right boundary data being too large.

[0209] The “potentially anomalous operator interval” here refers to the operator interval in which there may be operators with anomalous precision.

[0210] In this embodiment, as shown in the embodiment of Figure 2-5a, the first left boundary data is the overall input data of each operator interval when running on the first device, and the second left boundary data is the overall input data of each operator interval when running on the second device; the first right boundary data is the overall output data of each operator interval when running on the first device, and the second right boundary data is the overall output data of each operator interval when running on the second device. Therefore, based on the comparison difference between the first left boundary data and the second left boundary data of each operator interval, it can be determined whether the difference between the overall input data of each operator interval on different devices is too large; based on the comparison difference between the first right boundary data and the second right boundary data of each operator interval, it can be determined whether the difference between the overall output data of each operator interval on different devices is too large. Thus, based on the above comparison results, it can be determined whether there is an operator with suspected precision abnormality within the operator interval.

[0211] In one embodiment, step S2-350 specifically includes: for any operator interval, performing the following steps: if the difference between the first left boundary data and the second left boundary data does not reach a first threshold, and the difference between the first right boundary data and the second right boundary data reaches a second threshold, then further determining whether the difference corresponding to the operator interval has been propagated to subsequent operator intervals. Furthermore, if the difference corresponding to the operator interval has been propagated to subsequent operator intervals, then the operator interval is determined to be a potentially abnormal operator interval; if the difference corresponding to the operator interval has not been propagated to subsequent operator intervals, then further determining whether other operator intervals are potentially abnormal operator intervals.

[0212] The "subsequent operator interval" here refers to the operator interval that has a data dependency relationship with the current operator interval, and this dependent subsequent operator interval can be obtained by using the operator call stack in combination with network scripts.

[0213] Furthermore, after identifying the potential abnormal operator interval, it is possible to further determine whether there are abnormal operators within that interval.

[0214] In the above embodiments, the method further includes: determining whether a potential abnormal operator interval includes a single operator; if the abnormal operator interval includes only a single operator, and the input data and output data of the single operator are the left boundary data and right boundary data of the potential abnormal operator interval, then the single operator is determined to be an abnormal operator.

[0215] In this embodiment, if there is only one operator in the potential abnormal operator interval, and the input data of the operator is the left boundary data of the potential abnormal operator interval and the output data of the operator is the right boundary data of the potential abnormal operator interval, it means that by the difference between the first left boundary data and the second left boundary data and the difference between the first right boundary data and the second right boundary data of the potential abnormal operator interval, the difference between the input data and the output data of this single operator when running on different devices can be accurately determined, thereby determining whether the single operator is an abnormal operator.

[0216] For example, according to the specific partitioning method of the neural network model N6c in Figure 2-6c, for operator interval 6ci2, although operator interval 6ci2 only includes one operator 6cp3, the left boundary data 6cp2-out of operator interval 6ci2 does not belong to the input data of operator 6cp3. Therefore, according to the method of partitioning the neural network model in Figure 2-6c, it is impossible to determine whether the single operator 6cp3 in operator interval 6ci2 is an abnormal operator. At this time, operator interval 6ci2 is run again on the first device and the second device mentioned above, and the input data of operator interval 6ci2 is the same when it is run on the first device and the second device respectively. The output data of operator interval 6ci2 when it is run on the first device and the second device respectively is obtained. By comparing the output data of operator interval 6ci2 when it is run on the first device and the second device respectively, and based on the comparison result, it is possible to determine whether the single operator 6cp3 in operator interval 6ci2 is an abnormal operator.

[0217] In the above embodiments, the method further includes: if the potential abnormal operator interval includes multiple operators, the granularity is adjusted, and according to the adjusted granularity, the input and output data of the operators running on the first device and the second device are further sampled until a potential abnormal operator interval including only a single operator is obtained.

[0218] In this embodiment, if the current potential abnormal operator interval includes multiple operators, it is necessary to further confirm whether there are any abnormal operators in the current potential abnormal operator interval. In this case, this embodiment treats the potential abnormal operator interval as a new neural network model, and determines the abnormal operators in the new neural network model according to the method for determining abnormal operators in the neural network model described above. Furthermore, adjusting the granularity in this embodiment is to adjust the granularity to a smaller value, thereby further dividing the new neural network model to obtain a new operator interval with fewer operators.

[0219] Specifically, first, the granularity is adjusted to a smaller value; then, based on the adjusted granularity, the new neural network model, i.e. the current potential abnormal operator interval, is divided into one or more new operator intervals according to a preset order; then, according to the method described in the above embodiment, it is determined whether the new operator interval is a new potential abnormal operator interval, and so on, until a potential abnormal operator interval containing only a single operator is obtained, thereby identifying the abnormal operator in the neural network model.

[0220] In some embodiments, when the potential anomaly operator interval is run on the first device and the second device respectively, the input data of the leftmost operator in the potential anomaly operator interval is the same.

[0221] In this embodiment, when the potential anomaly operator interval is run again on different devices, the potential anomaly operator interval can be used as a new neural network model. The input data of the new neural network model is the same when it is run on different devices, that is, the first left boundary data and the second left boundary data are the same.

[0222] In some embodiments, for any operator interval, if the difference between the first left boundary data and the second left boundary data of the operator interval reaches a first threshold, the first left boundary data and the second left boundary data are reset to be the same, so that the operator interval runs on the first device and the second device respectively, and the third right boundary data of the operator interval running on the first device and the fourth right boundary data running on the second device are obtained; based on the difference between the third right boundary data and the fourth right boundary data, it is determined whether the operator interval is a potentially abnormal operator interval. Specifically, if the difference between the third right boundary data and the fourth right boundary data reaches a second threshold, the operator interval is determined to be a potentially abnormal operator interval. Otherwise, it is further determined whether other operator intervals are potentially abnormal operator intervals.

[0223] In some embodiments, for any operator interval, if the difference between the first right boundary data and the second right boundary data in the operator interval does not reach the second threshold, then it is further determined whether other operator intervals are potential abnormal operator intervals.

[0224] Here, "third right boundary data" refers to the right boundary data collected when the current operator interval is running on the first device; "fourth right boundary data" refers to the right boundary data collected when the operator interval is running on the second device. For ease of distinction, the right boundary data continuously sampled when the operator interval is running on the first device is called "third right boundary data", and the right boundary data continuously sampled when the operator interval is running on the second device is called "fourth right boundary data".

[0225] In some embodiments, if no anomalous operators are identified within the determined potential anomalous operator intervals, the granularity can be adjusted, and the neural network model can be re-divided into one or more operator intervals based on the new granularity to further determine the potential anomalous operator intervals within them. For example, if the granularity is 6, it can be adjusted to 3, and the potential anomalous operator intervals in the neural network model can be re-determined based on the adjusted granularity.

[0226] It should be noted that the method provided in any of the above embodiments can be implemented through an application programming interface (API). Specifically, in this disclosure, the method for determining anomalous operators in a neural network model provided in the above embodiments is encapsulated into an application programming interface (API). Whenever it is necessary to determine an anomalous operator in a neural network model, this application API is called to implement the above method for determining anomalous operators in a neural network model.

[0227] As can be seen from the embodiments described above, the method for determining anomalous operators in a ported neural network model provided in the first aspect firstly sets sampling parameters; then, based on the sampling parameters, it acquires the input and / or output data of some operators of the neural network model running on a first device to obtain a first result; and acquires the input and / or output data of the some operators of the neural network model running on a second device to obtain a second result; then, it compares the first result and the second result accordingly to determine the comparison difference between the first result and the second result; and determines the anomalous operators in the neural network model based on whether the comparison difference reaches a threshold. In this way, it is not necessary to compare the input and output data of all operators in the neural network model, but only to compare the input and / or output data of some operators in the neural network model running on different devices, and the anomalous operators in the neural network model can be determined based on the comparison results, thereby effectively shortening the time for determining anomalous operators and improving the efficiency of determining anomalous operators.

[0228] Secondly, some embodiments disclosed herein provide an application programming interface (API).

[0229] Figures 2-7 show schematic diagrams of a block diagram of an application programming interface provided by some embodiments of this disclosure.

[0230] As shown in Figure 2-7, the application interface 2-700 includes a parameter setting component 2-710, a first sampling component 2-720, a second sampling component 2-730, a comparison component 2-740, and an exception operator determination component 2-750.

[0231] The parameter setting component 2-710 is used to set sampling parameters, wherein the sampling parameters include the granularity of sampling the input and output data of operators in the neural network.

[0232] The first sampling component 2-720 described above is used to obtain, based on the sampling parameters, the input and / or output data of at least a portion of the operators of the neural network model running on the first device, so as to obtain a first result.

[0233] The aforementioned second sampling component 2-730 is used to acquire the input and / or output data of at least a portion of the operators when the neural network model is run on the second device to obtain a second result, wherein the input data of the neural network model is the same when the neural network model is run on the first device and the second device respectively.

[0234] Comparison component 2-740 is used to compare the first result and the second result accordingly to determine the comparison difference between the first result and the second result.

[0235] Anomaly operator determination component 2-750 is used to determine anomaly operators in the neural network model based on whether the comparison difference reaches a threshold.

[0236] It should be noted that the specific functions of the various components included in the above application programming interface are the same as those of the method for determining the anomaly operators in the transplanted neural network model provided in the first aspect above, and will not be elaborated here.

[0237] Thirdly, some embodiments disclosed herein provide a machine-readable medium. This machine-readable medium has an application programming interface (API) call stored thereon, which, if executed by one or more processors, causes the processors to perform the following operations: in step S2-810, setting sampling parameters, wherein the sampling parameters include the granularity of sampling input and output data of operators in a neural network model; in step S2-820, acquiring, based on the sampling parameters, input data and / or output data of at least a portion of the operators of the neural network model running on a first device to obtain a first result; and in step S2-830, acquiring, based on step S2-830, input data and / or output data of the at least a portion of the operators of the neural network model running on a second device to obtain a second result, wherein the input data of the neural network model is the same when the neural network model is run on the first device and the second device respectively; in step S2-840, comparing the first result and the second result correspondingly to determine a comparison difference between the first result and the second result; and in step S2-850, determining an anomalous operator in the neural network model based on whether the comparison difference reaches a threshold.

[0238] In this embodiment, step S2-810 is the same as step S2-310 in the above embodiment; step S2-820 is the same as step S2-320 in the above embodiment; step S2-830 is the same as step S2-330 in the above embodiment; step S2-840 is the same as step S2-340 in the above embodiment; and step S2-850 is the same as step S2-350 in the above embodiment, and will not be described again here.

[0239] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal applications. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0240] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this disclosure. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this disclosure, and on the specific implementation methods and application scope of this disclosure, are all within the scope of protection of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

[0241] Clause 1. A method for determining anomalous operators in a ported neural network model, wherein the porting refers to migrating the neural network model from a first device to a second device for operation, the method comprising:

[0242] Set sampling parameters, wherein the sampling parameters include the granularity of sampling the input and output data of the operators in the neural network model;

[0243] Based on the sampling parameters, at least a portion of the input and / or output data of the operators of the neural network model are obtained when it runs on the first device to obtain a first result; and

[0244] To obtain a second result, at least a portion of the operators of the neural network model are obtained when the model is run on the first device and the second device, respectively.

[0245] The first result and the second result are compared accordingly to determine the difference between them; and

[0246] Based on whether the comparison difference reaches a threshold, the abnormal operator in the neural network model is determined.

[0247] Clause 2. The method according to Clause 1, wherein setting the sampling parameters includes:

[0248] The sampling parameters are set according to the risk level of the operators in the neural network model, wherein the risk level of the operators in the neural network is inversely proportional to the granularity.

[0249] Clause 3. The method described in Clause 2, wherein the higher the computational complexity of the operator, the higher the level of risk.

[0250] Clause 4. The method according to any one of Clauses 1-3, wherein obtaining the input and / or output data of at least a portion of the operators of the neural network model running on the first device based on the sampling parameters to obtain the first result includes:

[0251] Based on the granularity, the neural network model is divided into one or more operator intervals, each operator interval including one or more operators;

[0252] Obtain the left and right boundary data of each operator interval when it is running on the first device, so as to obtain the first result corresponding to each operator interval.

[0253] Clause 5. The method according to Clause 4, wherein the neural network model is formed by connecting one or more operators in a certain topological order, and wherein the granularity includes structural granularity, and dividing the neural network model into one or more operator intervals according to the granularity includes:

[0254] One or more operators are divided according to the structural granularity and the topological structure of the neural network model to obtain one or more operator intervals.

[0255] Clause 6. The method according to Clause 5, wherein the left boundary data is the input data of the leftmost operator in the operator interval, and the right boundary data is the output data of the rightmost operator in the operator interval.

[0256] Clause 7. The method according to Clause 4, wherein the granularity includes data granularity, and the step of dividing the neural network model into one or more operator intervals according to the granularity includes:

[0257] Based on the data granularity and the execution order of each operator in the neural network model, the sampling nodes of the neural network model are obtained;

[0258] Based on the execution order, one or more operators contained in every two sampling nodes are taken as one operator interval to obtain one or more operator intervals.

[0259] Clause 8. The method according to Clause 7, wherein, when the neural network model is formed by one or more operators in a cascaded manner, the left boundary data is the input data of the leftmost operator in the operator interval, and the right boundary data is the output data of the rightmost operator in the operator interval.

[0260] Clause 9. The method according to Clause 4, wherein obtaining the input and / or output data of at least a portion of the operators of the neural network model running on the second device to obtain the second result comprises:

[0261] Obtain the left and right boundary data of each operator interval when it is running on the second device, so as to obtain the second result corresponding to each operator interval.

[0262] Clause 10. The method according to Clause 9, wherein the first result corresponding to each operator interval includes first left boundary data and first right boundary data of the operator interval, and the second result corresponding to each operator interval includes second left boundary data and second right boundary data of the operator interval, wherein comparing the first result and the second result accordingly to determine the comparison difference between the first result and the second result includes:

[0263] The first left boundary data and the second left boundary data, and the first right boundary data and the second right boundary data corresponding to each operator interval are compared to determine the comparison difference corresponding to each operator interval.

[0264] Clause 11. The method of Clause 10, wherein the threshold includes a first threshold and a second threshold, and determining the anomaly operator in the neural network model based on whether the comparison difference reaches the threshold includes:

[0265] For any of the operator intervals, perform the following steps:

[0266] If the difference between the first left boundary data and the second left boundary data does not reach the first threshold, and the difference between the first right boundary data and the second right boundary data reaches the second threshold, then the operator interval is determined to be a potential abnormal operator interval.

[0267] Clause 12. The method according to Clause 10 or 11, wherein the threshold includes a first threshold and a second threshold, and the step of determining the anomaly operator in the neural network model based on whether the comparison difference reaches the threshold includes:

[0268] For any operator interval, if the difference between the first left boundary data and the second left boundary data of the operator interval reaches the first threshold, then the first left boundary data and the second left boundary data of the operator interval are reset to be the same, and the third right boundary data and the fourth right boundary data of the operator interval running on the first device and the second device are obtained respectively.

[0269] Based on the comparison difference between the third right boundary data and the fourth right boundary data, it is determined whether the operator interval is a potential abnormal operator interval; wherein, when the comparison difference between the third right boundary data and the fourth right boundary data reaches the second threshold, the operator interval is determined to be a potential abnormal operator interval.

[0270] Clause 13. The method of any one of Clauses 10-12, wherein the threshold includes a first threshold and a second threshold, and the step of determining the anomaly operator in the neural network model based on whether the comparison difference reaches the threshold includes:

[0271] For any one of the operator intervals, if the difference between the first right boundary data and the second right boundary data of the operator interval does not reach the second threshold, then it is further determined whether the other operator intervals are potential abnormal operator intervals.

[0272] Clause 14. The method described pursuant to any one of Clauses 11-13 further comprises:

[0273] Determine whether the potential anomaly operator interval contains only a single operator;

[0274] If the potential abnormal operator interval includes only a single operator, and the input and output data of the single operator are the left and right boundary data of the potential abnormal operator interval, then the single operator is determined to be an abnormal operator.

[0275] Clause 15. The method described pursuant to any one of Clauses 11-14 further comprises:

[0276] If the potential abnormal operator interval includes multiple operators, the granularity is adjusted, and based on the adjusted granularity, the input and output data of the operators running on the first device and the second device are further sampled until a potential abnormal operator interval including only a single operator is obtained.

[0277] Clause 16. The method according to Clause 15, wherein when the potential anomaly operator intervals are run on the first device and the second device respectively, the input data of the leftmost operator in the potential anomaly operator intervals is the same.

[0278] Clause 17. The method according to any one of Clauses 1-16, wherein the method is implemented through an Application Programming Interface (API).

[0279] Clause 18. The method according to any one of Clauses 1-17, wherein the first device is a GPU device and the second device includes at least one of a CPU and an NPU.

[0280] Clause 19. An application programming interface, comprising:

[0281] A parameter setting component is used to set sampling parameters, wherein the sampling parameters include the granularity of sampling the input and output data of operators in the neural network;

[0282] A first sampling component is configured to acquire, based on the sampling parameters, input and / or output data of at least a portion of the operators of the neural network model running on a first device, to obtain a first result; and

[0283] The second sampling component acquires the input and / or output data of at least a portion of the operators of the neural network model when it is run on the second device to obtain a second result, wherein the input data of the neural network model is the same when the neural network model is run on the first device and the second device respectively.

[0284] A comparison component is used to compare the first result and the second result correspondingly to determine the comparison difference between the first result and the second result; and

[0285] An anomaly operator determination component is used to determine anomaly operators in the neural network model based on whether the comparison difference reaches a threshold.

[0286] Clause 20. A machine-readable medium having application programming interface (API) calls stored thereon, the API calls, if executed by one or more processors, causing the one or more processors to perform the following operations:

[0287] Set sampling parameters, wherein the sampling parameters include the granularity of sampling the input and output data of operators in the neural network model;

[0288] Based on the sampling parameters, at least a portion of the input and / or output data of the operators of the neural network model are obtained when it runs on the first device to obtain a first result; and

[0289] To obtain a second result, at least a portion of the operators of the neural network model are obtained when the model is run on the first device and the second device, respectively.

[0290] The first result and the second result are compared accordingly to determine the difference between them; and

[0291] Based on whether the comparison difference reaches a threshold, the anomaly operator in the neural network model is determined. 2024118340872

Claims

A method for extracting data, wherein, The data is a first object data of a specific type, and the first byte stream data is obtained by serializing the first object data. The method includes: Obtain the specific deserialization method associated with the first byte stream data; According to the specific deserialization method, the first byte stream data is deserialized to extract the first object data. According to the method of claim 1, wherein, The second byte stream data is obtained by serialization processing using the specific deserialization method; The method for obtaining the specific deserialization method associated with the first byte stream data includes: The second byte stream data is deserialized to obtain the specific deserialization method. The method according to claim 1 or 2, wherein, The first object data is a portion of the fifth object data, wherein the fifth byte stream data is obtained by serializing the fifth object data, and the step of deserializing the first byte stream data according to the specific deserialization method to extract the first object data includes: In response to the first byte stream data being the target during the deserialization process of the fifth byte stream data, the first byte stream data is deserialized using the specific deserialization method to extract the first object data. The method according to claim 3, wherein, The step of deserializing the first byte stream data according to the specific deserialization method to extract the first object data further includes: In response to the fact that the target during the deserialization process of the fifth byte stream data is non-first byte stream data, the non-first byte stream data is deserialized using a conventional deserialization method. The method according to any one of claims 1-4, wherein, The step of deserializing the first byte stream data to extract the first object data includes: The first byte stream data is deserialized to obtain the first object data, and the first object data is saved in a predetermined order. The method of claim 5, wherein the predetermined order includes the order in which the first object data is extracted. The method according to any one of claims 1-6, wherein, The first object data is tensor type data. The method according to any one of claims 1-7, wherein, The fifth object data is the input and / or output data of at least some operators in the neural network model when running on the first device and / or the second device. The method according to claim 8, wherein, The fifth object data includes the third object data and the fourth object data, wherein the third object data is data from at least some operators in the neural network model running on the first device, and the fourth object data is data from at least some operators in the neural network model running on the second device. The method further includes: The third object data and the fourth object data are aggregated on the same device to compare the first object data in the third object data and the first object data in the fourth object data to obtain a comparison result; The anomaly operators in the neural network model are determined based on the comparison results. The method according to claim 9, wherein, The same device is the first device, the second device, or a new device other than the first device and the second device. A computer storage medium, wherein, The computer storage medium stores a computer program, which, when executed by a processor, implements the method according to any one of claims 1-10.

Images