Data processing method, computing cluster, and related device

Abstract

A data processing method, a computing cluster, and a related device, relating to the technical field of artificial intelligence (AI). In the process of executing a first communication operator in an AI task, a first computing node uses a first scaling factor to convert first data of a lower precision in a first format into second data of a higher precision in a second format, the first data being data generated when executing a subtask in the AI task; the first computing node sends the first scaling factor and the second data to a second computing node; and in the process of executing the first communication operator, the second computing node uses the first scaling factor to convert the second data into third data in the first format, and executes vector calculation on the basis of the third data. In this way, the second computing node uses the scaling factor sent by the first computing node to convert low-precision second data to obtain high-precision third data, thereby improving the accuracy of the second computing node in restoring high-precision data, and thus improving the precision of the computing cluster in executing AI tasks.

Description

Data processing methods, computing clusters and related equipment

[0001] This application claims priority to Chinese Patent Application No. 202411644341.2, filed on November 15, 2024, entitled “Data Processing Method, Computing Cluster and Related Equipment”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence technology, and in particular to a data processing method, computing cluster and related equipment. Background Technology

[0003] Currently, artificial intelligence (AI) models are typically deployed in computing clusters. For example, different network layers of an AI model can be deployed across different computing nodes within the cluster. The computing cluster can then perform distributed training on the AI ​​model or run it across multiple computing nodes to provide inference services. During AI model training / inference, computing nodes in the cluster execute communication operators, such as AllGather and ReduceScatter operators, to interact with other computing nodes about the computational results generated during the forward propagation phase or the gradients calculated during the backpropagation phase. Typically, computing nodes use high-precision data formats for computation and communication, such as representing each data point with 16 or 32 bits, to improve the training / inference accuracy of the AI ​​model. However, exchanging high-precision data (such as gradient data) between different computing nodes not only consumes significant bandwidth resources but also results in lengthy communication times, thus impacting the overall training / inference efficiency of the AI ​​model.

[0004] In practical applications, when computing nodes execute certain types of communication operators, they usually convert the high-precision data obtained from the calculation into low-precision data. For example, they convert high-precision data represented by 16 bits into low-precision data represented by 8 bits. This allows the computing node to send the low-precision data to other computing nodes, thereby reducing the bandwidth resources required for data exchange between different computing nodes and improving communication efficiency.

[0005] However, after the data interaction is completed, the computing node continues to perform the training / inference process on the AI ​​model based on the received data, which can easily lead to the problem of low training / inference accuracy of the AI ​​model. Summary of the Invention

[0006] This application provides a data processing method that improves the training / inference accuracy of AI models while achieving high training / inference efficiency. Furthermore, this application also provides a corresponding computing cluster, computing device, computer-readable storage medium, and computer program product.

[0007] Firstly, this application provides a data processing method applied to a computing cluster comprising multiple computing nodes that execute AI tasks (such as training tasks for AI models) in parallel. These multiple computing nodes include a first computing node and a second computing node that need to communicate with each other. During the execution of a first communication operator in the AI ​​task, the first computing node uses a first scaling factor to convert first data in a first format into second data in a second format. The precision of the first format data is higher than that of the second format data; for example, the first format is FP16 (or FP32), and the second format is INT8 (or FP16). The converted first data is data generated when the first computing node executes a subtask within the AI ​​task. For example, when the AI ​​model is a model training task, the subtask may be, for example, a task training a portion of the network layers in the AI ​​model. Then, the first computing node sends the first scaling factor and the second data in the second format to the second computing node. During the execution of the first communication operator, the second computing node uses a first scaling factor to convert the second data in the second format into the third data in the first format, such as converting the second data in the FP16 format into the third data in the FP32 format, and performs vector calculations based on the third data in the first format, such as performing vector calculation processes in network layers such as random deactivation layers, residual layers, or normalization layers in the AI ​​model.

[0008] Because the first computing node not only sends low-precision second data to the second computing node but also sends a scaling factor, and the first computing node uses this scaling factor to convert high-precision first data into low-precision second data, the deviation between the high-precision third data obtained by the second computing node after converting the low-precision second data and the first data in the first computing node is usually small. This improves the accuracy of the high-precision data restored by the second computing node, thereby improving the accuracy of the computing cluster in executing AI tasks. For example, when the AI ​​task is specifically for training or inference of an AI model, the second computing node can continue vector calculations in the model training / inference process based on the more accurate third data, enabling the AI ​​model to achieve a higher level of training / inference accuracy. Simultaneously, communication between different computing nodes based on low-precision data effectively reduces the amount of communication data between them, thereby reducing the bandwidth resources required for data processing, improving data processing efficiency, and enabling the overall training / inference efficiency of the AI ​​model to reach a high level. In addition, the first computing node can perform a scaling operation only once before sending the low-precision second data to the second computing node. This can keep the additional computing overhead introduced by the first computing node in communicating with the second computing node based on the low-precision data at a low level.

[0009] In one possible implementation, the computing nodes in the computing cluster further include a third computing node. During the execution of the first communication operator, this third computing node can use a second scaling factor to convert fourth data in a first format into fifth data in a second format. This fourth data is data generated when the third computing node executes a subtask within the AI ​​task. Furthermore, the third computing node also sends the second scaling factor and the fifth data in the second format to the second computing node. Correspondingly, the second computing node receives the second scaling factor and the fifth data in the second format. Thus, when the second computing node performs vector computation based on the third data in the first format, specifically, it can perform vector computation based on the third data in the first format and a sixth data in the first format, where the sixth data is obtained by converting the fifth data in the second format using the second scaling factor. Because different computing nodes use different scaling factors when converting data formats, multiple computing nodes send their optimal scaling factors to a second computing node. This allows the second computing node to accurately restore the low-precision data sent by different computing nodes to high-precision data using different scaling factors. This improves the accuracy of the high-precision data restored by the second computing node from each computing node. Furthermore, based on the more accurately restored data, the second computing node can continue vector calculations during model training / inference, enabling the AI ​​model to achieve a higher level of training / inference accuracy.

[0010] In one possible implementation, when the second computing node converts the second data in the second format into the third data in the first format using the first scaling factor, specifically, this can be done during the process of receiving the second scaling factor and the fifth data. In this way, by executing the data format conversion process and the data receiving process in parallel, the second computing node can improve the efficiency of executing the first communication operator, thereby improving the efficiency of the computing cluster in executing AI tasks.

[0011] In one possible implementation, when the second computing node converts the second data in the second format into the third data in the first format using the first scaling factor, it can specifically perform a multiply-accumulate operation based on the first scaling factor, the second data, the second scaling factor, and the fifth data to obtain the sum of the third data and the sixth data. Therefore, when the second computing node performs vector calculations based on the third data in the first format and the sixth data in the first format, it can specifically perform vector calculations based on this sum. In this way, by performing multiply-accumulate operations, the second computing node can achieve data scaling and summation, which can improve the data computation efficiency of the second computing node and reduce the consumption of I / O resources.

[0012] In one possible implementation, the AI ​​task is a training or inference task for an AI model based on a transform architecture, and the first communication operator executed by the first computing node is a reduce hash operator.

[0013] In one possible implementation, the computing cluster further includes a third computing node. The first computing node can also determine a first local scaling factor for the seventh data in the first format, which is data generated when the first computing node executes a subtask in the AI ​​task. Additionally, the first computing node can obtain a second local scaling factor from the third computing node. This second local scaling factor is used to convert the data required by the third computing node to execute the second communication operator in the AI ​​task from the first format to the second format. Thus, the first computing node can determine a global scaling factor, which is the smaller of the first and second local scaling factors. During the execution of the second communication operator, the first computing node can use the global scaling factor to convert the seventh data in the first format into eighth data in the second format and send the eighth data in the second format to the second computing node. The second computing node can then perform matrix calculations based on the eighth data in the second format. In this way, the second computing node can directly perform subsequent matrix calculations based on the second format data without needing to perform format conversion. This not only reduces the computing power required for matrix calculations but also improves the efficiency of matrix calculations by performing matrix calculations based on low-precision, smaller-volume data. Meanwhile, when the AI ​​task is specifically a training task for an AI model, since the data precision in the matrix calculation process usually has a low impact on the training precision of the AI ​​model, the precision of the trained AI model can still reach a high level after the second computing node performs matrix calculation based on the second format data and performs subsequent AI model training.

[0014] In one possible implementation, the AI ​​task is a training or inference task for an AI model based on a transform architecture, and the second communication operator is the AllGather operator.

[0015] Secondly, this application provides a computing cluster comprising multiple computing nodes that execute artificial intelligence (AI) tasks in parallel. The multiple computing nodes include a first computing node and a second computing node. The first computing node, during the execution of a first communication operator in the AI ​​task, uses a first scaling factor to convert first data in a first format into second data in a second format. The precision of the first format data is higher than that of the second format data. The first data is data generated by the first computing node when executing a subtask within the AI ​​task. The first computing node then sends the first scaling factor and the second data in the second format to the second computing node. The second computing node, during the execution of the first communication operator, uses the first scaling factor to convert the second data in the second format into third data in a first format. Finally, it performs vector computation based on the third data in the first format.

[0016] In one possible implementation, the plurality of computing nodes further includes a third computing node; the third computing node is configured to, during the execution of the first communication operator, convert fourth data in a first format into fifth data in a second format using a second scaling factor, wherein the fourth data is data generated by the third computing node when executing a subtask in the AI ​​task; send the second scaling factor and the fifth data in the second format to the second computing node; the second computing node is further configured to receive the second scaling factor and the fifth data in the second format; the second computing node is configured to: perform vector calculations based on the third data in the first format and the sixth data in the first format, wherein the sixth data is obtained by converting the fifth data in the second format using the second scaling factor.

[0017] In one possible implementation, the second computing node is configured to: convert the second data in the second format into the third data in the first format using the first scaling factor during the process of receiving the second scaling factor and the fifth data.

[0018] In one possible implementation, the second computing node is configured to: perform a multiply-accumulate operation based on the first scaling factor, the second data, the second scaling factor, and the fifth data to obtain the sum of the third data and the sixth data; and perform vector calculations based on the sum.

[0019] In one possible implementation, the AI ​​task is a training or inference task for an AI model based on a transform architecture, and the first communication operator is a reduction hash operator.

[0020] In one possible implementation, the computing cluster further includes a third computing node; the first computing node is further configured to: determine a first local scaling factor for seventh data in a first format, the seventh data being data generated when the first computing node executes a subtask in an AI task; obtain a second local scaling factor from the third computing node, the second local scaling factor being used to convert the data required by the third computing node to execute a second communication operator in the AI ​​task from the first format to the second format; determine a global scaling factor, the global scaling factor being the smaller value between the first local scaling factor and the second local scaling factor; during the execution of the second communication operator, convert the seventh data in the first format into eighth data in the second format using the global scaling factor; send the eighth data in the second format to the second computing node; the second computing node is further configured to perform matrix calculations based on the eighth data in the second format.

[0021] In one possible implementation, the AI ​​task is a training or inference task for an AI model based on a transform architecture, and the second communication operator is a full aggregation operator.

[0022] Thirdly, this application provides a data processing method applied to a second computing node, which executes an artificial intelligence (AI) task in parallel with a first computing node. The method includes: receiving a first scaling factor and second data in a second format from the first computing node, wherein the first scaling factor is used to convert the first data in the first format into second data in the second format, the precision of the first data being higher than that of the second data, and the first data being data generated by the first computing node when executing a subtask in the AI ​​task; during the execution of a first communication operator in the AI ​​task, converting the second data in the second format into third data in the first format using the first scaling factor; and performing vector computation based on the third data in the first format.

[0023] In one possible implementation, the method further includes: receiving a second scaling factor from a third computing node and fifth data in a second format, the second scaling factor being used to convert fourth data in a first format into fifth data in a second format, the fourth data being data generated when the third computing node executes a subtask in an AI task; and performing vector computation based on the third data in the first format, including: performing vector computation based on the third data in the first format and sixth data in the first format, the sixth data being obtained by converting the fifth data in the second format using the second scaling factor.

[0024] In one possible implementation, converting second data in a second format into third data in a first format using a first scaling factor includes: during the process of receiving the second scaling factor and the fifth data, converting the second data in a second format into third data in a first format using the first scaling factor.

[0025] In one possible implementation, the second computing node uses a first scaling factor to convert the second data in the second format into the third data in the first format, including: the second computing node performs a multiply-accumulate operation based on the first scaling factor, the second data, the second scaling factor, and the fifth data to obtain the sum of the third data and the sixth data; the second computing node performs vector calculation based on the third data in the first format and the sixth data in the first format, including: the second computing node performs vector calculation based on the sum.

[0026] In one possible implementation, the AI ​​task is a training or inference task for an AI model based on a transform architecture, and the first communication operator is a reduction hash operator.

[0027] In one possible implementation, the AI ​​task is further executed by a third computing node, and the method further includes: a second computing node receiving eighth data in a second format from a first computing node, wherein the eighth data in the second format is obtained by converting seventh data in a first format using a global scaling factor, the seventh data being data generated by the first computing node when executing a subtask in the AI ​​task, the global scaling factor being the smaller of a first local scaling factor and a second local scaling factor, the first local scaling factor being determined based on the seventh data in the first format, and the second local scaling factor being used to convert the data required by the third computing node to execute the second communication operator in the AI ​​task from the first format to the second format; and the second computing node performing matrix calculations based on the eighth data in the second format.

[0028] In one possible implementation, the AI ​​task is a training or inference task for an AI model based on a transform architecture, and the second communication operator is a full aggregation operator.

[0029] Fourthly, this application provides a computing node that executes an artificial intelligence (AI) task in parallel with a first computing node. The computing node method includes: a communication module for receiving a first scaling factor and second data in a second format from the first computing node, wherein the first scaling factor is used to convert the first data in the first format into second data in the second format, and the precision of the first data is higher than that of the second data, and the first data is data generated by the first computing node when executing a subtask in the AI ​​task; a conversion module for converting the second data in the second format into third data in the first format using the first scaling factor during the execution of a first communication operator in the AI ​​task; and a computing module for performing vector calculations based on the third data in the first format.

[0030] In one possible implementation, the communication module is further configured to receive a second scaling factor from the third computing node and fifth data in a second format, the second scaling factor being used to convert fourth data in a first format into fifth data in a second format, the fourth data being data generated when the third computing node executes a subtask in the AI ​​task; the computing module is configured to: perform vector calculations based on the third data in a first format and the sixth data in a first format, the sixth data being obtained by converting the fifth data in a second format using the second scaling factor.

[0031] In one possible implementation, the conversion module is configured to: convert the second data in the second format into the third data in the first format using the first scaling factor during the process of the communication module receiving the second scaling factor and the fifth data.

[0032] In one possible implementation, the conversion module is configured to: perform a multiply-accumulate operation based on the first scaling factor, the second data, the second scaling factor, and the fifth data to obtain the sum of the third data and the sixth data; the calculation module is configured to: perform vector calculation based on the sum.

[0033] In one possible implementation, the AI ​​task is a training or inference task for an AI model based on a transform architecture, and the first communication operator is a reduction hash operator.

[0034] In one possible implementation, the AI ​​task is further executed by a third computing node. The communication module is also used to receive eighth data in a second format from the first computing node. The eighth data in the second format is obtained by converting the seventh data in the first format using a global scaling factor. The seventh data is the data generated when the first computing node executes a subtask in the AI ​​task. The global scaling factor is the smaller of a first local scaling factor and a second local scaling factor. The first local scaling factor is determined based on the seventh data in the first format. The second local scaling factor is used to convert the data required by the third computing node to execute the second communication operator in the AI ​​task from the first format to the second format. The calculation module is also used to perform matrix calculations based on the eighth data in the second format.

[0035] In one possible implementation, the AI ​​task is a training or inference task for an AI model based on a transform architecture, and the second communication operator is a full aggregation operator.

[0036] Fifthly, this application provides a computing device including a processor and a memory. The processor and the memory communicate with each other. The processor executes instructions stored in the memory to cause the computing device to perform a data processing method as described in the third aspect or any implementation thereof. It should be noted that the memory may be integrated into the processor or may be independent of the processor. The computing device may also include a bus. The processor is connected to the memory via the bus. The memory may include readable storage and random access memory.

[0037] In a sixth aspect, this application provides a computer-readable storage medium storing instructions that, when executed on a computing device, cause the computing device to perform the operation steps of the data processing method described in the first aspect or any implementation thereof, or to perform the operation steps of the data processing method described in the third aspect or any implementation thereof.

[0038] In a seventh aspect, this application provides a computer program product containing instructions that, when run on a computing device, causes the computing device to perform the operation steps of the data processing method described in the first aspect or any implementation thereof, or to perform the operation steps of the data processing method described in the third aspect or any implementation thereof.

[0039] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0040] Figure 1 is a schematic diagram of the structure of an exemplary data processing system provided in this application.

[0041] Figure 2 is a flowchart illustrating a data processing method provided in this application;

[0042] Figure 3 is a schematic diagram of converting FP16 format data into FP8 format data;

[0043] Figure 4 is a schematic diagram of the data processing and processing logic indicated by the ReduceScatter operator;

[0044] Figure 5 is a schematic diagram of the parallel execution of communication and computation processes by computing node 102;

[0045] Figure 6 is a flowchart illustrating another data processing method provided in this application;

[0046] Figure 7 is a schematic diagram showing that during the training of an AI model, the computing node performs matrix calculations based on low-precision data and vector calculations based on high-precision data;

[0047] Figure 8 is a schematic diagram of the structure of a computing node provided in this application;

[0048] Figure 9 is a schematic diagram of the hardware structure of a computing device provided in this application. Detailed Implementation

[0049] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a method of distinction used in describing objects with the same attributes in the embodiments of this application.

[0050] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0051] Referring to Figure 1, which is a schematic diagram of the structure of an exemplary data processing system 10, the data processing system 10 includes a computing cluster 100 and a communication network 200. The computing cluster 100 includes multiple computing nodes, and the communication network 200 includes multiple switching nodes. For ease of understanding, Figure 1 illustrates an example of a computing cluster 100 comprising four computing nodes (computing nodes 101 to 104) and three switching nodes (switching nodes 201 to 203). Furthermore, different computing nodes in the computing cluster 100 can interact with each other through the switching nodes in the communication network 200. For example, gradient data calculated by computing node 101 can be forwarded to computing node 102 through switching node 201.

[0052] For example, a computing node in computing cluster 100 refers to a node with data computing capabilities, which can be implemented by a processor or by a computing device including a processor, such as a server. The processor can be any type of processor or any combination thereof, such as a central processing unit (CPU), an accelerator, an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system-on-chip (SoC), a software-defined infrastructure (SDI) chip, an artificial intelligence (AI) chip, or a data processing unit (DPU). An accelerator can be, for example, a graphics processing unit (GPU), a neural network processing unit (NPU), or a tensor processing unit (TPU).

[0053] A switching node is a node with data exchange capabilities, such as a switch or router. For example, multiple switching nodes in a communication network 200 can be deployed in multiple layers. For instance, taking a two-layer deployment as an example, some switching nodes can act as leaf nodes, while others can act as spine nodes. Furthermore, different leaf nodes can interact with each other through spine nodes, as shown in Figure 1.

[0054] In practical applications, the computing cluster 100 can run AI tasks, and these AI tasks can be executed in parallel by multiple computing nodes within the cluster. For example, the AI ​​task could be a training task or an inference task for an AI model. When the AI ​​model has a large number of parameters, the computing cluster 100 can utilize multiple computing nodes for distributed training of the AI ​​model, or utilize multiple computing nodes to run the AI ​​model to provide corresponding inference services. For example, the AI ​​model could be a large language model (LLM), a bidirectional encoder representations from transformers (BERT) model, or a generative pre-trained Transformer 3 (GPT-3) model, or other types of models such as GPT-4; there is no limitation on the specific type.

[0055] During the execution of AI tasks, the computing nodes in computing cluster 100 typically execute communication operators and interact with other computing nodes. For example, when the AI ​​task is specifically a training task for an AI model, during the distributed training process of the AI ​​model, multiple computing nodes can exchange their calculated gradient data during the backpropagation phase to update the parameter values ​​in the AI ​​model using the gradient data calculated by multiple computing nodes. Taking the data interaction between computing node 101 and computing node 102 as an example, before sending data to computing node 102, computing node 101 can first convert higher-precision data A into lower-precision data B. For example, computing node 101 can convert data A represented by 16 bits into data B represented by 8 bits. Then, computing node 101 can send the lower-precision data B to computing node 102. Computing node 102 can convert the received data B into higher-precision data C, and then perform subsequent training or inference processes based on the higher-precision data C. However, in real-world applications, the higher-precision data C obtained by computing node 102 from the lower-precision data B often differs significantly from the data A that computing node 101 needs to send to computing node 102. In other words, computing node 102 has a lower accuracy in restoring the higher-precision data. As a result, computing node 102 continues to perform the training or inference process based on the lower-accuracy data C, which can easily lead to lower training or inference accuracy of the AI ​​model.

[0056] Based on this, in the data processing system 10 provided in this application, during the execution of communication operators in the AI ​​task, before sending data to the computing node 102, the computing node 101 can first convert higher-precision data 'a' into lower-precision data 'b', where data 'a' is the data generated by the computing node 101 when executing a subtask in the AI ​​task. Then, the computing node 101 not only sends the lower-precision data 'b' to the computing node 102, but also sends the scaling factor used in the conversion of data 'b' to the computing node 102. In this way, the computing node 102 can convert the lower-precision data 'b' into higher-precision data 'c' according to the received scaling factor, and perform vector calculations based on data 'c', such as performing vector calculation operations in the AI ​​model training or inference process based on data 'c'.

[0057] Since compute node 101 converts higher-precision data 'a' into lower-precision data 'b' based on the scaling factor, the resulting data c (after compute node 102 converts the lower-precision data b into higher-precision data c) typically has a smaller deviation from the data 'a' in compute node 101. This improves the accuracy of the higher-precision data restored by compute node 102. Therefore, compute node 102 can improve the accuracy of AI tasks performed by compute cluster 100 by performing vector calculations in subtasks based on the higher-precision data c. For example, when the AI ​​task specifically involves training or inference of an AI model, compute node 102 can continue with subsequent vector calculations in model training / inference based on the higher-precision data c, thereby improving the training / inference accuracy of the AI ​​model.

[0058] Meanwhile, communication between different computing nodes based on lower precision data can effectively reduce the amount of communication data between different computing nodes. This allows the computing cluster 100 to achieve a high level of precision in executing AI tasks while reducing the bandwidth resources required for data processing and improving data processing efficiency.

[0059] In addition, before sending the lower-precision data b to the computing node 102, the computing node 101 only needs to perform a scaling operation once, which keeps the additional computational overhead introduced by the computing node 101 in communicating with the computing node 102 based on the lower-precision data at a low level.

[0060] It is worth noting that the data processing system 10 shown in Figure 1 is merely an illustrative example and is not intended to be limiting. For example, in other possible implementations, the data processing system 10 may include more types of nodes, such as management nodes or scheduling nodes. The management node can be used to manage the computing power of the computing nodes in the computing cluster 100, such as putting some computing nodes in the computing cluster 100 into hibernation to reduce power consumption; or, the management node can be used to configure routing for the switching nodes in the communication network 200. The scheduling node can be used to schedule AI models to at least one computing node in the computing cluster 100, and different AI models can be scheduled to different computing nodes in the computing cluster 100. Furthermore, the number of computing nodes and the number of switching nodes in the data processing system 10 can be other numbers, or different computing nodes can be connected via a bus, thus eliminating the need for data communication through switching nodes; this is not limited.

[0061] For ease of understanding, embodiments of the data processing method provided in this application will be described below with reference to the accompanying drawings.

[0062] Referring to Figure 2, which is a flowchart illustrating a data processing method according to an embodiment of this application, this method can be applied to the data processing system 10 shown in Figure 1, or to other applicable data processing systems. For ease of explanation, this embodiment uses the data processing system 10 shown in Figure 1 as an example for illustrative purposes.

[0063] The data processing method shown in Figure 2 may specifically include:

[0064] S201: During the execution of the communication operator in the AI ​​task, the computing node 101 uses a scaling factor 1 to convert the first format data 1 into the second format data 2. The precision of the first format data is higher than that of the second format data. Data 1 is the data generated by the computing node 101 when executing the sub-task in the AI ​​task.

[0065] In this embodiment, during the execution of an AI task in the computing cluster 100, computing nodes 101, 102, and 103 in the computing cluster 100 will each execute one or more subtasks of the AI ​​task. Furthermore, during the execution of subtasks, computing nodes 101 to 103 can execute corresponding communication operators to communicate with each other and execute the data generated by the subtasks.

[0066] For example, taking the AI ​​task specifically as the training task for an AI model, computing nodes 101 to 103 can all deploy the same network layer from the AI ​​model, and each computing node can compute corresponding data for that network layer (i.e., data generated by executing sub-tasks in the AI ​​task). Computing nodes 101 and 103 can send the data they computed for that network layer to computing node 102, so that computing node 102 can perform reduction on the data computed by these three computing nodes, and feed back the reduction result to computing nodes 101 and 103, so that each computing node can continue to execute the subsequent training process based on the reduction result, such as updating the parameter values ​​in the network layer using the reduction result. Computing nodes 101 to 103 can interact with each other through at least one exchange node in the communication network 200, which will not be elaborated further below.

[0067] Typically, the data generated by each computing node executing subtasks can be in a first format. This first format is usually a high-precision data format. For example, in model training scenarios, the formats of various parameters and activation values ​​in an AI model can be 32-bit floating-point (FP32), 16-bit floating-point (FP16), or BF16 (brain floating-point 16-bit) formats. FP32 precision data means that each data point is represented using 32 bits; FP16 and BF16 precision data mean that each data point is represented using 16 bits. Therefore, the data generated by computing nodes 101 to 103 during the training of the AI ​​model can be high-precision data. Distributed training of the AI ​​model based on high-precision data can help improve the accuracy of the trained AI model. At the same time, higher data precision usually means a larger data volume, resulting in more bandwidth resources required for data interaction between different computing nodes.

[0068] Based on this, before sending data to computing node 102, computing node 101 can first convert the higher precision data into lower precision data to reduce the amount of data that needs to be transmitted between different computing nodes.

[0069] In practical implementation, computing node 101 can generate data 1 in the first format during the execution of subtasks in the AI ​​task. For example, in a model inference scenario, computing node 101 can calculate the activation value (i.e., data 1) based on the weight values ​​in the network layer; or, in a model training scenario, computing node 101 can calculate the gradient of the current network layer (i.e., data 1) based on the gradient in the next network layer, etc. Furthermore, the data 1 in the first format is data in a high-precision format, such as FP32 or FP16 format data.

[0070] Then, compute node 101 can execute a communication operator to send the computed data 1 in the first format to compute node 102.

[0071] In one possible implementation, during the execution of the communication operator, the computing node 101 can first determine a scaling factor 1 based on the data 1 in the first format. This scaling factor 1 can scale the value of the data 1 in the first format to the numerical range that the data in the second format can represent. The second format can be a relatively low-precision data format, such as an 8-bit floating-point number (FP8), a 6-bit floating-point number (FP6), a 4-bit floating-point number (FP4), an 8-bit integer (INT8), a 4-bit integer (INT4), HiF8, or NF4, etc., and is not limited thereto. For example, the computing node 101 can calculate the quotient between the data 1 and the maximum value that the second format can represent, round the quotient down, and then use the reciprocal of the rounded integer as the scaling factor 1 corresponding to the data 1.

[0072] Then, computing node 101 can use the scaling factor 1 to convert the first format data 1 into the second format data 2, the value of which is within the range of values ​​that the second format data can represent.

[0073] In practice, compute node 101 can first scale the data 1 in the first format using a scaling factor 1. Specifically, this can be done by calculating the product between the data 1 in the first format and the scaling factor 1. The value of this product is within the numerical range that the data in the second format can represent. At this point, the data format of the product is still the first format. Then, compute node 101 can cast this product into the data 2 in the second format with lower precision. For example, this can be achieved by rounding down the high-order exponent and truncating the low-order mantissa.

[0074] For example, suppose the calculated product is represented in FP16 format, as shown in the upper part of Figure 3, which includes a sign bit represented by 1 bit, an exponent bit represented by 8 bits, and a mantissa bit represented by 7 bits. Suppose the FP16 representation of this product is specifically "0 00000111 0110010". Computation node 101 can convert the FP16 format product into FP8 format data, as shown in the lower part of Figure 3, which includes a sign bit represented by 1 bit, an exponent bit represented by 4 bits, and a mantissa bit represented by 3 bits. Therefore, when converting a product in FP16 format to FP8 format data, compute node 101 can retain the lower 4 bits of the 8-bit exponent and erase the higher 4 bits of the exponent, that is, only retain the "0111" in the 8-bit exponent "00000111". Furthermore, compute node 101 can also truncate the 7-bit mantissa in the FP16 format to 3 bits, that is, only retain the first 3 bits "011" in the 7-bit mantissa "0110010". In this way, the FP16 format product "0 00000111 0110010" is converted into the FP8 format data "0 0111 011".

[0075] In practical applications, computing node 101 can also convert data 1 in the first format into data 2 in the second format in other ways, without any limitation.

[0076] S202: Computing node 101 sends scaling factor 1 and second format data 2 to computing node 102.

[0077] In this embodiment, computing node 101 sends not only low-precision data 2 to computing node 102, but also scaling factor 1. Typically, the second-format data 2 is a tensor containing a large number of values, while scaling factor 1 is a single value. Therefore, the additional communication overhead incurred by computing node 101 sending scaling factor 1 to computing node 102 is negligible.

[0078] S203: During the execution of the communication operator in the AI ​​task, the computing node 103 uses the scaling factor 2 to convert the first format data 3 into the second format data 4. The data 3 is the data generated when the computing node 101 executes the sub-task in the AI ​​task.

[0079] S204: Computing node 103 sends scaling factor 2 and second format data 4 to computing node 102.

[0080] The process of computing node 103 determining scaling factor 2 and converting the format of data 3 is similar to the process of computing node 101 determining scaling factor 1 and converting the format of data 1. Therefore, the implementation of steps S203 to S204 can be found in the relevant descriptions of steps S201 to S202 above, and will not be repeated here.

[0081] It is worth noting that, since the values ​​of the first format data calculated on different computing nodes may vary significantly, the scaling factor 1 used by computing node 101 for data format conversion may differ from the scaling factor 2 used by computing node 103 for data format conversion. Therefore, computing nodes 101 and 103 each provide their determined scaling factors to computing node 102, enabling computing node 102 to determine the optimal scaling factor for each computing node when performing data format conversion.

[0082] S205: During the execution of the communication operator, the computing node 102 converts the second format data 2 into the first format data 5 according to the received scaling factor 1, and converts the second format data 4 into the first format data 6 according to the received scaling factor 2.

[0083] In this embodiment, for the low-precision data sent by each computing node, computing node 102 can restore the low-precision data to high-precision data according to the scaling factor sent by the computing node, so as to use the high-precision data to perform the vector calculation process in the subtask, thereby ensuring the accuracy of the computing cluster 100 in performing AI tasks.

[0084] In a specific implementation, for the second format data 2 (low precision) sent by computing node 101, computing node 102 can use the scaling factor 1 sent by computing node 101 to convert data 2 into (un-cast) the first format data 5 (high precision).

[0085] For example, computing node 102 can pad the exponent and mantissa in the data of the first format with zeros according to the second format to obtain the data of the first format. For example, suppose the data 2 of the second format is specifically the data "0 0111 011" in FP8 format, and the first format is FP16 format. Since FP16 can include 8 bits of exponent and 7 bits of mantissa, computing node 102 can pad the data "0 0111 011" in FP8 format to 8 bits of exponent and 3 bits of mantissa to 7 bits of mantissa by padding with zeros, and the resulting data of the first format is "0 00000111 0110000".

[0086] Then, compute node 101 can amplify the zero-padded first-format data based on scaling factor 1 to obtain the aforementioned first-format data 5, thereby completing the conversion of the second-format data 2. For example, compute node 101 can multiply the zero-padded first-format data with the reciprocal of scaling factor 1, and the resulting product is the first-format data 5.

[0087] Similarly, for the second format data 4 sent by computing node 103, computing node 102 can also use the scaling factor 2 sent by computing node 102 to convert the second format data 4 into the first format data 6. The specific implementation method can be found in the above description of the implementation method for converting to obtain the first format data 5, and will not be repeated here.

[0088] It is worth noting that, for data from different computing nodes, computing node 102 does not use a uniform scaling factor to convert the second-format data sent by each computing node. Instead, it converts the second-format data sent by each computing node according to the scaling factor sent by that computing node. This makes the deviation between the first-format data converted by computing node 102 based on the received scaling factor and the first-format data in other computing nodes generally smaller. That is, the deviation between the first-format data 5 converted by computing node 102 using scaling factor 1 and the first-format data 1 in computing node 101 is small; the deviation between the first-format data 6 converted by computing node 102 using scaling factor 1 and the first-format data 3 in computing node 101 is small. In this way, the accuracy of the high-precision first-format data restored by computing node 102 can be improved, that is, the data precision loss caused by converting data from the first format to the second format and then from the second format to the first format can be minimized. As a result, computing node 102 can perform subsequent vector calculations more accurately, which is beneficial to improving the accuracy of AI task execution.

[0089] S206: Computation node 102 performs vector calculations based on data 5 and data 6 in the first format.

[0090] Typically, computing node 102 also generates data in a first format, hereinafter referred to as data 7. Therefore, computing node 102 can perform vector calculations based on data 5, data 6, and its own generated data 7. Furthermore, the processing logic indicated by the communication operator also includes that when computing node 102 performs reduction or other processing on the received data, it can further process data 5, data 6, and data 7 before performing vector calculations, and then perform vector calculations based on the processing results. For example, during distributed training of an AI model, multiple computing nodes can perform data interaction and reduction processing according to the logic indicated by the communication operator, and then computing node 102 can perform vector calculations based on the reduction results.

[0091] For example, the communication operator executed by multiple computing nodes can be, for instance, the ReduceScatter operator. Taking computing nodes 101 to 103 participating in the execution of the ReduceScatter operator as an example, the data processing and processing logic indicated by the ReduceScatter operator is shown in Figure 4. Each computing node sends data of a portion of its computation results to other computing nodes, while simultaneously receiving data of other dimensions from other computing nodes. Thus, after completing the data interaction, the communication results obtained by each computing node are shown in Figure 4. Specifically, computing node 102 can sum data 5, data 6, and data 7, and the calculated sum is used to perform subsequent vector calculations.

[0092] In practical applications, during the execution of the ReduceScatter operator, compute node 102 can merge the multiplication operation (i.e., multiplying with the reciprocal of the scaling factor) in the process of format conversion of the second format data with the addition operation for the data 7 in compute node 102. Specifically, it can be merged into a multiply-accumulate operation, which can improve data computing efficiency and reduce the consumption of IO resources.

[0093] In a specific implementation, the computing node 102 can calculate the sum of the data in the first format and the data in the second format by performing a multiplication-accumulation operation based on the scaling factor 1, the data in the second format 2, the scaling factor 2, and the data in the second format 5. Thus, the computing node 102 can perform subsequent vector calculations based on this sum.

[0094] In addition, the computing node 102 can further improve the efficiency of its subtasks by executing communication and computation processes in parallel.

[0095] Specifically, as shown in Figure 5, after receiving the second-format data 2 sent by computing node 101, computing node 102 converts the second-format data 2 into the first-format data 5 while simultaneously receiving the second-format data 4 sent by computing node 103. Thus, the computation process of computing node 102 performing format conversion on data 2 can be masked during the communication process of computing node 102 receiving the data 4 sent by computing node 103. This avoids the computation time generated by format conversion of data 2 from affecting the overall efficiency of executing the AI ​​task.

[0096] In a further possible implementation, during the distributed training of the AI ​​model, computing node 102 may execute multiple communication operators, such as the AllReduce operator and the ReduceScatter operator. After executing some communication operators, computing node 102 may perform vector computation. For example, after executing the ReduceScatter operator, computing node 102 may continue to execute network layers such as Dropout, residual layers, or LayerNorm in the AI ​​model, and perform vector computation during the execution of these network layers. Since the data precision during vector computation typically has a high impact on the training accuracy of the AI ​​model, computing node 102 can, based on the above method, convert the received second-format data 2 and data 5 into first-format data 3 and data 6 before performing subsequent vector computation, thereby ensuring the data precision during vector computation and thus guaranteeing the overall accuracy of the AI ​​task executed by the computing cluster 100.

[0097] For example, computation node 102 can aggregate data 5, data 6, and data 7, such as aggregating three 256-dimensional tensor data into a 768-dimensional tensor data, or it can calculate the sum of data 5, data 6, and data 7, as shown in Figure 4. Then, the second computation node 102 can perform vector calculations based on the aggregation result or the sum, such as performing corresponding vector calculations in the random deactivation layer based on the aggregation result or the sum.

[0098] Furthermore, during the execution of subtasks in the AI ​​task, computing node 102 may also execute another part of the communication operators. After executing this other part of the communication operators, computing node 102 may perform matrix calculations. For example, in a model training scenario, after executing the AllReduce operator, computing node 102 may continue to execute the self-attention layer in the AI ​​model, and perform matrix calculations during the execution of this network layer. Since the data precision during matrix calculations usually has a low impact on the training precision of the AI ​​model, computing node 102, upon receiving data in the second format, can directly perform subsequent matrix calculation operations based on the second format data without performing format conversion. This not only reduces the computational power required for matrix calculations, but also improves the efficiency of matrix calculations by having computing node 102 perform matrix calculations based on low-precision, small-volume data. Meanwhile, since the data precision during matrix calculation usually has a low impact on the training precision of AI models, the precision of the AI ​​model trained by computing node 102 after performing matrix calculations based on the second format data and then performing subsequent AI model training can still reach a high level.

[0099] To facilitate understanding, the following section, with reference to Figure 6, describes the data processing process when computing node 102 performs matrix calculations based on low-precision data, and uses the parallel execution of AI tasks by computing nodes 101 to 103 as an example for illustration.

[0100] As shown in Figure 6, the data processing procedure may specifically include the following steps.

[0101] S601: For the data 8 in the first format, the compute node 101 determines a local scaling factor 1. The local scaling factor 1 is used to convert the data in the first format into data in the second format, wherein the precision of the data in the first format is higher than that of the data in the second format. The data 8 in the first format is the data generated when the compute node 101 executes a subtask in the AI ​​task.

[0102] S602: Compute node 102 determines a local scaling factor 2 for the first format data 9. The local scaling factor 2 is used to convert the first format data into a second format data, wherein the first format data 9 is the data generated when compute node 102 executes a subtask in the AI ​​task.

[0103] S603: Compute node 103 determines a local scaling factor 3 for the data 10 in the first format. The local scaling factor 3 is used to convert the data in the first format into data in the second format. The data 10 in the first format is the data generated when compute node 103 executes a subtask in the AI ​​task.

[0104] Each computing node determines the implementation method of the local scaling factor used for data format conversion for the first format data. For details on determining scaling factor 1 in the above embodiment of Figure 2, it can be referred to, and will not be repeated here.

[0105] S604: Each compute node from compute node 101 to compute node 103 synchronizes its local scaling factor to the other compute nodes.

[0106] In this way, by synchronizing the local scaling factors among multiple compute nodes, each compute node can obtain the local scaling factors of the three compute nodes.

[0107] S605: Each compute node determines a global scaling factor from multiple local scaling factors, which is the minimum value among the multiple local scaling factors.

[0108] Each compute node can compare the values ​​of local scaling factor 1, local scaling factor 2, and local scaling factor 3 to determine the smallest local scaling factor, which is then designated as the global scaling factor. The local scaling factor in each compute node is used to reduce the values ​​of data in the first format to the range that can be represented by data in the second format. Therefore, the smaller the local scaling factor, the stronger its ability to reduce the values ​​of data in the first format, meaning it can reduce larger values ​​to the range that can be represented by data in the second format. This ensures that after subsequently reducing the values ​​of the first format data in each compute node using the global scaling factor, the reduced values ​​in each compute node will remain within the range that can be represented by data in the second format.

[0109] S606: During the execution of communication operators in AI tasks, each computing node uses a global scaling factor to convert the data in the first format to obtain the data in the second format.

[0110] For example, the second communication operator could be, for instance, the AllReduce operator, etc., without limitation.

[0111] Specifically, computing node 101 can use a global scaling factor to convert data 8 in the first format to obtain data 11 in the second format; computing node 101 can use a global scaling factor to convert data 9 in the first format to obtain data 12 in the second format; and computing node 103 can use a global scaling factor to convert data 10 in the first format to obtain data 13 in the second format.

[0112] S607: Compute node 101 sends data 11 in the second format to compute node 102, and compute node 103 sends data 13 in the second format to compute node 102.

[0113] In this embodiment, since multiple computing nodes use a unified global scaling factor to convert the data in the first format, when computing nodes 101 and 103 provide data in the second format to computing node 102, they do not need to send all scaling factors to computing node 102.

[0114] S608: Computation node 102 performs matrix calculations based on data 11, data 12 and data 13 in the second format.

[0115] In this way, multiple computing nodes interact based on low-precision second-format data, which can effectively reduce the amount of communication data between different computing nodes, reduce communication costs, and improve communication efficiency. Moreover, it can also reduce the computing power consumption required for computing node 102 to perform matrix calculations and improve computing efficiency. At the same time, in model training scenarios, after computing node 102 performs matrix calculations based on low-precision data, the training accuracy of the AI ​​model can still be maintained at a high level.

[0116] In practical applications, when multiple computing nodes in a computing cluster 100 execute AI tasks in parallel, for different communication operators, the computing nodes can communicate based on low-precision data and perform matrix calculations based on low-precision data, and vector calculations based on high-precision data. For example, as shown in Figure 7, when executing the AllGather operator, different computing nodes can process data based on low-precision formats such as HiF8, and the computing nodes can perform matrix calculations in the self-attention layer based on low-precision data. When executing the ReduceScatter operator, different computing nodes can process data based on low-precision formats such as HiF8, but before performing vector calculations in the random deactivation layer, the computing nodes can first convert the low-precision data into high-precision formats such as FP16, and then perform vector calculations based on the high-precision data. In this way, the communication cost generated during distributed training of AI models can be reduced by more than 50%, the communication latency can be reduced by more than 40%, and the overall training performance of the AI ​​model can be improved by more than 50%. The overall training performance of an AI model can be measured using at least one of the following metrics: training time, computing power consumption, and communication cost, or it can be measured using other types of metrics.

[0117] It is worth noting that the embodiments shown in Figures 2 and 6 above are only illustrative examples. In actual applications, other methods can be used to implement distributed training of AI models based on the above embodiments. Exemplary examples are provided below.

[0118] In the first implementation example, the embodiments shown in Figures 2 and 6 above mainly describe the process by which computing node 102 performs vector calculations or matrix calculations using low-precision data sent by other computing nodes. In actual application scenarios, during the parallel execution of AI tasks using computing nodes 101 to 103, computing node 102 can also send low-precision data to computing node 101 and / or computing node 103. Thus, computing node 101 and computing node 103 can perform vector calculations or matrix calculations based on the received low-precision data. The specific implementation process is similar to the implementation process of computing node 102 performing vector calculations or matrix calculations based on the received low-precision data, and will not be elaborated here.

[0119] In the second implementation example, in the embodiments shown in Figures 2 and 6 above, the communication operator executed by the computing node 102 can be, in addition to the AllGather operator and the ReduceScatter operator, one or any combination of the AllReduce operator, AllToAll operator, and Broadcast operator, or other types of communication operators, without limitation.

[0120] In the third implementation example, the embodiments shown in Figures 2 and 6 above are illustrated by using three computing nodes to execute AI tasks in parallel. In other embodiments, AI tasks can also be executed in parallel using only computing nodes 101 and 102, or a larger number of computing nodes can be used to execute AI tasks in parallel.

[0121] It is worth noting that other reasonable combinations of steps that can be conceived by those skilled in the art based on the above description also fall within the scope of protection of this application. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.

[0122] The data processing method provided by the embodiments of this application has been described above with reference to Figures 1 to 7. Next, the structure of the computing node and computing device provided by the embodiments of this application will be described with reference to the accompanying drawings.

[0123] Referring to Figure 8, a schematic diagram of a computing node structure is shown. The computing node 800 shown in Figure 8 is applied to a computing cluster, which includes multiple computing nodes, including a first computing node, and the computing node 800 is one of the multiple computing nodes, which execute AI tasks in parallel.

[0124] As shown in Figure 8, computing node 800 includes:

[0125] The communication module 801 is used to receive a first scaling factor and second data in a second format from the first computing node. The first scaling factor is used to convert the first data in the first format into the second data in the second format. The precision of the data in the first format is higher than that of the data in the second format. The first data is the data generated when the first computing node executes a subtask in the AI ​​task.

[0126] The conversion module 802 is used to convert second data in a second format into third data in a first format using a first scaling factor during the execution of the first communication operator in the AI ​​task.

[0127] The calculation module 803 is used to perform vector calculations based on third data in a first format.

[0128] In one possible implementation, the communication module 801 is further configured to receive a second scaling factor from the third computing node and fifth data in a second format, wherein the second scaling factor is used to convert the fourth data in the first format into the fifth data in the second format, and the fourth data is data generated when the third computing node executes a subtask in the AI ​​task.

[0129] The calculation module 803 is used to perform vector calculations based on the third data in the first format and the sixth data in the first format, wherein the sixth data is obtained by converting the fifth data in the second format using a second scaling factor.

[0130] In one possible implementation, the conversion module 802 is configured to: convert the second data in the second format into the third data in the first format using the first scaling factor during the process of the communication module receiving the second scaling factor and the fifth data.

[0131] In one possible implementation, the conversion module 802 is configured to: perform a multiply-accumulate operation based on the first scaling factor, the second data, the second scaling factor, and the fifth data to obtain the sum of the third data and the sixth data;

[0132] Calculation module 803 is used to perform vector calculations based on sum values.

[0133] In one possible implementation, the AI ​​task is a training or inference task for an AI model based on a transform architecture, and the first communication operator is a reduction hash operator.

[0134] In one possible implementation, the AI ​​task is also executed by a third computing node;

[0135] The communication module 801 is also used to receive eighth data in a second format from the first computing node, wherein the eighth data in the second format is obtained by converting the seventh data in the first format using a global scaling factor. The seventh data is the data generated when the first computing node executes a subtask in the AI ​​task. The global scaling factor is the smaller value between a first local scaling factor and a second local scaling factor. The first local scaling factor is determined based on the seventh data in the first format. The second local scaling factor is used to convert the data required by the second communication operator in the AI ​​task executed in the third computing node from the first format to the second format.

[0136] The calculation module 803 is also used to perform matrix calculations based on the eighth data in the second format.

[0137] In one possible implementation, the AI ​​task is a training or inference task for an AI model based on a transform architecture, and the second communication operator is a full aggregation operator.

[0138] Since the computing node 800 shown in Figure 8 corresponds to the computing node 102 in the embodiments shown in Figures 2 and 6, the specific implementation of the computing node 800 shown in Figure 8 and its technical effects can be found in the relevant descriptions in the embodiments shown in Figures 2 and 6, and will not be repeated here.

[0139] Figure 9 is a schematic diagram of a computing device provided in this application. As shown in Figure 9, the computing device 900 includes a processor 901, a memory 902, a communication interface 903, and a bus 904. The processor 901, memory 902, and communication interface 903 communicate with each other via the bus 904. The bus 904 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in Figure 9, but this does not mean that there is only one bus or one type of bus. The communication interface 903 is used for communication with external devices, such as receiving scaling factors and data sent by other computing devices.

[0140] It should be understood that in the embodiments of this application, processor 901 may be a CPU, or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete device assemblies, etc. General-purpose processors may be microprocessors or any conventional processors, etc.

[0141] The memory 902 may include read-only memory and random access memory, and provides instructions and data to the processor 901. The memory 902 may also include non-volatile random access memory. For example, the memory 902 may also store device type information.

[0142] The memory 902 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0143] The memory 902 stores executable code, and the processor 901 executes the executable code to perform the method performed by the computing node 102 in the embodiments shown in FIG2 and FIG6.

[0144] It should be understood that the computing device 900 according to the embodiments of this application may correspond to the computing node 102 in the embodiments of this application, and may correspond to the method executed by the computing node 102 in the methods shown in FIG2 and FIG6 in the embodiments of this application. The above and other operations and / or functions implemented by the computing device 900 are respectively to implement the flow of the corresponding methods in FIG2 and FIG6. For the sake of brevity, they will not be described in detail here.

[0145] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the aforementioned data processing method.

[0146] This application also provides a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application are generated.

[0147] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0148] The computer program product can be a software installation package. When any of the aforementioned data processing methods is required, the computer program product can be downloaded and executed on a computing device.

[0149] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0150] The terminology used in the above embodiments is for the purpose of describing specific embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the embodiments of this application, “one or more” refers to one, two, or more; the character “ / ” generally indicates that the preceding and following objects are in an “or” relationship. In the embodiments of this application, “simultaneously” means within the same time period, including situations where they are at the same moment.

[0151] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0152] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The method is applied to a computing cluster, which includes multiple computing nodes that execute artificial intelligence (AI) tasks in parallel. The multiple computing nodes include a first computing node and a second computing node. The method includes: During the execution of the first communication operator in the AI ​​task, the first computing node uses a first scaling factor to convert the first data in the first format into the second data in the second format. The precision of the data in the first format is higher than that of the data in the second format. The first data is the data generated by the first computing node when executing the sub-task in the AI ​​task. The first computing node sends the first scaling factor and the second data in the second format to the second computing node; During the execution of the first communication operator, the second computing node uses the first scaling factor to convert the second data in the second format into the third data in the first format. The second computing node performs vector calculations based on the third data in the first format.

2. The method according to claim 1, characterized in that, The plurality of computing nodes further includes a third computing node, and the method further includes: During the execution of the first communication operator, the third computing node uses a second scaling factor to convert the fourth data in the first format into the fifth data in the second format. The fourth data is the data generated when the third computing node executes the sub-task in the AI ​​task. The third computing node sends the second scaling factor and the fifth data in the second format to the second computing node; The second computing node receives the second scaling factor and the fifth data in the second format; The second computing node performs vector calculations based on the third data in the first format, including: The second computing node performs the vector calculation based on the third data in the first format and the sixth data in the first format, wherein the sixth data is obtained by converting the fifth data in the second format using the second scaling factor.

3. The method according to claim 2, characterized in that, The step of converting the second data in the second format into the third data in the first format using the first scaling factor includes: During the process of receiving the second scaling factor and the fifth data, the second computing node uses the first scaling factor to convert the second data in the second format into the third data in the first format.

4. The method according to claim 2 or 3, characterized in that, The step of converting the second data in the second format into the third data in the first format using the first scaling factor includes: The second computing node performs a multiply-accumulate operation based on the first scaling factor, the second data, the second scaling factor, and the fifth data to obtain the sum of the third data and the sixth data. The second computing node performs the vector calculation based on the third data in the first format and the sixth data in the first format, including: The second computing node performs the vector calculation based on the sum value.

5. The method according to any one of claims 1 to 4, characterized in that, The AI ​​task is a training or inference task for an AI model based on a transform architecture, and the first communication operator is a reduction hash operator.

6. The method according to any one of claims 1 to 5, characterized in that, The computing cluster further includes a third computing node, and the method further includes: The first computing node determines a first local scaling factor for the seventh data in the first format, wherein the seventh data is the data generated when the first computing node executes a subtask in the AI ​​task; The first computing node obtains the second local scaling factor of the third computing node, and the second local scaling factor is used to convert the data required by the third computing node to execute the second communication operator in the AI ​​task from the first format to the second format. The first computing node determines the global scaling factor, which is the smaller value between the first local scaling factor and the second local scaling factor; During the execution of the second communication operator, the first computing node uses the global scaling factor to convert the seventh data in the first format into the eighth data in the second format; The first computing node sends the eighth data in the second format to the second computing node; The second computing node performs matrix calculations based on the eighth data in the second format.

7. The method according to claim 6, characterized in that, The AI ​​task is a training or inference task for an AI model based on a transform architecture, and the second communication operator is a full aggregation operator.

8. A computing cluster, characterized in that, The computing cluster includes multiple computing nodes, which execute artificial intelligence (AI) tasks in parallel. The multiple computing nodes include a first computing node and a second computing node. The first computing node is used to convert first data in a first format into second data in a second format using a first scaling factor during the execution of the first communication operator in the AI ​​task. The precision of the first format data is higher than that of the second format data. The first data is the data generated by the first computing node when executing the sub-task in the AI ​​task. Send the first scaling factor and the second data in the second format to the second computing node; The second computing node is used to convert the second data in the second format into the third data in the first format using the first scaling factor during the execution of the first communication operator. Perform vector calculations based on the third data in the first format.

9. The computing cluster according to claim 8, characterized in that, The plurality of computing nodes also includes a third computing node; The third computing node is used to convert the fourth data in the first format into the fifth data in the second format using a second scaling factor during the execution of the first communication operator. The fourth data is the data generated by the third computing node when executing the sub-task in the AI ​​task. Send the second scaling factor and the fifth data in the second format to the second computing node; The second computing node is also configured to receive the second scaling factor and the fifth data in the second format; The second computing node is configured to: perform the vector calculation based on the third data in the first format and the sixth data in the first format, wherein the sixth data is obtained by converting the fifth data in the second format using the second scaling factor.

10. The computing cluster according to claim 9, characterized in that, The second computing node is configured to: during the process of receiving the second scaling factor and the fifth data, use the first scaling factor to convert the second data in the second format into the third data in the first format.

11. The computing cluster according to claim 9 or 10, characterized in that, The second computing node is used for: Based on the first scaling factor, the second data, the second scaling factor, and the fifth data, perform a multiply-accumulate operation to obtain the sum of the third data and the sixth data; The vector calculation is performed based on the sum value.

12. The computing cluster according to any one of claims 8 to 11, characterized in that, The AI ​​task is a training or inference task for an AI model based on a transform architecture, and the first communication operator is a reduction hash operator.

13. The computing cluster according to any one of claims 8 to 12, characterized in that, The computing cluster also includes a third computing node; The first computing node is further configured to: For the seventh data in the first format, a first local scaling factor is determined, wherein the seventh data is the data generated when the first computing node executes a subtask in the AI ​​task; Obtain the second local scaling factor of the third computing node, the second local scaling factor being used to convert the data required by the second communication operator in the AI ​​task in the third computing node from the first format to the second format; Determine the global scaling factor, which is the smaller value between the first local scaling factor and the second local scaling factor; During the execution of the second communication operator, the seventh data in the first format is converted into the eighth data in the second format using the global scaling factor; Send the eighth data in the second format to the second computing node; The second computing node is also used to perform matrix calculations based on the eighth data in the second format.

14. The computing cluster according to claim 13, characterized in that, The AI ​​task is a training or inference task for an AI model based on a transform architecture, and the second communication operator is a full aggregation operator.

15. A data processing method, characterized in that, The method is applied to a second computing node, which executes artificial intelligence (AI) tasks in parallel with the first computing node. The method includes: The system receives a first scaling factor from the first computing node and second data in a second format. The first scaling factor is used to convert the first data in the first format into the second data in the second format. The precision of the data in the first format is higher than that of the data in the second format. The first data is the data generated by the first computing node when it executes a subtask in the AI ​​task. During the execution of the first communication operator in the AI ​​task, the second data in the second format is converted into the third data in the first format using the first scaling factor; Perform vector calculations based on the third data in the first format.

16. The computing node according to claim 15, characterized in that, The method further includes: The system receives a second scaling factor from a third computing node and fifth data in the second format. The second scaling factor is used to convert fourth data in the first format into fifth data in the second format. The fourth data is data generated when the third computing node executes a subtask in the AI ​​task. The vector calculation based on the third data in the first format includes: The vector calculation is performed based on the third data in the first format and the sixth data in the first format, wherein the sixth data is obtained by converting the fifth data in the second format using the second scaling factor.

17. The computing node according to claim 15 or 16, characterized in that, The step of converting the second data in the second format into the third data in the first format using the first scaling factor includes: During the process of receiving the second scaling factor and the fifth data, the second data in the second format is converted into the third data in the first format using the first scaling factor.

18. A computing device, characterized in that, The computing device includes a processor and a memory; The processor is configured to execute instructions stored in the memory to cause the computing device to perform the steps of the method as described in any one of claims 15 to 17.

Images