Data processing method based on accelerator, and related apparatus

Abstract

A data processing method based on an accelerator, which method is applied to the technical field of artificial intelligence (AI). In the data processing method based on an accelerator, when an AI accelerator is used to execute operations indicated by a model, on the basis of the characteristics of the AI accelerator comprising computing units for executing different types of operations, a first computing unit of the AI accelerator first executes a first operation on part of input data to obtain output data. In this way, when the first computing unit continues to execute the first operation on the other part of input data, a second computing unit of the AI accelerator can simultaneously execute a second operation on the output data of the first computing unit, thus ensuring that the first computing unit and the second computing unit execute operations simultaneously, thereby reducing the overall latency of the AI accelerator in executing operations.

Description

An accelerator-based data processing method and related apparatus

[0001] This application claims priority to Chinese Patent Application No. 202411586539.X, filed on November 7, 2024, entitled "A Data Processing Method and Related Apparatus Based on an Accelerator", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI) technology, and in particular to an accelerator-based data processing method and related apparatus. Background Technology

[0003] Currently, deep learning models have become a hot topic in the field of artificial intelligence. Among them, model architectures represented by large language models are considered a potential way to achieve general artificial intelligence, and have been widely used in natural language generation, speech translation, intelligent customer service, and intelligent assistants.

[0004] During the operation of a deep learning model, the input data needs to undergo a series of sequential operator calculations to obtain the model's prediction result. For sequentially connected operators in a deep learning model, the input data of the subsequent operator is the output data of the previous operator. Therefore, the subsequent operator often needs to wait for the previous operator to complete its calculation before it can begin its own calculation.

[0005] Since deep learning models are primarily composed of a series of sequential operators, their computational latency is often closely related to their scale. As the scale of deep learning models increases, while model performance improves, computational latency also increases. Therefore, reducing the computational latency of deep learning models based on existing architectures has become an urgent problem to be solved. Summary of the Invention

[0006] This application provides an accelerator-based data processing method that can improve the overall efficiency of AI accelerators in performing computations.

[0007] Firstly, an accelerator-based data processing method is provided, which is applied to an AI accelerator. The AI ​​accelerator includes a first computing unit and a second computing unit, which are used to perform different types of operations.

[0008] The accelerator-based data processing method includes: a first computing unit performing a first operation on first data to obtain second data, wherein the first data is a portion of target data, the target data is the data for which the first operation needs to be performed, and the data obtained after performing the first operation on the target data needs to undergo the second operation. Specifically, the target data can be input data of an AI model or output data of a certain neural network layer in the AI ​​model.

[0009] While the first computing unit performs the first operation on the third data, the second computing unit performs the second operation on the second data, where the third data is a portion of the target data. That is, the second computing unit does not need to wait for the first computing unit to complete the first operation on all data in the target data before starting its work; instead, it can begin working as soon as the first computing unit completes the first operation on a portion of the data, thereby minimizing the waiting time for the first computing unit.

[0010] In this solution, when using an AI accelerator to execute the operations indicated by the model, the AI ​​accelerator leverages its ability to have computing units for performing different types of operations. The first computing unit on the AI ​​accelerator performs a first operation on a portion of the input data to obtain output data. Thus, while the first computing unit continues to perform the first operation on the remaining input data, the second computing unit on the AI ​​accelerator can simultaneously perform a second operation on the output data of the first computing unit. This ensures that both the first and second computing units execute operations concurrently, reducing the overall latency of the AI ​​accelerator's operations.

[0011] Specifically, for two sequential operators with different operation types in a deep learning model, this solution segments the input data of the operators while maintaining the data dependency between them. This allows both the first and second computational units in the AI ​​accelerator to execute the operators multiple times to process the segmented data. In this way, after the first computational unit executes the previous operator and obtains the output data, the second computational unit can execute the next operator based on the output data of the previous operator. This ensures that the next operator can be executed as early as possible while the previous operator is being executed, avoiding the need for the second computational unit to wait for the first computational unit to finish running the previous operator before it can start running the next operator. This effectively improves the overall efficiency of the AI ​​accelerator's computation.

[0012] In one possible implementation, the first computing unit stores the second data in a target cache; the second computing unit reads the second data from the target cache. The target cache is a cache shared by the first and second computing units.

[0013] That is, the first computing unit and the second computing unit exchange data through the target cache.

[0014] In one possible implementation, the first computing unit and the second computing unit share multiple caches, with the target cache being the cache with the highest read / write speed among these caches. That is, the target cache used to transfer the second data between the first and second computing units is the cache with the highest read / write speed in the AI ​​accelerator.

[0015] In this scheme, the first computing unit outputs the processed data to the target cache with the highest read / write speed after processing each part of the data. The second computing unit then reads the data output by the first computing unit from the target cache in a timely manner to perform the operation. This ensures that the second computing unit can always read the data that needs to be operated from the target cache with the highest read / write speed, saving the second computing unit the time of data transfer and improving the overall efficiency of the AI ​​accelerator.

[0016] In one possible implementation, the first operation includes tensor operations and the second operation includes vector operations; or, the first operation includes vector operations and the second operation includes tensor operations.

[0017] In one possible implementation, the first operation is matrix multiplication, and the target data includes a first matrix and a second matrix. The first data includes a first submatrix obtained by partitioning the first matrix and a second submatrix obtained by partitioning the second matrix.

[0018] That is, the first submatrix is ​​obtained by dividing the first matrix, and the second submatrix is ​​obtained by dividing the second matrix.

[0019] In one possible implementation, the method of partitioning the first and second matrices is related to the size of the first matrix, the size of the second matrix, and the size of the cache space in the first computing unit. Specifically, the cache space in the first computing unit determines the amount of matrix data that the first computing unit can load at one time, while the size of the first and second matrices determines the amount of matrix data that needs to be processed.

[0020] In practical applications, the execution device can pre-determine the partitioning method of the first matrix and the second matrix based on the size of the first matrix, the size of the second matrix, and the size of the cache space in the first computing unit, so that the first computing unit can minimize the amount of data transfer when performing operations on the partitioned submatrices.

[0021] In one possible implementation, when the first and second matrices are partitioned using the first partitioning method, the number of rows in the first submatrix is ​​less than the number of rows in the first matrix, and the number of columns in the first submatrix is ​​less than the number of columns in the first matrix. Similarly, the number of rows in the second submatrix is ​​less than the number of rows in the second matrix, and the number of columns in the second submatrix is ​​less than the number of columns in the second matrix.

[0022] In one possible implementation, where the first and second matrices are partitioned using the second partitioning method, the first submatrix has fewer rows than the first matrix, and the first submatrix has the same number of columns as the first matrix. Similarly, the second submatrix has the same number of rows as the second matrix, and the second submatrix has fewer columns than the second matrix.

[0023] In one possible implementation, the first data comprises multiple sub-data. The first computing unit performs a first operation on the first data, specifically including: the first computing unit sequentially performs the first operation on the multiple sub-data to obtain data after the multi-part operation, and the data after the multi-part operation is used to constitute the second data.

[0024] That is, the second computing unit performs only one round of calculations after the first computing unit has performed multiple rounds of calculations.

[0025] In one possible implementation, the second data is obtained by the first computing unit performing a single first operation, and the second data includes multiple sub-data. The second computing unit performs a second operation on the second data, specifically including: the second computing unit sequentially performs the second operation on the multiple sub-data to obtain data after multiple operations, and the data after multiple operations is used to constitute the data obtained after performing the second operation on the second data.

[0026] That is, for every round of calculation performed by the first computing unit, the second computing unit needs to continue performing multiple rounds of calculation.

[0027] Secondly, an AI accelerator is provided, comprising a first computing unit and a second computing unit, which are used to perform different types of operations; the first computing unit is used to perform a first operation on first data to obtain second data, wherein the first data is a portion of target data, the target data is the data for which the first operation needs to be performed, and the data obtained after performing the first operation on the target data needs to be subjected to a second operation; while the first computing unit performs the first operation on third data, the second computing unit is used to perform the second operation on the second data, wherein the third data is a portion of the target data.

[0028] In one possible implementation, the first computing unit is further configured to store the second data into a target cache; the second computing unit is further configured to read the second data from the target cache; wherein the target cache is a cache shared by the first computing unit and the second computing unit.

[0029] In one possible implementation, the first computing unit and the second computing unit share multiple caches, with the target cache being the cache with the highest read / write speed among the multiple caches.

[0030] In one possible implementation, the first operation includes tensor operations and the second operation includes vector operations; or, the first operation includes vector operations and the second operation includes tensor operations.

[0031] In one possible implementation, the first operation is a matrix multiplication operation, and the target data includes a first matrix and a second matrix; the first data includes a first submatrix obtained by partitioning the first matrix and a second submatrix obtained by partitioning the second matrix.

[0032] In one possible implementation, the way the first matrix and the second matrix are split is related to the size of the first matrix, the size of the second matrix, and the size of the cache space in the first computing unit.

[0033] In one possible implementation, when the first matrix and the second matrix are partitioned in the first partitioning manner, the number of rows in the first submatrix is ​​less than the number of rows in the first matrix, and the number of columns in the first submatrix is ​​less than the number of columns in the first matrix; the number of rows in the second submatrix is ​​less than the number of rows in the second matrix, and the number of columns in the second submatrix is ​​less than the number of columns in the second matrix.

[0034] In one possible implementation, when the first matrix and the second matrix are partitioned in the second way, the number of rows in the first submatrix is ​​less than the number of rows in the first matrix, and the number of columns in the first submatrix is ​​equal to the number of columns in the first matrix; the number of rows in the second submatrix is ​​equal to the number of rows in the second matrix, and the number of columns in the second submatrix is ​​less than the number of columns in the second matrix.

[0035] In one possible implementation, the first data includes multiple sub-data; the first computing unit is specifically used to perform a first operation on the multiple sub-data in sequence to obtain data after the multiple operations, and the data after the multiple operations is used to constitute the second data.

[0036] In one possible implementation, the second data is obtained by the first computing unit performing a single first operation, and the second data includes multiple sub-data. The second computing unit is specifically used to perform a second operation on the multiple sub-data in sequence to obtain data after multiple operations, and the data after multiple operations is used to constitute the data obtained after performing the second operation on the second data.

[0037] Thirdly, a chip system is provided, the chip system including an AI accelerator and a communication interface for communicating with a module outside the chip, the AI ​​accelerator for running computer programs or instructions, such that a device on which the chip system is installed can perform the methods of any of the above aspects.

[0038] Fourthly, a computing device is provided, the computing device including the AI ​​accelerator of the second aspect, the AI ​​accelerator in the computing device being used to implement the operational steps of the method of any of the above aspects.

[0039] Fifthly, a computing device cluster is provided, comprising at least one computing device, wherein any one computing device is used to run a computer program or instructions, such that the computing device cluster can perform the methods described in any of the above aspects. Alternatively, some or all of the computing devices are used together to run a computer program or instructions, such that the computing device cluster can perform the methods described in any of the above aspects.

[0040] In a sixth aspect, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the methods of any of the above aspects.

[0041] In a seventh aspect, a computer program product containing instructions is provided, which, when executed on a computer, enable the computer to perform the methods of any of the above aspects.

[0042] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0043] Figure 1 is a schematic diagram of the workflow of a computing unit in an existing AI accelerator;

[0044] Figure 2 is a schematic diagram of the structure of an AI accelerator provided in this application;

[0045] Figure 3 is a flowchart illustrating an accelerator-based data processing method provided in this application;

[0046] Figure 4 is a schematic diagram of the cooperation between a first computing unit and a second computing unit provided in this application;

[0047] Figure 5 is a schematic diagram of a matrix segmentation method provided in this application;

[0048] Figure 6 is a schematic diagram of another matrix segmentation method provided in this application;

[0049] Figure 7 is a schematic diagram of a matrix processing flow based on a second segmentation method provided in this application;

[0050] Figure 8 is a schematic diagram of a matrix processing flow based on a first segmentation method provided in this application;

[0051] Figure 9 is a flowchart illustrating a method for determining matrix segmentation provided in this application;

[0052] Figure 10 is a schematic diagram of a data processing process using a deep learning model provided in this application;

[0053] Figure 11 is a flowchart illustrating a method for searching for parallel computational operators provided in this application;

[0054] Figure 12 is a schematic diagram of a process for fusing parallelizable operators provided in this application;

[0055] Figure 13 is a schematic diagram of operator fusion in a point feedforward network provided in this application;

[0056] Figure 14 is a schematic diagram comparing the computation delay of a related technical solution provided in this application and the operator of this solution;

[0057] Figure 15 is a schematic diagram of fusing matrix multiplication operators and inverse quantization operators according to this application;

[0058] Figure 16 is a schematic diagram of the processing flow after fusing a matrix multiplication operator and an inverse quantization operator according to this application;

[0059] Figure 17 is a schematic diagram of the structure of an AI accelerator provided in this application;

[0060] Figure 18 is a schematic diagram of the structure of a computing device provided in this application;

[0061] Figure 19 is a schematic diagram of the structure of a computer-readable storage medium provided in this application. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some, and not all, of the embodiments of this application. Those skilled in the art will recognize that, with the emergence of new application scenarios, the technical solutions provided by this application are also applicable to similar technical problems.

[0063] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such descriptions can be used interchangeably where appropriate to allow embodiments to be implemented in a sequence other than that illustrated or described in this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved. The division of units in this application is a logical division. In practical applications, there may be other division methods. For example, multiple units may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interface, and the indirect coupling or communication connection between units may be electrical or other similar forms, none of which are limited in this application. Furthermore, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed among multiple circuit units. Some or all of the units can be selected to achieve the purpose of the solution in this application according to actual needs.

[0064] To facilitate understanding, some technical terms used in this application will be introduced below.

[0065] (1) Deep learning model

[0066] Deep learning models refer to neural network models based on deep learning methods, typically including convolutional neural network models, autoencoder neural network models, and large language models.

[0067] (2) Large Language Model (LLM)

[0068] Large language models are deep learning models trained on massive amounts of text data that can generate natural language text or understand the meaning of language text. Large language models can handle various natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence.

[0069] (3) Tensor

[0070] A tensor is a multidimensional data storage format, and the dimension of the data is called the order of the tensor. A tensor can be seen as a generalization of vectors and matrices in multidimensional space; that is, a vector can be seen as a one-dimensional tensor, and a matrix can be seen as a two-dimensional tensor.

[0071] (4) Operators

[0072] In the field of deep learning, an operator generally refers to a computational unit that can perform simple or complex calculations such as addition, subtraction, multiplication, and division. On acceleration hardware, these calculations are usually performed in units of operators.

[0073] (5) AI Accelerator

[0074] AI accelerators refer to a class of hardware specifically designed for AI computing, capable of accelerating the training or inference of AI models. For example, AI accelerators can include graphics processing units (GPUs), neural processing units (NPUs), and tensor processing units (TPUs).

[0075] During the operation of a deep learning model, the input data needs to undergo a series of sequential operator calculations to obtain the model's prediction result. For sequentially connected operators in a deep learning model, the input data of the subsequent operator is the output data of the previous operator. Therefore, the subsequent operator often needs to wait for the previous operator to complete its calculation before it can begin its own calculation.

[0076] Generally, operators in deep learning models can be divided into two types: operators that perform tensor operations (simply called tensor operators) and operators that perform vector operations (simply called vector operators). Tensor operations include matrix multiplication of multiple matrices, while vector operations include activation functions, element-wise multiplication, quantization, and dequantization. Because tensor and vector operations differ significantly in their computational methods, some AI accelerators often use separate computation units to perform tensor and vector operations respectively. For example, an AI accelerator might have a tensor computation unit specifically for performing tensor operations, and a vector computation unit specifically for performing vector operations.

[0077] Because deep learning models consist of a series of sequential operators, current AI accelerators typically employ a serial execution scheme when executing these operators. That is, the AI ​​accelerator executes each operator sequentially according to the order of their operation, thereby processing the input data and obtaining the final output.

[0078] The applicant's research revealed that, in terms of hardware implementation, current AI accelerator computing units typically require three stages when executing operators: data transfer, data computation, and result output. Furthermore, due to the limited storage space within the computing units of AI accelerators, these units often need to perform multiple rounds of data transfer to complete the computation of all input data.

[0079] For example, please refer to Figure 1, which is a schematic diagram of the workflow of a computing unit in an existing AI accelerator. As shown in Figure 1, the deep learning model includes a tensor operator and a vector operator in sequence, where the output of the tensor operator is the input of the vector operator. In the hardware implementation, the tensor computing unit in the AI ​​accelerator repeatedly performs three processes—data transfer, data computation, and result output—during the execution of the tensor operator until all input data has been computed. That is, the tensor computing unit divides the input data into multiple parts, transfers only a portion of the data to the tensor computing unit for computation at a time, and outputs the obtained data outside the tensor computing unit after the computation is completed.

[0080] Furthermore, while the tensor computation unit repeatedly executes the three processes of data transport, data computation, and result output, the vector computation unit remains in a waiting state until the tensor computation unit completes the computation of all input data and obtains all output data. Then, the vector computation unit begins to process the output data of the tensor computation unit, such as repeating the three processes of data transport, data computation, and result output to complete the data processing.

[0081] As shown in Figure 1, during the operation of the AI ​​accelerator, only one of the two computing units (i.e., the tensor computing unit and the vector computing unit) is always in a working state, resulting in a low overall utilization rate of the AI ​​accelerator and a long overall latency for the AI ​​accelerator to perform operations.

[0082] In view of this, this application provides an accelerator-based data processing method. When using an AI accelerator to execute the operations indicated by the model, the method leverages the characteristic of the AI ​​accelerator having computing units for performing different types of operations. The first computing unit on the AI ​​accelerator performs a first operation on a portion of the input data to obtain output data. Thus, while the first computing unit continues to perform the first operation on other portions of the input data, the second computing unit on the AI ​​accelerator can simultaneously perform a second operation on the output data of the first computing unit. This ensures that the first and second computing units execute operations simultaneously, improving the overall utilization rate of the AI ​​accelerator and reducing the overall latency of the AI ​​accelerator's operations.

[0083] Please refer to Figure 2, which is a schematic diagram of the structure of an AI accelerator provided in this application. As shown in Figure 2, the AI ​​accelerator includes a first computing unit, a second computing unit, and a cache. Both the first and second computing units are connected to the cache and are used to read data from the cache and write computed data into the cache. Furthermore, the first and second computing units are used to perform different types of operations. For example, the first computing unit is used to perform tensor operations, and the second computing unit is used to perform vector operations. Alternatively, the first computing unit is used to perform vector operations, and the second computing unit is used to perform tensor operations.

[0084] Furthermore, the first computing unit may include one or more physical components for performing computations, and the second computing unit may also include one or more physical components for performing computations. For example, when the AI ​​accelerator is a GPU, the first computing unit may include one or more tensor cores, and the second computing unit may include one or more Compute Unified Device Architecture Cores (CUDA Cores). As another example, when the AI ​​accelerator is a TPU, the first computing unit may include one or more matrix multiply units (MXUs), and the second computing unit may include one or more vector units. Yet another example is when the AI ​​accelerator is an NPU, the first computing unit may include one or more cube cores, and the second computing unit may include one or more vector cores.

[0085] Please refer to Figure 3, which is a flowchart illustrating an accelerator-based data processing method provided in this application. The accelerator-based data processing method shown in Figure 3 is applied to the AI ​​accelerator shown in Figure 2. Furthermore, the accelerator-based data processing method shown in Figure 3 includes the following steps 301-304.

[0086] Step 301: The first calculation unit performs a first operation on the first data to obtain the second data, wherein the first data is a part of the target data, the target data is the data for which the first operation needs to be performed, and the data obtained after performing the first operation on the target data needs to be subjected to the second operation.

[0087] In this application, an AI accelerator is used to run AI models (such as deep learning models) to achieve the training or inference of the AI ​​models. During the process of running the AI ​​model in the AI ​​accelerator, the first computing unit and the second computing unit in the AI ​​accelerator are used to execute the operations indicated in the AI ​​model, thereby realizing the processing of the input data of the AI ​​model.

[0088] In this step, the target data can specifically be the input data of the AI ​​model (such as text data, image data, video data, or audio data), or the output data of a certain neural network layer in the AI ​​model (such as features of text, images, videos, or audio). Furthermore, during the AI ​​accelerator's operation of the AI ​​model, the AI ​​accelerator needs to perform a first operation on the target data. And, after the AI ​​accelerator performs the first operation on the target data and obtains the processed data, it needs to continue performing a second operation on the processed data.

[0089] Since the first computing unit is used to perform the first operation, it is used to process the target data in this step. Furthermore, the first computing unit first performs the first operation on a portion of the target data (i.e., the first data) to obtain the second data. That is, the first computing unit does not perform the first operation on the entire target data at once, but rather divides the target data into multiple parts and performs the first operation on each of the resulting parts sequentially.

[0090] Step 302: The first computing unit stores the second data in the target cache.

[0091] After performing the first operation on the first data and obtaining the second data, the first computing unit stores the second data in a target cache so that the second computing unit can continue processing the second data. The target cache is a shared cache between the first and second computing units. That is, both the first and second computing units can read data from the target cache and store data in it.

[0092] Step 303: The second computing unit reads the second data from the target cache.

[0093] After the first computing unit stores the calculated second data into the target cache, the second computing unit can read the second data from the target cache so that it can continue to perform the second operation on the second data.

[0094] It should be noted that since the first calculation unit performs the first operation on a portion of the target data (i.e., the first data) to obtain the second data, the second data can be understood as a portion of the output data obtained by performing the first operation on the target data.

[0095] Step 304: During the period when the first calculation unit performs the first operation on the third data, the second calculation unit performs the second operation on the second data, where the third data is a portion of the target data.

[0096] Since the first calculation unit divides the target data into multiple parts to perform the first operation, after the first calculation unit performs the first operation on a part of the target data (i.e., the first data), the first calculation unit will continue to obtain the next part of the target data (i.e., the third data) and perform the first operation on the third data.

[0097] Furthermore, since the second data obtained by the first calculation unit performing the first operation on the first data is a part of the input data of the second calculation unit, when the first calculation unit continues to perform the first operation on the third data, the second calculation unit can synchronously perform the second operation on the second data, thereby realizing the synchronous operation of the first calculation unit and the second calculation unit.

[0098] In this way, the second computing unit does not need to wait for the first computing unit to complete the first operation on all the data in the target data before it starts working. Instead, it can start working after the first computing unit has completed the first operation on a portion of the data, thereby minimizing the waiting time of the first computing unit and improving the efficiency of the AI ​​accelerator in performing calculations.

[0099] Optionally, the first operation performed by the first computing unit includes tensor operations, and the second operation performed by the second computing unit includes vector operations. Alternatively, the first operation performed by the first computing unit includes vector operations, and the second operation performed by the second computing unit includes tensor operations. Tensor operations include matrix multiplication of multiple matrices, while vector operations include operations such as activation functions, element-wise multiplication, quantization, or dequantization.

[0100] It should be noted that steps 301-304 above only describe the process of the first computing unit processing a portion of the target data (i.e., the first data), and the second computing unit performing operations on the data processed by the first computing unit (i.e., the second data). In practical applications, the first computing unit will sequentially perform the first operation on the multiple portions of data divided from the target data and output the processed data to the target cache. Furthermore, the second computing unit will sequentially read the data output by the first computing unit from the target cache and perform the second operation on the read data, until the second computing unit has performed the second operation on all the data output by the first computing unit, thus completing the process of sequentially performing the first and second operations on the target data.

[0101] For example, please refer to Figure 4, which is a schematic diagram of the cooperative operation of a first computing unit and a second computing unit provided in this application. As shown in Figure 4, the first computing unit and the second computing unit cyclically execute three processes during operation: data transfer, data calculation, and result output. Since the calculation process of the second computing unit depends on the output data of the first computing unit, the second computing unit is in a waiting state while the first computing unit is not yet outputting data. That is, when the first computing unit executes the first round of data transfer, data calculation, and result output processes, the second computing unit is in a waiting state.

[0102] After the first computing unit outputs data, the second computing unit can transfer the data output by the first computing unit to the second computing unit and begin performing data calculations. That is, when the first computing unit begins executing the second round of data transfer, data calculation, and result output, the second computing unit can also synchronously begin executing the first round of data transfer, data calculation, and result output, thereby achieving synchronous operation of the first and second computing units.

[0103] In this way, since the second computing unit starts processing data very early, by the time the first computing unit has finished processing all the data, the data that the second computing unit has not yet processed is only a small part of all the data output by the first computing unit. Therefore, the time for the first and second computing units to complete data processing together will be greatly shortened, thereby improving the efficiency of AI accelerator in processing data.

[0104] In summary, this solution leverages the characteristic of AI accelerators having computing units for performing different types of operations. The first computing unit on the AI ​​accelerator performs a first operation on a portion of the input data to obtain output data. Thus, while the first computing unit continues to perform the first operation on the remaining input data, the second computing unit on the AI ​​accelerator can simultaneously perform a second operation on the output data of the first computing unit. This ensures that both the first and second computing units execute operations concurrently, preventing the second computing unit from being in a waiting state while the first computing unit is working, thereby reducing the overall latency of the AI ​​accelerator's operations.

[0105] Optionally, the first computing unit and the second computing unit share multiple caches, with the target cache being the cache with the highest read / write speed among these caches. That is, the target cache used to transfer the second data between the first and second computing units is the cache with the highest read / write speed in the AI ​​accelerator. Therefore, when the second computing unit reads the second data, it can quickly retrieve the second data from the target cache, shortening the time the second computing unit spends transferring data.

[0106] Understandably, to facilitate data exchange between computing units, AI accelerators often employ multiple caches (such as L1, L2, and global caches). Generally, different caches have varying read / write speeds and capacities. Since higher read / write speeds come with higher costs, caches with higher read / write speeds typically have lower capacities. For example, in an AI accelerator with L1, L2, and global caches, the L1 cache has the highest read / write speed and the lowest capacity; the L2 cache has a medium read / write speed and a medium capacity; and the global cache has the lowest read / write speed and the highest capacity.

[0107] In related technologies, different computing units do not operate simultaneously. For example, a vector computing unit often waits for the tensor computing unit to finish processing all its data before processing the data output by the tensor computing unit. Therefore, when the capacity of the cache with the highest read / write speed is low, all the data output by the tensor computing unit often overflows the cache with the highest read / write speed, thus entering other caches with lower read / write speeds, or even overflowing all caches and entering other storage media with even slower read / write speeds in the AI ​​accelerator (such as Dynamic Random Access Memory (DRAM) in the AI ​​accelerator). As a result, when the vector computing unit performs vector operations on the data output by the tensor computing unit, it needs to read data from caches or memory with lower read / write speeds, leading to a longer latency in data transfer for the vector computing unit.

[0108] In this scheme, the first computing unit outputs the processed data to the target cache with the highest read / write speed after processing each part of the data. The second computing unit then reads the data output by the first computing unit from the target cache in a timely manner to perform the operation. This ensures that the second computing unit can always read the data that needs to be operated from the target cache with the highest read / write speed, saving the second computing unit the time of data transfer and improving the overall efficiency of the AI ​​accelerator.

[0109] The above describes the process by which the first computing unit segments the input data and performs multiple rounds of calculations, and the second computing unit performs calculations on the data output by the first computing unit. In practical applications, since the amount of data that the first and second computing units can process in each round of calculations is relatively fixed, there may be various situations in which the first and second computing units work together.

[0110] Case 1: For the data obtained by the first computing unit performing one round of calculations, the second computing unit performs one round of calculations to complete the data processing.

[0111] In other words, the data obtained by the first computing unit in one round of computation can be used as the input data for the second computing unit in one round of computation. That is, the second computing unit also performs one round of computation for each round performed by the first computing unit. The execution of one round of computation by either the first or second computing unit includes three processes: data transfer, data calculation, and result output.

[0112] For example, in steps 301-304 above, the first data read by the first computing unit is all the data required for the first computing unit to perform one round of calculation, and the second data obtained by the first computing unit from performing calculations on the first data is all the data required for the second computing unit to perform one round of calculation. In this case, the data obtained by the first computing unit in each round of calculation can be completely read by the second computing unit to perform one round of calculation.

[0113] In scenario 2, for the data obtained by the first computing unit through multiple rounds of computation, the second computing unit can complete the data processing by performing only one round of computation.

[0114] In other words, the data obtained by the first computing unit in one round of computation is less than the data required by the second computing unit in one round of computation. The data obtained by the first computing unit in multiple rounds of computation must be combined before it can be used as input data for the second computing unit in one round of computation. Therefore, in this case, the second computing unit performs only one round of computation after the first computing unit has performed multiple rounds.

[0115] For example, in steps 301-304 above, the first data includes multiple sub-data. The first computing unit performs a first operation on the first data to obtain the second data, specifically including: the first computing unit sequentially performs the first operation on the multiple sub-data to obtain data after multiple operations, and the data after multiple operations is used to constitute the second data. That is, the first computing unit divides the first data into multiple sub-data, and when the first computing unit performs one round of operation, it reads a portion of the sub-data for processing. Therefore, the first computing unit needs to perform multiple rounds of operation to complete the processing of the first data and obtain the second data.

[0116] For example, suppose the amount of data obtained by the first computing unit in one round of operation is A, and the amount of data required by the second computing unit in one round of operation is 5A (for example, the amount of data that the second computing unit can process at one time is 5A). Then, the second computing unit will only perform its operation based on the data obtained by the first computing unit in its five rounds of operation after the first computing unit has completed five rounds of operation. That is, the second computing unit will only perform one round of operation after the first computing unit has performed five rounds of operation.

[0117] Case 3: For the data obtained by the first computing unit in one round of calculation, the second computing unit needs to perform multiple rounds of calculation to complete the data processing.

[0118] In other words, the data obtained by the first computing unit in one round of computation is more than the data required by the second computing unit in one round of computation. For the data obtained by the first computing unit in one round of computation, the second computing unit only reads a portion of the data for processing in its own round. Therefore, the second computing unit needs to perform multiple rounds of computation to complete the data processing. That is, for every round of computation performed by the first computing unit, the second computing unit needs to continue performing multiple rounds of computation.

[0119] For example, in steps 301-304 above, the second data is obtained by the first computing unit performing a single first operation. That is, the first computing unit performs one round of operation on the first data to obtain the second data.

[0120] Furthermore, the second data includes multiple sub-data components; for example, the second data may be divided into multiple sub-data components by the second computing unit. The second computing unit performs a second operation on the second data, specifically including: the second computing unit sequentially performs the second operation on the multiple sub-data components to obtain data after the multi-part operation. The data after the multi-part operation is used to construct the data obtained after performing the second operation on the second data.

[0121] For example, suppose the amount of data obtained by the first computing unit in one round of operation is 3A, and the amount of data required by the second computing unit in one round of operation is A (for example, the amount of data that the second computing unit can process at one time is A). Then, after the first computing unit completes one round of operation, the second computing unit needs to perform 3 rounds of operation to complete the processing of the data obtained by the first computing unit in one round of operation.

[0122] To facilitate understanding, the following will explain in detail how the first computing unit divides the data to achieve one round of computation.

[0123] For example, if the first operation performed by the first computing unit is a matrix multiplication operation, the target data mentioned above includes a first matrix and a second matrix, that is, the first matrix and the second matrix are two matrices that need to be multiplied.

[0124] Taking the first data as the data used by the first computing unit to perform one round of operations as an example, the first data processed by the first computing unit includes a first submatrix obtained by dividing the first matrix and a second submatrix obtained by dividing the second matrix. That is, the first submatrix is ​​obtained by dividing the first matrix, and the second submatrix is ​​obtained by dividing the second matrix.

[0125] Specifically, in this scheme, based on the characteristics of matrix multiplication, the original matrix is ​​divided into submatrices, and matrix multiplication is performed on each submatrix. The results of the submatrices are then concatenated to obtain the result of direct matrix multiplication of the original matrix. This ensures that matrix processing can be completed even when the first computing unit cannot fully load the matrix that needs to be multiplied.

[0126] Optionally, in this application, the method of partitioning the first matrix and the second matrix is ​​related to the size of the first matrix, the size of the second matrix, and the size of the cache space in the first computing unit. Specifically, the cache space in the first computing unit determines the amount of matrix data that the first computing unit can load at one time, while the size of the first matrix and the size of the second matrix determine the amount of matrix data that needs to be processed.

[0127] In practical applications, the execution device can pre-determine the partitioning method of the first matrix and the second matrix based on the size of the first matrix, the size of the second matrix, and the size of the cache space in the first computing unit, so that the first computing unit can minimize the amount of data transfer when performing operations on the partitioned submatrices.

[0128] Specifically, in this application, there are two ways to divide the first matrix and the second matrix, such as the first division method and the second division method.

[0129] For example, please refer to Figure 5, which is a schematic diagram of matrix partitioning provided in this application. As shown in Figure 5, when the partitioning method of the first matrix and the second matrix is ​​the first partitioning method, the number of rows of the first submatrix is ​​less than the number of rows of the first matrix, and the number of columns of the first submatrix is ​​less than the number of columns of the first matrix. The number of rows of the second submatrix is ​​less than the number of rows of the second matrix, and the number of columns of the second submatrix is ​​less than the number of columns of the second matrix.

[0130] That is, the first submatrix is ​​obtained by partitioning the first matrix into its rows and columns, thus reducing the number of rows and columns in the first submatrix. The second submatrix is ​​obtained by partitioning the second matrix into its rows and columns, thus reducing the number of rows and columns in the second submatrix.

[0131] For example, please refer to Figure 6, which is a schematic diagram of another matrix partitioning method provided by this application. As shown in Figure 6, when the partitioning method of the first matrix and the second matrix is ​​the second partitioning method, the number of rows of the first submatrix is ​​less than the number of rows of the first matrix, and the number of columns of the first submatrix is ​​equal to the number of columns of the first matrix. The number of rows of the second submatrix is ​​equal to the number of rows of the second matrix, and the number of columns of the second submatrix is ​​less than the number of columns of the second matrix.

[0132] That is, the first submatrix is ​​obtained by partitioning the rows of the first matrix, thus reducing the number of rows while keeping the number of columns unchanged. The second submatrix is ​​obtained by partitioning the columns of the second matrix, thus reducing the number of columns while keeping the number of rows unchanged.

[0133] Currently, the second partitioning method mentioned above is mainly used in related technologies to partition matrices. However, in some cases, using the second partitioning method for matrix partitioning can lead to a large amount of data that the computing unit needs to handle, affecting the efficiency of the computing unit in performing matrix multiplication operations.

[0134] Specifically, based on the two matrix partitioning methods introduced above, the following will explain in detail how to determine the actual matrix partitioning method to be used among the two methods.

[0135] Please refer to Figure 7, which is a schematic diagram of a matrix processing flow based on a second partitioning method provided in this application. As shown in Figure 7, assume that the current tensor computation unit needs to perform matrix multiplication on matrices A and B to obtain matrix C; where the size of matrix A is [M,K], the size of matrix B is [K,N], and the size of matrix C is [M,N]. When partitioning the matrix using the second partitioning method described above, the K-axis is used as the inner axis (i.e., the accumulation axis) for partitioning. Then, in one round of computation, the size of the submatrix A′ obtained by partitioning matrix A is [m1,K], the size of the submatrix B′ obtained by partitioning matrix B is [K,n1], and the size of the submatrix C′ output by the tensor computation unit after performing matrix multiplication on submatrices A′ and B′ is [m1,n1]. In the next round of computation, the tensor computation unit continues to move the next part of the submatrix from matrices A and B to perform the operation until the entire matrix C is calculated.

[0136] In the partitioning method shown in Figure 7, each submatrix A′ needs to be multiplied with multiple different submatrixes B′. That is, for the same submatrix A′, the tensor computation unit needs to read different submatrixes B′ and A′ along the inner loop direction of matrix B (i.e., along the columns of matrix B) and perform operations on them. After completing all the operations on the same submatrix A′, the unit then reads a new submatrix A′ along the outer loop direction of matrix A (i.e., along the rows of matrix A) and performs all the operations on the new submatrix A′.

[0137] Therefore, for the same submatrix A′, during the process of traversing and calculating all submatrixes B′, submatrix A′ only needs to be moved to the tensor computation unit once, and the entire matrix A only needs to be moved to the tensor computation unit once. However, for each submatrix A′, submatrix B′ needs to be moved multiple times, so the number of times matrix B is moved is the same as the number of times submatrix A′ is moved. Therefore, for the tensor computation unit, the total amount of input data transferred (DT) can be represented by, for example, the following formula 1.

[0138] Please refer to Figure 8, which is a schematic diagram of a matrix processing flow based on a first partitioning method provided in this application. As shown in Figure 8, assume that the current tensor computation unit needs to perform matrix multiplication on matrices A and B to obtain matrix C; where the size of matrix A is [M,K], the size of matrix B is [K,N], and the size of matrix C is [M,N]. When partitioning the matrix using the first partitioning method described above, the K-axis is used as the inner axis (i.e., the accumulation axis) for partitioning, and the K-axis itself also needs to be partitioned. Then, in one round of computation, the size of the submatrix A′ obtained by partitioning matrix A is [m2,k], the size of the submatrix B′ obtained by partitioning matrix B is [k,n2], and the size of the submatrix C′ output by the tensor computation unit after performing matrix multiplication on submatrices A′ and B′ is [m2,n2]. In the next round of computation, the tensor computation unit continues to move the next part of the submatrix from matrices A and B to perform the operation until the entire matrix C is calculated.

[0139] In the first segmentation method shown in Figure 8, the operation loop is in the order of M, N, K. The outer loop traverses the M axis, the middle loop traverses the N axis, and the inner loop traverses the K axis for accumulation.

[0140] Specifically, the difference between the first segmentation method shown in Figure 8 and the second segmentation method shown in Figure 7 is that for submatrices B′[k,n2] with different N-axis segments, submatrices A′[m2,k] with the same M-axis segmentation need to be repeatedly transported into the tensor computation unit. Therefore, the number of times matrix A is transported is equal to the number of N-axis segments. Thus, for the tensor computation unit, the total amount of input data transported, DT, can be expressed, for example, by the following formula 2.

[0141] Therefore, by comparing Formula 1 and Formula 2 above, we can determine the data transfer amount corresponding to the first and second partitioning methods, and thus select the partitioning method with the smallest data transfer amount to partition the matrix.

[0142] For example, please refer to Figure 9, which is a flowchart illustrating a method for determining matrix partitioning according to this application. As shown in Figure 9, the dimensions of the input data (i.e., the two matrices to be multiplied) are first obtained. Assume the dimensions of matrix A are [M, K] and the dimensions of matrix B are [K, N]. Then, based on the size of the buffer space of the tensor computation unit, the M-axis partitioning size m2 and N-axis partitioning size n2 with the lowest data transfer volume are calculated when the matrix is ​​partitioned using the first partitioning method. Furthermore, based on the size of the buffer space of the tensor computation unit, the M-axis partitioning size m1 with the lowest data transfer volume is calculated when the matrix is ​​partitioned using the second partitioning method.

[0143] Secondly, based on Formulas 1 and 2 above, we can compare whether 1 / N+1 / m1 > 1 / n2+1 / m2 exists. If 1 / N+1 / m1 > 1 / n2+1 / m2, then the first partitioning method is used to partition the matrix; if 1 / N+1 / m1 is less than or equal to 1 / n2+1 / m2, then the second partitioning method is used to partition the matrix.

[0144] In summary, compared to related technologies that consistently use the second partitioning method described above to partition the matrix, this scheme determines the partitioning method based on the size of the input matrix and the size of the cache space of the tensor computation unit. This approach minimizes the amount of data transfer during matrix multiplication operations performed by the tensor computation unit, thereby improving the computational efficiency of the tensor computation unit.

[0145] The above describes the accelerator-based data processing method provided in this application. To facilitate understanding, the following will provide a detailed explanation of how to implement the above accelerator-based data processing method in practical applications, using specific examples.

[0146] Please refer to Figure 10, which is a flowchart illustrating a data processing procedure using a deep learning model provided in this application. As shown in Figure 10, the process of using a deep learning model to perform data processing includes the following steps 101-105.

[0147] Step 101: Construct a model computation graph based on a deep learning model.

[0148] First, the execution device constructs a model computation graph based on the structure of the deep learning model. This computation graph consists of operators and the connections between them. Operators indicate the computational operations performed on the data, and tensor operations and vector operations are not integrated into the same operator. The connections between operators indicate the data dependencies between them. That is, when a previous operator points to a subsequent operator, it means that the output data of the previous operator is the input data of the subsequent operator.

[0149] The execution device can be a physical device such as a server or a server cluster.

[0150] Step 102: Search for operators that can be computed in parallel based on the model computation graph.

[0151] Based on the operation types corresponding to the operators indicated in the model computation graph and the connection relationships between the operators, the execution device searches the model computation graph for the existence of operators that can be computed in parallel.

[0152] Step 103: Merge the parallelizable operators to obtain the fused operator.

[0153] If the model computation graph search has operators that can be computed in parallel, the execution device merges the operators that can be computed in parallel to obtain a merged operator.

[0154] Step 104: Replace the original parallelizable operators in the model computation graph with fusion operators.

[0155] After obtaining the fusion operator, the execution device uses the fusion operator to replace the original parallelizable operators in the model computation graph, thereby updating the model computation graph.

[0156] Step 105: Based on the model computation graph, perform model training or inference on the AI ​​accelerator.

[0157] If the model computation graph search has operators that can be computed in parallel, then based on the updated model computation graph (i.e., the model computation graph including the fusion operator), the execution device can perform training or inference of the deep learning model on the AI ​​accelerator, thereby completing the processing of input data based on the deep learning model.

[0158] If no parallelizable operator is found in the model computation graph search, then based on the model computation graph obtained in step 101, the execution device performs training or inference of the deep learning model on the AI ​​accelerator.

[0159] Specifically, step 102 described above can be accomplished by executing the process shown in Figure 11. Please refer to Figure 11, which is a schematic diagram of a process for searching for parallelizable operators provided in this application. As shown in Figure 11, step 102 described above includes the following steps 1021-1025.

[0160] Step 1021: Determine whether the input data of operator n needs to be segmented.

[0161] Since the operators in the model computation graph are connected in a specific order, each operator in the graph has a corresponding index. Therefore, this approach starts with the first operator in the model computation graph and determines whether there are operators that can be computed in parallel, following the connection order of the operators. Thus, when initially processing the model computation graph, the execution device sets the value of n to 1 and begins to determine whether the input data for operator n needs to be split.

[0162] Specifically, if the computing units in an AI accelerator need to perform multiple rounds of computation to process the input data of operator n, it means that the input data of operator n needs to be segmented. If the computing units in an AI accelerator only need to perform one round of computation on the input data of operator n, it means that the input data of operator n does not need to be segmented.

[0163] Step 1022: If the input data of operator n needs to be segmented, determine the type T1 of the computational unit used by operator n.

[0164] If the input data of operator n needs to be segmented, the execution device then determines the type T1 of the computation unit used by operator n. Generally, there are two types of computation units in AI accelerators: tensor computation units for performing tensor operations and vector computation units for performing vector operations.

[0165] Furthermore, if the input data of operator n does not need to be segmented, the execution device will increment the value of n by 1 and jump to execute step 1021 above.

[0166] Step 1023: Increment the value of n by 1 to determine the type T2 of the computational unit used by operator n.

[0167] After determining the type T1 of the computational unit used by operator n, the execution device increments the value of n by 1 and continues to determine the type T2 of the computational unit used by operator n. That is, the execution device continues to determine the type of the computational unit used by the next operator.

[0168] Step 1024: Determine whether T1 is the same as T2.

[0169] After obtaining the types of computational units used by the two operators, the execution device determines whether the type T1 of the computational unit used by operator n-1 is the same as the type T2 of the computational unit used by operator n.

[0170] Step 1025: If T1 and T2 are not the same, output the operators n and n-1 that can be computed in parallel.

[0171] If T1 and T2 are different, it means that the type of computational unit used by operator n-1 is different from the type of computational unit used by operator n. Therefore, the execution device outputs operators n and n-1 that can be computed in parallel. After outputting operators n and n-1 that can be computed in parallel, the execution device increments the value of n by 1 and jumps to step 1021 above to continue the next round of search until all operators have been traversed.

[0172] If T1 and T2 are different, it means that the type of the computing unit used by operator n-1 is the same as the type of the computing unit used by operator n. Operator n and operator n-1 cannot be computed in parallel. Therefore, the execution device jumps to execute the above step 1021.

[0173] Specifically, step 103 described above can be accomplished by executing the process shown in Figure 12. Please refer to Figure 12, which is a schematic diagram of a process for fusing parallelizable operators provided in this application. As shown in Figure 12, step 103 includes the following steps 1031-1038.

[0174] Step 1031: Input the operators n and n-1 that can be computed in parallel.

[0175] Step 1032: Determine the operator to use the tensor computation unit.

[0176] Steps 1032 and 1033 are optional steps, mainly used to optimize the data partitioning strategy for operators using tensor computation units in two parallel computation operators.

[0177] Specifically, for operators n and n-1 that can be computed in parallel, the operator that uses tensor computation units is determined among these two operators. For example, if the operation type of operator n is tensor operation, then operator n is the operator that uses tensor computation units. Similarly, if the operation type of operator n-1 is tensor operation, then operator n-1 is the operator that uses tensor computation units.

[0178] Step 1033: Optimize the data partitioning strategy of operators using tensor computation units.

[0179] Among them, the data partitioning strategy for optimizing the use of tensor computation units refers to selecting the data partitioning method with the minimum data transfer volume for the operators of tensor computation units. The specific optimization process can be referred to the description of the embodiment corresponding to Figure 9, which will not be repeated here.

[0180] Specifically, the computational latency of tensor operators (i.e., operators using tensor computation units) in most existing deep learning models is greater than that of vector operators (i.e., operators using vector computation units). Therefore, the computational latency of the fused operator is often limited by the latency of tensor operations. Based on this, reducing the computational latency of tensor operators by optimizing the data transfer volume of tensor operations can further improve the computational efficiency of the fused operator.

[0181] Step 1034: Obtain the size o of the output data of operator n-1 in a single round of operation. n-1 .

[0182] Step 1035: Obtain the size i of the input data for a single round of operation of operator n. n .

[0183] Step 1036: Calculate the number of operation rounds c that operator n-1 needs to perform in order to meet the size requirements of the input data of operator n.

[0184] Since the output data of operator n-1 will be used as the input data of operator n, the size o of the output data of a single round of operation of operator n-1 is... n-1 And the size i of the input data for a single round of operation of operator n. n This allows us to determine the number of operation rounds, c, required by operator n-1 to meet the input data size requirements for a single round of operation by operator n. In other words, operator n-1 needs to perform c operations to obtain the input data required for a single round of operation by operator n.

[0185] Generally, if the size of the output data of a single round of operation of operator n-1 is o n-1 The size of the input data i in a single round of operation of operator n is greater than or equal to the size of the input data i. n If operator n-1 needs to perform 1 round of operations, then the number of rounds c is 1; if the size of the output data of operator n-1 in a single round is o n-1 The size of the input data i is less than the size of the single-round operation of operator n. n Then, the number of operation rounds c that operator n-1 needs to perform is an integer greater than 1.

[0186] Step 1037: Determine the parallel processing method of operator n according to the number of operation rounds c.

[0187] Specifically, when the execution device determines the parallel processing method of operator n according to the number of operation rounds c, the execution device can implement it according to the following multiple rules to achieve operator fusion.

[0188] Rule 1: The execution method of operator n-1 does not need to be changed.

[0189] Rule 2: Store the intermediate data output by operator n-1 in a cache that is shared by the tensor computation unit and the vector computation unit and has the highest read / write speed.

[0190] Rule 3: Each execution of operator n must ensure that the required c operations of operator n-1 have been completed.

[0191] Rule 4: Each execution of operator n must ensure that the corresponding computational unit is in a waiting state.

[0192] Step 1038: Output the fused operator.

[0193] After determining the parallel processing method of operator n, the parallel processing methods of operator n-1 and operator n can be written at the code level, thereby realizing the fusion of operator n-1 and operator n, and outputting the fused operator.

[0194] For example, please refer to Figure 13, which is a schematic diagram of operator fusion in a point-wise feedforward network provided by this application. As shown in Figure 13, for the point-wise feedforward network (FFN), which is widely used in the Transformer model, the operators can be fused to obtain fused operators, thereby improving the overall computational efficiency of the point-wise feedforward network. The point-wise feedforward network includes a matrix multiplication operator 1, an activation function operator, and a matrix multiplication operator 2 connected in sequence.

[0195] In Figure 13, when fusing parallelizable operators, there are two optional fusion schemes. In fusion scheme 1, matrix multiplication operator 1 and activation function operator are parallelizable operators, so the execution device fuses matrix multiplication operator 1 and activation function operator. In fusion scheme 2, activation function operator and matrix multiplication operator 2 are parallelizable operators, so the execution device fuses activation function operator and matrix multiplication operator 2.

[0196] Taking fusion scheme 1 as an example, matrix multiplication operator 1 uses tensor computation units, while the activation function operator uses vector computation units. Furthermore, the computation delay of matrix multiplication operator 1 is greater than that of the activation function operator. Therefore, the computation delay of matrix multiplication operator 1 can be used to mask the computation delay of the activation operator. The computation delays of matrix multiplication operator 1 and the activation operator during execution can be seen in Figure 14. Figure 14 is a schematic diagram comparing the computation delays of a related technical solution provided in this application and the operators in this scheme.

[0197] As shown in Figure 14, in related technical solutions, the AI ​​accelerator only begins executing the activation function operator after completely executing matrix multiplication operator 1. Therefore, the total time for the AI ​​accelerator to execute matrix multiplication operator 1 and the activation function operator is the sum of the time for executing matrix multiplication operator 1 and the time for executing the activation function operator. However, in this solution, since matrix multiplication operator 1 and the activation function operator are processed in parallel, after the tensor computation unit in the AI ​​accelerator performs one or more rounds of computation on matrix multiplication operator 1, the vector computation unit will start executing the activation function operator based on the output data of the tensor computation unit, without waiting for matrix multiplication operator 1 to finish executing. This allows the computation delay of the activation function operator to be masked by the computation delay of matrix multiplication operator 1, effectively shortening the total time for the AI ​​accelerator to execute matrix multiplication operator 1 and the activation function operator.

[0198] Currently, full quantization techniques (i.e., parameter quantization and activation quantization) are widely used to accelerate inference for large models. However, to ensure inference accuracy, some operators in the model (i.e., non-quantized operators) cannot be quantized. Therefore, it is necessary to dequantize the calculation results of quantized operators and then use the dequantized results as inputs for non-quantized operators. Among these, matrix multiplication and dequantization operators account for the largest proportion in the model. Therefore, fusing matrix multiplication and dequantization operators can achieve better memory access and improve the overall computational efficiency of AI accelerators.

[0199] For example, please refer to Figure 15, which is a schematic diagram of fusing matrix multiplication and dequantization operators according to this application. As shown in Figure 15, for the matrix multiplication and dequantization operators connected sequentially in the model, the output data type of the matrix multiplication operator is int32, and the output data type of the dequantization operator is bf16. Here, int32 is an integer type, representing a 32-bit signed integer. Furthermore, bf16 is a floating-point type, representing a 16-bit floating-point format, where 8 bits represent the exponent and 7 bits represent the decimal. Compared to int32, bf16 has higher data precision.

[0200] Please refer to Figure 16, which is a schematic diagram of the processing flow after fusing a matrix multiplication operator and an inverse quantization operator according to this application. As shown in Figure 16, after the matrix multiplication operator and the inverse quantization operator are fused, the tensor computation unit in the AI ​​accelerator is responsible for executing the matrix multiplication operator, and the vector computation unit is responsible for executing the inverse quantization operator to achieve parallel computation of the two operators. Specifically, at time step T = n, the tensor computation unit loads and computes part of the input matrix data and writes the computation result to the cache. At time step T = n+1, the vector computation unit reads the computation result of the tensor computation unit from the cache, performs inverse quantization on the computation result, and then writes the inverse quantization result to the output position. At the same time, the tensor computation unit loads and computes another part of the input matrix data again and writes the computation result to the cache. And so on, at each subsequent time step, the vector computation unit reads the computation result obtained by the tensor computation unit in the previous time step and performs inverse quantization on the read computation result, thereby realizing the synchronous operation of the vector computation unit and the tensor computation unit.

[0201] The method provided in this application has been described in detail above. Next, the device provided in this application for performing the above method will be described.

[0202] Please refer to Figure 17, which is a schematic diagram of the structure of an AI accelerator provided in this application. As shown in Figure 10, the AI ​​accelerator includes: a first computing unit 1701 and a second computing unit 1702. The first computing unit 1701 and the second computing unit 1702 are used to perform different types of operations. The first computing unit 1701 is used to perform a first operation on first data to obtain second data, wherein the first data is a part of the target data, the target data is the data to be operated on, and the data obtained after performing the first operation on the target data needs to be operated on for the second operation. During the period when the first computing unit 1701 performs the first operation on the third data, the second computing unit 1702 is used to perform the second operation on the second data, and the third data is a part of the target data.

[0203] In one possible implementation, the first computing unit 1701 is further configured to store the second data into a target cache; the second computing unit 1702 is further configured to read the second data from the target cache; wherein the target cache is a cache shared by the first computing unit 1701 and the second computing unit 1702.

[0204] In one possible implementation, the first computing unit 1701 and the second computing unit 1702 share multiple caches, and the target cache is the cache with the highest read / write speed among the multiple caches.

[0205] In one possible implementation, the first operation includes tensor operations and the second operation includes vector operations; or, the first operation includes vector operations and the second operation includes tensor operations.

[0206] In one possible implementation, the first operation is a matrix multiplication operation, and the target data includes a first matrix and a second matrix; the first data includes a first submatrix obtained by partitioning the first matrix and a second submatrix obtained by partitioning the second matrix.

[0207] In one possible implementation, the way the first matrix and the second matrix are split is related to the size of the first matrix, the size of the second matrix, and the size of the cache space in the first computing unit 1701.

[0208] In one possible implementation, when the first matrix and the second matrix are partitioned in the first partitioning manner, the number of rows in the first submatrix is ​​less than the number of rows in the first matrix, and the number of columns in the first submatrix is ​​less than the number of columns in the first matrix; the number of rows in the second submatrix is ​​less than the number of rows in the second matrix, and the number of columns in the second submatrix is ​​less than the number of columns in the second matrix.

[0209] In one possible implementation, when the first matrix and the second matrix are partitioned in the second way, the number of rows in the first submatrix is ​​less than the number of rows in the first matrix, and the number of columns in the first submatrix is ​​equal to the number of columns in the first matrix; the number of rows in the second submatrix is ​​equal to the number of rows in the second matrix, and the number of columns in the second submatrix is ​​less than the number of columns in the second matrix.

[0210] In one possible implementation, the first data includes multiple sub-data; the first computing unit 1701 is specifically used to perform a first operation on the multiple sub-data in sequence to obtain data after the multiple operations, and the data after the multiple operations is used to constitute the second data.

[0211] In one possible implementation, the second data is obtained by the first computing unit 1701 performing a single first operation, and the second data includes multiple sub-data. The second computing unit 1702 is specifically used to perform a second operation on the multiple sub-data in sequence to obtain data after multiple operations. The data after multiple operations is used to constitute the data obtained after performing the second operation on the second data.

[0212] Please refer to Figure 18, which is a schematic diagram of a computing device provided in this application. The computing device 1800 shown in Figure 18 can be used to execute the accelerator-based data processing method provided in this embodiment. As shown in Figure 18, the computing device 1800 includes: a bus 1802, a processor 1804, an AI accelerator 1805, a memory 1806, and a communication interface 1808. The processor 1804, the memory 1806, and the communication interface 1808 communicate with each other via the bus 1802. The computing device 1800 can be a server or a terminal device. It should be understood that this application does not limit the number of processors, AI accelerators, and memories in the computing device 1800.

[0213] Bus 1802 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 18, but this does not imply that there is only one bus or one type of bus. Bus 1802 can include pathways for transmitting information between various components of computing device 1800 (e.g., memory 1806, processor 1804, communication interface 1808).

[0214] Processor 1804 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0215] The memory 1806 may include volatile memory, such as random access memory (RAM). The processor 1804 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0216] The memory 1806 stores executable program code, which the AI ​​accelerator 1805 executes to implement the aforementioned accelerator-based data processing method. That is, the memory 1806 stores instructions for executing the accelerator-based data processing method.

[0217] The communication interface 1808 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 1800 and other devices or communication networks.

[0218] This application also provides a chip system comprising a processing unit and a communication unit. The processing unit may be, for example, an AI accelerator, and the communication unit may be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in a storage unit to cause the chip system to perform the methods described in the above embodiments. Optionally, the storage unit may be an in-chip storage unit, such as a register or cache. Alternatively, the storage unit may be an external storage unit located within a wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0219] Referring to Figure 19, which is a schematic diagram of the structure of a computer-readable storage medium provided in this application. This application also provides a computer-readable storage medium in which, in some embodiments, the method disclosed in Figure 3 above can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0220] Figure 19 schematically illustrates a conceptual partial view of an example computer-readable storage medium arranged according to at least some of the embodiments shown herein, the example computer-readable storage medium including a computer program for executing computer processes on a computing device.

[0221] In one embodiment, the computer-readable storage medium 1900 is provided using a signal bearer medium 1901. The signal bearer medium 1901 may include one or more program instructions 1902, which, when executed by one or more processors, can provide the functions or parts thereof described above with reference to FIG3.

[0222] In some examples, the signal carrying medium 1901 may include a computer-readable medium 1903, such as, but not limited to, a hard disk drive, a compact disc (CD), a digital video optical disc (DVD), a digital magnetic tape, a memory, ROM, or RAM, etc.

[0223] In some embodiments, the signal-bearing medium 1901 may comprise a computer-recordable medium 1904, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, and so on. In some embodiments, the signal-bearing medium 1901 may comprise a communication medium 1905, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, and so on). Therefore, for example, the signal-bearing medium 1901 may be transmitted by a wireless communication medium 1905 (e.g., a wireless communication medium conforming to the IEEE 1102.X standard or other transmission protocols).

[0224] One or more program instructions 1902 may be, for example, computer-executable instructions or logical implementation instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to one or more program instructions 1902 conveyed to the computing device via a computer-readable medium 1903, a computer-recordable medium 1904, and / or a communication medium 1905.

[0225] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0226] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0227] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0228] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, computer instructions can be transferred from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. An accelerator-based data processing method, characterized in that, The method is applied to an artificial intelligence (AI) accelerator, which includes a first computing unit and a second computing unit. The first computing unit and the second computing unit are used to perform different types of operations. The method includes: The first computing unit performs a first operation on the first data to obtain second data, wherein the first data is a portion of the target data, the target data is the data for which the first operation needs to be performed, and the data obtained after performing the first operation on the target data needs to undergo a second operation. While the first computing unit performs the first operation on the third data, the second computing unit performs the second operation on the second data, wherein the third data is a portion of the target data.

2. The method according to claim 1, characterized in that, The method further includes: The first computing unit stores the second data in the target cache; The second computing unit reads the second data from the target cache; The target cache is a cache shared by the first computing unit and the second computing unit.

3. The method according to claim 2, characterized in that, The first computing unit and the second computing unit share multiple caches, and the target cache is the cache with the highest read and write speed among the multiple caches.

4. The method according to any one of claims 1-3, characterized in that, The first operation includes tensor operations, and the second operation includes vector operations; Alternatively, the first operation may include vector operations, and the second operation may include tensor operations.

5. The method according to any one of claims 1-4, characterized in that, The first operation is a matrix multiplication operation, and the target data includes a first matrix and a second matrix; The first data includes a first submatrix obtained by dividing the first matrix and a second submatrix obtained by dividing the second matrix.

6. The method according to claim 5, characterized in that, The way the first matrix and the second matrix are split is related to the size of the first matrix, the size of the second matrix, and the size of the cache space in the first computing unit.

7. The method according to claim 6, characterized in that, When the first matrix and the second matrix are partitioned in the first partitioning method, the number of rows in the first submatrix is ​​less than the number of rows in the first matrix, and the number of columns in the first submatrix is ​​less than the number of columns in the first matrix; The number of rows in the second submatrix is ​​less than the number of rows in the second matrix, and the number of columns in the second submatrix is ​​less than the number of columns in the second matrix.

8. The method according to claim 6, characterized in that, When the first matrix and the second matrix are partitioned in the second partitioning method, the number of rows in the first submatrix is ​​less than the number of rows in the first matrix, and the number of columns in the first submatrix is ​​equal to the number of columns in the first matrix; The number of rows in the second submatrix is ​​equal to the number of rows in the second matrix, and the number of columns in the second submatrix is ​​less than the number of columns in the second matrix.

9. The method according to any one of claims 1-8, characterized in that, The first data includes multiple sub-data components; The first computing unit performs a first operation on the first data to obtain second data, including: The first computing unit sequentially performs the first operation on the multiple sub-data to obtain multi-part processed data, which is used to constitute the second data.

10. The method according to any one of claims 1-8, characterized in that, The second data is obtained by the first computing unit performing the first operation once. The second data includes multiple sub-data. The second computing unit performs the second operation on the second data, including: The second calculation unit sequentially performs the second operation on the multiple sub-data to obtain data after the multiple operations, and the data after the multiple operations is used to form the data obtained after performing the second operation on the second data.

11. An AI accelerator, characterized in that, The AI ​​accelerator includes a first computing unit and a second computing unit, which are used to perform different types of operations. The first calculation unit is used to perform a first operation on the first data to obtain second data, wherein the first data is a portion of the target data, the target data is the data for which the first operation needs to be performed, and the data obtained after performing the first operation on the target data needs to undergo a second operation. During the period when the first computing unit performs the first operation on the third data, the second computing unit performs the second operation on the second data, wherein the third data is a portion of the target data.

12. The AI ​​accelerator according to claim 11, characterized in that, The first computing unit is further configured to store the second data into the target cache; The second computing unit is further configured to read the second data from the target cache; The target cache is a cache shared by the first computing unit and the second computing unit.

13. The AI ​​accelerator according to claim 12, characterized in that, The first computing unit and the second computing unit share multiple caches, and the target cache is the cache with the highest read and write speed among the multiple caches.

14. The AI ​​accelerator according to any one of claims 11-13, characterized in that, The first operation includes tensor operations, and the second operation includes vector operations; Alternatively, the first operation may include vector operations, and the second operation may include tensor operations.

15. The AI ​​accelerator according to any one of claims 11-14, characterized in that, The first operation is a matrix multiplication operation, and the target data includes a first matrix and a second matrix; The first data includes a first submatrix obtained by dividing the first matrix and a second submatrix obtained by dividing the second matrix.

16. The AI ​​accelerator according to claim 15, characterized in that, The way the first matrix and the second matrix are split is related to the size of the first matrix, the size of the second matrix, and the size of the cache space in the first computing unit.

17. The AI ​​accelerator according to claim 16, characterized in that, When the first matrix and the second matrix are partitioned in the first partitioning method, the number of rows in the first submatrix is ​​less than the number of rows in the first matrix, and the number of columns in the first submatrix is ​​less than the number of columns in the first matrix; The number of rows in the second submatrix is ​​less than the number of rows in the second matrix, and the number of columns in the second submatrix is ​​less than the number of columns in the second matrix.

18. The AI ​​accelerator according to claim 16, characterized in that, When the first matrix and the second matrix are partitioned in the second partitioning method, the number of rows in the first submatrix is ​​less than the number of rows in the first matrix, and the number of columns in the first submatrix is ​​equal to the number of columns in the first matrix; The number of rows in the second submatrix is ​​equal to the number of rows in the second matrix, and the number of columns in the second submatrix is ​​less than the number of columns in the second matrix.

19. The AI ​​accelerator according to any one of claims 11-18, characterized in that, The first data includes multiple sub-data components; The first computing unit is specifically used to perform the first operation on the multiple sub-data in sequence to obtain data after multiple operations, and the data after multiple operations is used to constitute the second data.

20. The AI ​​accelerator according to any one of claims 11-18, characterized in that, The second data is obtained by the first computing unit performing the first operation once, and the second data includes multiple sub-data. The second calculation unit is specifically used to perform the second operation on the multiple sub-data in sequence to obtain data after the multiple operations, and the data after the multiple operations is used to form the data obtained after performing the second operation on the second data.

21. A chip system, characterized in that, The chip system includes an AI accelerator; the AI ​​accelerator is used to perform the method as described in any one of claims 1 to 10.

22. A computing device, characterized in that, The device includes a memory and an AI accelerator; the memory stores code, and the AI ​​accelerator is configured to execute the code, wherein when the code is executed, the computing device performs the method as described in any one of claims 1 to 10.

23. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed by the computer, cause the computer to perform the method according to any one of claims 1 to 10.

24. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 10.

Images