Method and apparatus for processing vector operator on basis of tensor core

Abstract

The present application relates to a method and apparatus for processing a vector operator on the basis of a tensor core. The method comprises: upon receiving a vector operator and determining that the length of the vector operator is an integer multiple of a preset value, converting the vector operator into an N×M matrix to be operated, wherein N and M are both integer multiples of k, and the preset value is k2; on the basis of an operation requirement of the vector operator, invoking a matching operation matrix, the number of elements of which is an integer multiple of the preset value; pre-multiplying the matching operation matrix by the matrix to be operated, so as to obtain a matrix to be determined; pre-multiplying the matrix to be determined by the matching operation matrix, so as to obtain a target matrix; and determining any element in the target matrix as a result value of the operation requirement of the vector operator.

Description

Methods and apparatus for processing vector operators based on Tensor Core

[0001] Related applications

[0002] This application claims priority to Chinese patent application filed on November 26, 2024, with application number 2024117036131, entitled "Method, apparatus and computer device for processing vector operators based on Tensor Core", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus and computer device for processing vector operators based on Tensor Core. Background Technology

[0004] Tensor Cores, or TCs for short, are custom hardware units within a GPU (Graphics Processing Unit) specifically designed to accelerate matrix multiplication operations. Essentially, they are Application Specific Integrated Circuits (ASICs). Each Tensor Core can perform mixed-precision matrix multiplication operations of a specific size within a single instruction. While CUDA Cores (a processing unit within the GPU) can perform general-purpose computing, their versatility limits their performance in specific computations. Tensor Cores, on the other hand, are hardware-optimized computational units for matrix multiplication operations, offering significantly greater matrix operation capabilities than CUDA Cores. This is particularly crucial for training and inferring neural networks in AI (Artificial Intelligence) models.

[0005] However, since Tensor Core is specifically designed to support matrix multiplication operations, it cannot support other operations. In other words, Tensor Core cannot directly perform operations on non-matrix operations. Summary of the Invention

[0006] According to various embodiments of this application, a method, apparatus, computer device, computer-readable storage medium, and computer program product for processing vector operators based on Tensor Core are provided.

[0007] Firstly, this application provides a method for processing vector operators based on Tensor Cores, the method comprising:

[0008] Upon receiving a vector operator, and determining that the length of the vector operator is an integer multiple of a preset value, the vector operator is converted into an N×M matrix to be operated on; where N and M are both integer multiples of k, and the preset value is k. 2 ;

[0009] Based on the computational requirements of the vector operator, a matching operation matrix with an element count that is an integer multiple of the preset value is invoked;

[0010] Multiply the matching operation matrix on the left by the matrix to be operated on to obtain the matrix to be determined;

[0011] Multiply the matrix to be determined on the left by the matching operation matrix to obtain the target matrix; and

[0012] Determine any element in the target matrix as the result value required by the operation of the vector operator.

[0013] In one embodiment, after receiving the vector operator, the method further includes:

[0014] If it is determined that the length of the vector operator is not an integer multiple of the preset value, the vector operator is padded to an integer multiple of the preset value based on the element value 0.

[0015] In one embodiment, the matching operation matrix whose number of elements is an integer multiple of the preset value includes:

[0016] If the number of elements in the matching operation matrix is ​​not an integer multiple of the preset value, the number of elements in the matching operation matrix is ​​padded to an integer multiple of the preset value based on the element value 0.

[0017] In one embodiment, the step of calling a matching operation matrix with an element count that is an integer multiple of the preset value based on the operational requirements of the vector operator includes:

[0018] When the operation requirement of the vector operator is summation, an auxiliary matrix of k×k with all elements having a value of 1 is called as the matching operation matrix.

[0019] In one embodiment, after receiving the vector operator, the method further includes:

[0020] If it is determined that the vector operator is not a reduce-type vector operator, the vector operator is converted into a reduce-type vector operator.

[0021] In one embodiment, the calculation process of the target matrix reuses the register space of the calculation process of the matrix to be determined.

[0022] Secondly, this application also provides an apparatus for processing vector operators based on Tensor Cores, the apparatus comprising:

[0023] The data conversion module is configured to, upon receiving a vector operator and determining that the length of the vector operator is an integer multiple of a preset value, convert the vector operator into an N×M matrix to be operated on; where N and M are both integer multiples of k, and the preset value is k. 2 ;

[0024] The matrix calling module is configured to call a matching operation matrix whose number of elements is an integer multiple of the preset value based on the operation requirements of the vector operator;

[0025] The matrix processing module is configured to multiply the matching operation matrix on the left by the matrix to be operated on, to obtain the matrix to be determined;

[0026] The result output module is configured to left-multiply the matrix to be determined by the matching operation matrix to obtain the target matrix; and

[0027] The result output module is further configured to determine any element in the target matrix as the result value required by the operation of the vector operator.

[0028] In one embodiment, the data conversion module is further configured to, after receiving the vector operator, pad the vector operator to an integer multiple of the preset value based on the element value 0 if it is determined that the length of the vector operator is not an integer multiple of the preset value.

[0029] In one embodiment, the matrix invocation module is configured to invoke a matching operation matrix whose number of elements is an integer multiple of the preset value, including:

[0030] If the number of elements in the matching operation matrix is ​​not an integer multiple of the preset value, the number of elements in the matching operation matrix is ​​padded to an integer multiple of the preset value based on the element value 0.

[0031] In one embodiment, the matrix invocation module is configured to invoke a matching operation matrix whose number of elements is an integer multiple of the preset value based on the operation requirements of the vector operator, including:

[0032] When the operation requirement of the vector operator is summation, an auxiliary matrix of k×k with all elements having a value of 1 is called as the matching operation matrix.

[0033] In one embodiment, the data conversion module is further configured to convert the vector operator into a reduce-type vector operator after receiving the vector operator, if it is determined that the vector operator is not a reduce-type vector operator.

[0034] In one embodiment, the calculation process of the target matrix reuses the register space of the calculation process of the matrix to be determined.

[0035] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0036] Upon receiving a vector operator, and determining that the length of the vector operator is an integer multiple of a preset value, the vector operator is converted into an N×M matrix to be operated on; where N and M are both integer multiples of k, and the preset value is k. 2 ;

[0037] Based on the computational requirements of the vector operator, a matching operation matrix with an element count that is an integer multiple of the preset value is invoked;

[0038] Multiply the matching operation matrix on the left by the matrix to be operated on to obtain the matrix to be determined;

[0039] Multiply the matrix to be determined on the left by the matching operation matrix to obtain the target matrix; and

[0040] Determine any element in the target matrix as the result value required by the operation of the vector operator.

[0041] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0042] Upon receiving a vector operator, and determining that the length of the vector operator is an integer multiple of a preset value, the vector operator is converted into an N×M matrix to be operated on; where N and M are both integer multiples of k, and the preset value is k. 2 ;

[0043] Based on the computational requirements of the vector operator, a matching operation matrix with an element count that is an integer multiple of the preset value is invoked;

[0044] Multiply the matching operation matrix on the left by the matrix to be operated on to obtain the matrix to be determined;

[0045] Multiply the matrix to be determined on the left by the matching operation matrix to obtain the target matrix; and

[0046] Determine any element in the target matrix as the result value required by the operation of the vector operator.

[0047] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0048] Upon receiving a vector operator, and determining that the length of the vector operator is an integer multiple of a preset value, the vector operator is converted into an N×M matrix to be operated on; where N and M are both integer multiples of k, and the preset value is k. 2 ;

[0049] Based on the computational requirements of the vector operator, a matching operation matrix with an element count that is an integer multiple of the preset value is invoked;

[0050] Multiply the matching operation matrix on the left by the matrix to be operated on to obtain the matrix to be determined;

[0051] Multiply the matrix to be determined on the left by the matching operation matrix to obtain the target matrix; and

[0052] Determine any element in the target matrix as the result value required by the operation of the vector operator.

[0053] Details of one or more embodiments of this application are set forth in the following drawings and description. Other features, objects, and advantages of this application will become apparent from the specification, drawings, and claims. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 is a schematic diagram of a Tensor Core unit in a GPU in related technologies;

[0056] Figure 2 is a schematic diagram of MMA operation in the Tensor Core unit in related technologies;

[0057] Figure 3 is a flowchart illustrating the method for processing vector operators based on Tensor Core in some embodiments;

[0058] Figure 4 is a schematic diagram of the acceleration effect of the Softmax operator based on Tensor Core acceleration in some embodiments;

[0059] Figure 5 is a schematic diagram of the acceleration effect of the Softmax operator based on CUDA Core acceleration in some embodiments;

[0060] Figure 6 is a structural block diagram of a device for processing vector operators based on Tensor Core in some embodiments;

[0061] Figure 7 is an internal structure diagram of a computer device in some embodiments. Detailed Implementation

[0062] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0063] In modern computing environments, graphics processing units (GPUs) are widely used in scientific computing, artificial intelligence, and image processing due to their powerful parallel computing capabilities. In typical AI models, such as the ResNet (residual neural network) series for vision, the YOLO (a real-time object detection system) series for detection, and the BERT (a natural language processing model) series for NLP, matrix operations account for over 90% of the computation. In modern large-scale model scenarios, this proportion is typically even higher.

[0064] Please refer to Figure 1. Although CUDA Cores already possess powerful computing capabilities, the most advanced NVIDIA GPUs currently contain tens of thousands of CUDA Cores (e.g., the NVIDIA L40S GPU contains 18,176 CUDA Cores). Generally, we refer to the traditional multi-core CPU (Central Processing Unit) architecture as a multi-core architecture, while GPUs, due to their large number of CUDA Cores, are called many-core architectures. The number of CUDA Cores typically indicates the graphics card's computational power in fp32 data precision. It's worth explaining that float, a single-precision data type, occupies 4 bytes in x86 platform memory. Since each byte in a computer occupies 8 bits, the float data type occupies 32 bits in a computer, hence the abbreviation fp32, which stands for float point 32bit. Simply put, the more CUDA Cores a GPU has, the stronger its single-precision (float or fp32) computational power is generally (of course, it also depends on the GPU's clock speed). The versatility and large number of CUDA Cores give them unparalleled advantages over traditional CPUs in typical parallel computing scenarios. In Figure 1, the Instruction Cache is an instruction cache, the warp scheduler is a warp scheduler, the Dispatch Unit is a scheduling unit, the Register File is a register file, and "LD / ST" stands for load and store, which are read units or arithmetic units.

[0065] As mentioned earlier, since matrix operations account for over 90% of the total computation in AI models, starting with the Volta architecture, NVIDIA began introducing a custom circuit in its GPU chips specifically designed to accelerate matrix multiplication operations. This was called the first-generation Tensor Core, further accelerating deep learning applications and receiving support from the programming model level starting with CUDA 9.0. The latest Hopper architecture (superchip architecture) has evolved into the fourth generation of Tensor Cores. Each Tensor Core is an independent hardware unit capable of performing a 4x4x4 matrix multiply-accumulate (MMA) operation within a single instruction. In contrast, using CUDA Cores requires repeatedly calling FMA (Fused Multiply-accumulate) instructions, as shown in Figure 2.

[0066] Among them, the 4x4x4 matrix multiplication and accumulation operation, i.e., D 4×4=A 4×4 ×B 4×4 +C 4×4 Tensor Cores (TCs) play a crucial role in deep learning applications such as autonomous driving, camera processing, real-time anomaly detection, and traditional linear algebra operations, including matrix multiplication of large matrices. Actual GPU Tensor Core performance can be an order of magnitude faster than conventional FP32 / INT32 GPU CUDA Cores. However, adapting arbitrary algorithms to Tensor Cores is not easy because Tensor Cores differ from conventional GPU CUDA Cores. While the latter can execute the entire instruction set, TCs can only perform one operation (MMA), but at a significantly higher speed.

[0067] Although matrix operations account for a very high proportion of computations in AI models, and NVIDIA GPUs have accelerated them through the introduction of Turbo C, the non-matrix operation parts of AI models cannot be ignored, especially in inference scenarios that pursue ultimate performance.

[0068] The method for processing vector operators based on Tensor Cores provided in this application can be applied to a terminal, where the GPU includes a custom hardware unit, Tensor Core, specifically designed to accelerate matrix multiplication operations. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted displays, etc. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc.

[0069] To facilitate a clear understanding of this application, some terms related to this application are explained.

[0070] Graphics Processing Unit (GPU): A GPU is a piece of hardware in a computer specifically designed for processing graphics and images. It accelerates graphics rendering, video editing, and gaming experiences by processing large amounts of data in parallel. GPUs not only play a crucial role in gaming but also demonstrate powerful computing capabilities in fields such as scientific computing, artificial intelligence, and deep learning. With technological advancements, GPU performance continues to improve, providing robust support for various high-performance computing tasks. Currently, GPUs have become the standard hardware infrastructure for AI training and inference.

[0071] Compute Unified Device Architecture (CUDA) is a parallel computing platform and programming model. It allows developers to use NVIDIA GPUs for general-purpose computing, not just graphics processing. CUDA provides a complete set of tools and libraries, enabling ordinary C / C++ developers to leverage the powerful parallel processing capabilities of GPUs to accelerate computationally intensive tasks such as scientific simulations, deep learning, and data analysis. CUDA supports multiple programming languages, including C, C++, Python, and Fortran (and several other languages), and is compatible with widely used development environments and compilers. With CUDA, developers can write efficient parallel code that fully utilizes the thousands of computing cores of GPUs, significantly improving application performance.

[0072] CUDA Cores, or CC for short, are key components of NVIDIA GPUs for high-performance computing. They provide users with powerful computing capabilities, enabling complex computational tasks to be completed in a short time, greatly driving technological progress and innovation across various industries. While the computing power of each CUDA Core within an NVIDIA GPU is far less than that of a single CPU core, its lightweight nature allows for the integration of tens of thousands of CUDA Cores within a single NVIDIA GPU. For typical parallel computing tasks such as image algorithms and deep learning, the extremely high degree of parallelism fully utilizes CUDA Cores to improve computational efficiency, achieving efficiency improvements of tens, hundreds, or even thousands of times compared to traditional CPU cores in these tasks.

[0073] Tensor Cores, or TCs for short, are custom hardware units within NVIDIA GPUs specifically designed to accelerate matrix multiplication operations. Essentially, they are Application Specific Integrated Circuits (ASICs). Introduced with the Volta architecture GPUs, each Tensor Core can perform mixed-precision matrix multiplication operations of a specific size within a single instruction. While CUDA Cores enable general-purpose computing, their versatility limits their performance in specific computations. Tensor Cores, on the other hand, are hardware-optimized computational units for matrix multiplication operations, offering significantly greater matrix operation capabilities than CUDA Cores. This is particularly crucial for training and inferring neural networks in AI models, as matrix multiplication constitutes the largest computational load in neural networks. By optimizing matrix multiplication and accumulation operations, Tensor Cores have significantly improved the performance and efficiency of deep learning models. Tensor Cores have become one of the core technologies enabling NVIDIA GPUs to maintain their leading position in AI computing, continuously pushing the boundaries of artificial intelligence technology.

[0074] Vector operators: AI models are built from layers of operators. Tensor data flows into the next layer after being processed by each layer of operators, continuing the computation until the final layer outputs the data. Just as the name of the well-known deep learning training framework TenorFlow suggests, AI computation is essentially the flow of tensors between operator layers. Therefore, the computation process of AI models is often represented by graphs. AI models contain a wide variety of operators, such as convolution, activation (ReLU, GeLU, Tanh), normalization (Batch Norm, Layer Norm, Group Norm), and MHA (Multi-Head Attention). From the perspective of the ratio of computational cost to data volume, operators can be divided into computationally intensive and memory-intensive types. Computationally intensive operators involve a number of operations that far exceed the size of the data. Typical computationally intensive operators include matrix multiplication and convolution, whose computational cost is typically on the order of magnitude of an order of magnitude, while the memory size is typically on the order of a number of orders of magnitude. Typical memory-intensive operators include activation functions, normalization, and Softmax (an activation function). Memory-intensive operators are usually vector operators. Vector operators are the opposite of matrix operators, or they are purely element-wise computations, meaning the operator operation only involves the relationship between individual elements without interaction between elements, such as the ReLU operator. Alternatively, they may include simple reduce operations (reduce sum, reduce max / min, etc.) after element-wise operations. Element-wise operations involve mathematical operations on elements at corresponding positions within vectors, matrices, or tensors, such as addition, subtraction, multiplication, and division. Here, sum is for summation, max is for finding the maximum value, and min is for finding the minimum value.

[0075] In an exemplary embodiment, as shown in Figure 3, a method for processing vector operators based on Tensor Cores is provided. Taking the application of this method to a terminal as an example, the method includes the following steps 202 to 210. Wherein:

[0076] Step 202: Upon receiving the vector operator and confirming that the length of the vector operator is an integer multiple of a preset value, convert the vector operator into an N×M matrix to be operated on; where N and M are both integer multiples of k, and the preset value is k. 2 .

[0077] In this application, the vector operators can be non-matrix operations, such as vector reduce operators. The reduce operators are used to perform specific calculations (such as sum, Max / Min) on a certain dimension of the input multidimensional tensor data, thereby achieving the purpose of reducing dimensionality.

[0078] An N×M matrix to be operated on is also a type of matrix operation, which can be processed by Tensor Cores.

[0079] For the preset value, this application provides an optional embodiment of 16, then the corresponding k is 4.

[0080] For example, when a vector operator that is not a matrix operation is received, this application needs to convert the vector operator into one that can be processed by the Tensor Core. In this case, the length of the received vector operator can be determined first to confirm whether the length of the vector operator is an integer multiple of the preset value of 16. When the length of the vector operator is an integer multiple of the preset value, the view of the vector operator is converted into an N×M matrix to be operated on, for example, a 4×4 matrix to be operated on.

[0081] Step 204: Based on the computational requirements of the vector operator, call the matching operation matrix whose number of elements is an integer multiple of the preset value.

[0082] Among them, the matching operation matrix can be an operation matrix that has an operational relationship with the vector operator, that is, the vector operator itself needs to perform operations with another operation matrix to obtain the relevant operation results; in addition, the matching operation matrix can also be a pre-set operation matrix based on the operation requirements of the vector operator in order to realize the relevant operations of the vector operator.

[0083] For example, based on the computational requirements of the vector operator, the matching operation matrix associated with the vector operator is called; for the matching operation matrix here, this application provides an optional implementation method of calling a matching operation matrix with an element count that is an integer multiple of a preset value, such as calling a matching operation matrix with 16 elements, that is, a 4×4 matching operation matrix.

[0084] Step 206: Multiply the matching operation matrix on the left by the matrix to be operated on to obtain the matrix to be determined.

[0085] For example, after obtaining the matrix to be operated on corresponding to the vector operator and the matching operation matrix to be operated on, the matching operation matrix is ​​multiplied on the left by the matrix to be operated on to obtain the matrix to be determined.

[0086] Step 208: Multiply the matrix to be determined by the matching operation matrix on the left to obtain the target matrix; and Step 210: Determine any element in the target matrix as the result value required by the vector operator operation.

[0087] For example, after step 206 is executed, the left multiplication of the matrix to be determined into the matching operation matrix is ​​performed to obtain the target matrix associated with the vector operator, that is, the operation result obtained by the vector operator based on the Tensor Core. Then, any element in the target matrix is ​​determined to be the result value required by the operation of the vector operator, for example, the first element in the target matrix is ​​determined to be the result value required by the operation of the vector operator.

[0088] It should be noted that steps 202-210 above can be performed using the Tensor Cores of the GPU in the terminal.

[0089] In the above method for processing vector operators based on Tensor Cores, the vector operator is converted into an N×M matrix to be operated on when it is received and the length of the vector operator is determined to be an integer multiple of a preset value; where N and M are both integer multiples of k, and the preset value is k. 2 Based on the computational requirements of vector operators, a matching operation matrix with an element count that is an integer multiple of a preset value is invoked; the matching operation matrix is ​​multiplied on the left by the matrix to be operated on to obtain the matrix to be determined; the matrix to be determined is multiplied on the left by the matching operation matrix to obtain the target matrix; and any element in the target matrix is ​​determined to be the result value required by the vector operator. It can be seen that this application enables Tensor Core to perform non-matrix operations by simply transforming the vector operator to be operated on, thereby improving the utilization of GPU resources and improving the overall performance of the system.

[0090] In an exemplary embodiment, after receiving the vector operator, the method further includes: if it is determined that the length of the vector operator is not an integer multiple of a preset value, padding the vector operator to an integer multiple of the preset value based on the element value 0.

[0091] For example, when confirming the length of the vector operator received in step 202, there may be cases where the length is less than the preset value or is not an integer multiple of the preset value. In order to ensure that the matrix to be operated on obtained by the subsequent conversion of the vector operator can be a 4×4 matrix, this application provides an optional implementation method: when it is determined that the length of the vector operator is not an integer multiple of the preset value, the vector operator can be padded to an integer multiple of the preset value based on the element value 0. For example, if the length of the vector operator is 14, then two element values ​​of 0 are used to padded the vector operator to a length of 16.

[0092] By padding the length of the vector operator to an integer multiple of the preset value, it is beneficial to ensure that the matrix to be operated on obtained by the subsequent transformation of the vector operator is a 4×4 matrix, which is a matrix that can be operated on by the Tensor Core.

[0093] It should also be added that vector operators whose length is less than an integer multiple of the preset value can be converted into an N×M matrix to be operated on, and then the missing elements in the N×M matrix to be operated on can be filled with the element value 0.

[0094] However, the method provided in this application, which first pads the length of the vector operator to an integer multiple of the preset value and then performs the conversion of the vector operator into the matrix to be operated on, can save more computing power.

[0095] In an exemplary embodiment, calling a matching operation matrix whose number of elements is an integer multiple of a preset value includes: calling the matching operation matrix, and if it is determined that the number of elements of the matching operation matrix is ​​not an integer multiple of the preset value, padding the number of elements of the matching operation matrix to an integer multiple of the preset value based on the element value 0.

[0096] For example, for the matching operation matrix whose number of elements is an integer multiple of a preset value, which is called in step 204 above, this application provides an optional implementation method: calling the matching operation matrix, for example, the matrix that is already provided and to be operated on with the vector operator, and then determining the number of elements in the matching operation matrix to determine whether the number of elements in the matching operation matrix is ​​an integer multiple of the preset value; if it is determined that the number of elements in the matching operation matrix is ​​not an integer multiple of the preset value, it is also possible to choose to pad the number of elements in the matching operation matrix to an integer multiple of the preset value based on the element value 0, thereby obtaining a matrix (matching operation matrix) that can be operated on by the Tensor Core.

[0097] In an exemplary embodiment, based on the computational requirements of the vector operator, a matching operation matrix with an element count that is an integer multiple of a preset value is invoked, including: when the computational requirement of the vector operator is summation, an auxiliary matrix of k×k with each element value being 1 is invoked as the matching operation matrix.

[0098] For example, regarding the content performed in step 204 above, this application also provides an alternative implementation method: when the operation requirement of the vector operator is summation, that is, when no matrix is ​​originally provided to be operated with the vector operator, a pre-set k×k auxiliary matrix can be called as the matching operation matrix, and the value of each element in the auxiliary matrix is ​​1. For example, it can be a 4×4 auxiliary matrix with all element values ​​being 1.

[0099] This auxiliary matrix enables the Tensor Core to perform summation on the matrix to be operated on, which is converted from the vector operator, without affecting the result of the summation operation, thus ensuring the accuracy of the result obtained after the operation.

[0100] In one exemplary embodiment, after receiving the vector operator, the method further includes: if it is determined that the vector operator is not a reduce-type vector operator, converting the vector operator into a reduce-type vector operator.

[0101] For example, after receiving the vector operator in step 202 above, this application also provides an alternative implementation method: if it is determined that the vector operator is not a reduce-type vector operator, the vector operator can be converted into a reduce-type vector operator, or at least some of the vector operators in the operation requirements can be converted into reduce-type vector operators; the vector operators converted into reduce-type vector operators can perform related operations based on the method of processing vector operators based on Tensor Core provided by this application.

[0102] In one exemplary embodiment, the process of calculating the target matrix reuses the register space used in the process of calculating the matrix to be determined. That is, an alternative implementation provided by this application is that the register space used in the process of calculating the target matrix can be the same as the register space used in the process of calculating the matrix to be determined, so as to avoid additional memory overhead.

[0103] In summary, this application transforms non-matrix operations into matrix operations through simple transformations and accelerates the computation using Tensor Cores (TC), enabling both single-operator and fusion-operator versions suitable for training and inference scenarios, respectively. The applicant's practical experience demonstrates that TC-optimized vector operators achieve a 1-3x performance improvement. For example, vector reduce operators can be transformed into matrix multiplication operations through simple mathematical transformations to accelerate them using Tensor Cores. This method can simultaneously implement single-operator and fusion-operator versions for accelerating AI model training and inference.

[0104] The technical effects achieved by this application include at least the following three points.

[0105] Firstly, it maximizes GPU performance: TC only supports matrix multiplication operations, and non-matrix multiplication operations can only be performed in CC. This application transforms some vector operators into matrix operations through simple number transformations with negligible cost, and uses TC's powerful matrix operation capabilities to accelerate the operation of vector operators, thereby improving the utilization of GPU hardware resources.

[0106] Secondly, hybrid programming of CC and TC: CC and TC are independent hardware units, and their collaborative work is a major research direction in academia and industry, aiming to improve the overall resource utilization of GPUs. The transformation strategy proposed in this application can directly utilize TC to accelerate the operation of pure vector reduce operators, and can use hybrid programming of CC and TC to fully utilize their respective advantages for vector reduce operator variants.

[0107] Third, improve AI training and inference performance: The performance of a single operator directly affects the time consumption and throughput of model training and inference. The performance of an AI model depends not only on the performance parameters at the hardware level, but also on the extreme optimization of hardware performance by efficient operators implemented by high-performance computing engineers. For training, especially in large model training scenarios, efficient operators can greatly save training time, and less training time can also reduce the probability of hardware failure. For inference scenarios, better performance can save inference latency and improve service throughput, providing a better user experience and supporting higher user concurrency.

[0108] This application addresses the issue that non-matrix operators can only be operated on CUDA Cores. It provides a simple method to enable some non-matrix operators to also utilize the powerful computing capabilities of Tensor Cores, thereby improving GPU resource utilization and enhancing the overall system performance.

[0109] Regarding the method for processing vector operators based on Tensor Cores provided in this application, an alternative implementation is also provided. Taking a float type vector summation operator of length 16 as an example, the relevant method steps include steps 111-115, wherein:

[0110] Step 111, Review: Assume the vector to be summed is a = [a0, a1, ..., a...]. 15 Change its view to treat it as a 4x4 matrix fragment, i.e., a fragment in MMA: In the field of AI, the view only changes the view itself, without changing the actual memory layout of the tensor data, so there is no performance loss from additional operations.

[0111] Step 112: Initialize a 4x4 auxiliary matrix b, all values ​​being 1.0, using the Tensor Core MMA instruction fill_fragment. In actual encoding, since the value of b is fixed at 1.0, there is no need to declare data b of the same size as vector a. A fragment can be shared within a single warp, so the resulting memory overhead is negligible.

[0112] Step 113: Multiply matrix b by matrix a on the left to obtain the following matrix c.

[0113] Step 114: Multiply matrix c by matrix b on the left to obtain the following matrix d.

[0114] The calculation of matrix d can reuse the register space used for the calculation of matrix c, that is, c = c * b does not need to declare the use of a different register space, thus avoiding additional memory overhead.

[0115] Step 115: Take any value from matrix d (usually the first one), which is the final calculation result.

[0116] The above steps show that using Tensor Core with two matrix multiplication instructions, namely steps 113 and 114, achieves the summation of 16 elements. Using CC warp shuffle instructions such as `shfl_xor_sync` would require four instruction calls. Therefore, using TC reduces the number of instruction calls by at least half. For example, considering half or int8 types, larger fragment multiplications can be performed within a single MMA instruction, resulting in even greater savings in instruction calls compared to CC's shuffle instructions.

[0117] In AI algorithms, typical variants of reduce operators similar to this example are most obviously vector inner product and vector modulo. These can be achieved simply by replacing the vector 'b' with the actual vector and then applying a shuffle instruction to the diagonal elements of 'c'. Furthermore, various normalization operators (Batch Norm, Layer Norm, RMSnorm, etc.) and Softmax operators can be extended to more scenarios with slight modifications to the above transformations and hybrid implementations using CUDA Cores.

[0118] Figures 4 and 5 respectively illustrate the acceleration effect of the Softmax operator based on TC acceleration and CUDA Core implementation for reference. Here, Time represents time in milliseconds (ms); Speedup represents acceleration.

[0119] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0120] Based on the same inventive concept, this application also provides an apparatus for implementing the above-described method for processing vector operators based on Tensor Cores. The solution provided by this apparatus is similar to the implementation described in the above-described method. Therefore, the exemplary limitations in one or more apparatus embodiments for processing vector operators based on Tensor Cores provided below can be found in the limitations of the method for processing vector operators based on Tensor Cores described above, and will not be repeated here.

[0121] In an exemplary embodiment, as shown in FIG6, a device 800 for processing vector operators based on Tensor Core is provided, including: a data conversion module 81, a matrix calling module 82, a matrix processing module 83, and a result output module 84, wherein:

[0122] The data conversion module 81 is configured to convert an N×M matrix to be operated on when it receives a vector operator and determines that the length of the vector operator is an integer multiple of a preset value; where N and M are both integer multiples of k, and the preset value is k. 2 ;

[0123] The matrix call module 82 is configured to call a matching operation matrix whose number of elements is an integer multiple of a preset value based on the operation requirements of vector operators;

[0124] The matrix processing module 83 is configured to multiply the matching operation matrix on the left by the matrix to be operated on, to obtain the matrix to be determined.

[0125] The output module 84 is configured to left-multiply the matrix to be determined by the matching operation matrix to obtain the target matrix; and

[0126] The output module 84 is also configured to determine any element in the target matrix as the result value required by the vector operator.

[0127] In an exemplary embodiment, the data conversion module 81 is further configured to, after receiving a vector operator, pad the vector operator to an integer multiple of the preset length based on the element value 0 if it is determined that the length of the vector operator is not an integer multiple of the preset value.

[0128] In an exemplary embodiment, the matrix calling module 82 is configured to call a matching operation matrix whose number of elements is an integer multiple of a preset value, including: calling the matching operation matrix, and when it is determined that the number of elements of the matching operation matrix is ​​not an integer multiple of the preset value, padding the number of elements of the matching operation matrix to an integer multiple of the preset value based on the element value 0.

[0129] In an exemplary embodiment, the matrix calling module 82 is configured to call a matching operation matrix with an element count that is an integer multiple of a preset value based on the operation requirements of the vector operator, including: when the operation requirement of the vector operator is summation, calling an auxiliary matrix of k×k with each element value of 1 as the matching operation matrix.

[0130] In an exemplary embodiment, the data conversion module 81 is further configured to convert the vector operator into a reduce-type vector operator after receiving the vector operator and determining that the vector operator is not a reduce-type vector operator.

[0131] In one exemplary embodiment, the process of calculating the target matrix reuses the register space used for the process of calculating the matrix to be determined.

[0132] The modules in the aforementioned Tensor Core-based vector operator processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware within or independently of the processor in a computer device, or stored in software within the memory of the computer device, so that the processor can invoke and execute the operations corresponding to each module.

[0133] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram is shown in Figure 7. The computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The I / O interfaces of the computer device are used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for processing vector operators based on Tensor Cores.

[0134] Those skilled in the art will understand that the structure shown in Figure 7 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. The computer device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0135] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0136] Upon receiving a vector operator, and confirming that the length of the vector operator is an integer multiple of a preset value, the vector operator is converted into an N×M matrix to be operated on; where N and M are both integer multiples of k, and the preset value is k. 2 ;

[0137] Based on the computational requirements of vector operators, a matching operation matrix with an element count that is an integer multiple of a preset value is invoked;

[0138] Multiply the matching operation matrix on the left by the matrix to be operated on to obtain the matrix to be determined;

[0139] Multiply the matrix to be determined by the matching operation matrix on the left to obtain the target matrix; and

[0140] Determine any element in the target matrix as the result value required by the vector operator operation.

[0141] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0142] Upon receiving a vector operator, and confirming that the length of the vector operator is an integer multiple of a preset value, the vector operator is converted into an N×M matrix to be operated on; where N and M are both integer multiples of k, and the preset value is k. 2 ;

[0143] Based on the computational requirements of vector operators, a matching operation matrix with an element count that is an integer multiple of a preset value is invoked;

[0144] Multiply the matching operation matrix on the left by the matrix to be operated on to obtain the matrix to be determined;

[0145] Multiply the matrix to be determined by the matching operation matrix on the left to obtain the target matrix; and

[0146] Determine any element in the target matrix as the result value required by the vector operator operation.

[0147] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0148] Upon receiving a vector operator, and confirming that the length of the vector operator is an integer multiple of a preset value, the vector operator is converted into an N×M matrix to be operated on; where N and M are both integer multiples of k, and the preset value is k. 2 ;

[0149] Based on the computational requirements of vector operators, a matching operation matrix with an element count that is an integer multiple of a preset value is invoked;

[0150] Multiply the matching operation matrix on the left by the matrix to be operated on to obtain the matrix to be determined;

[0151] Multiply the matrix to be determined by the matching operation matrix on the left to obtain the target matrix; and

[0152] Determine any element in the target matrix as the result value required by the vector operator operation.

[0153] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0154] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0155] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0156] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for processing vector operators based on Tensor Cores, characterized in that, The method includes: Upon receiving a vector operator, and determining that the length of the vector operator is an integer multiple of a preset value, the vector operator is converted into an N×M matrix to be operated on; where N and M are both integer multiples of k, and the preset value is k. 2 ; Based on the computational requirements of the vector operator, a matching operation matrix with an element count that is an integer multiple of the preset value is invoked; Multiply the matching operation matrix on the left by the matrix to be operated on to obtain the matrix to be determined; Multiply the matrix to be determined on the left by the matching operation matrix to obtain the target matrix; and Determine any element in the target matrix as the result value required by the operation of the vector operator.

2. The method according to claim 1, characterized in that, After receiving the vector operator, the process further includes: If it is determined that the length of the vector operator is not an integer multiple of the preset value, the vector operator is padded to an integer multiple of the preset value based on the element value 0.

3. The method according to claim 1, characterized in that, The matching operation matrix whose number of called elements is an integer multiple of the preset value includes: If the number of elements in the matching operation matrix is ​​not an integer multiple of the preset value, the number of elements in the matching operation matrix is ​​padded to an integer multiple of the preset value based on the element value 0.

4. The method according to claim 1, characterized in that, The operation requirement based on the vector operator, which calls a matching operation matrix whose number of elements is an integer multiple of the preset value, includes: When the operation requirement of the vector operator is summation, an auxiliary matrix of k×k with all elements having a value of 1 is called as the matching operation matrix.

5. The method according to claim 1, characterized in that, After receiving the vector operator, the process further includes: If it is determined that the vector operator is not a reduce-type vector operator, the vector operator is converted into a reduce-type vector operator.

6. The method according to any one of claims 1-5, characterized in that, The calculation process for the target matrix reuses the register space used for the calculation process of the matrix to be determined.

7. A device for processing vector operators based on Tensor Core, characterized in that, The device includes: The data conversion module is configured to, upon receiving a vector operator and determining that the length of the vector operator is an integer multiple of a preset value, convert the vector operator into an N×M matrix to be operated on; where N and M are both integer multiples of k, and the preset value is k. 2 ; The matrix calling module is configured to call a matching operation matrix whose number of elements is an integer multiple of the preset value based on the operation requirements of the vector operator; The matrix processing module is configured to multiply the matching operation matrix on the left by the matrix to be operated to obtain the matrix to be determined. The result output module is configured to left-multiply the matrix to be determined by the matching operation matrix to obtain the target matrix; and The result output module is further configured to determine any element in the target matrix as the result value required by the operation of the vector operator.

8. The apparatus according to claim 7, characterized in that, The data conversion module is further configured to, after receiving the vector operator, pad the vector operator to an integer multiple of the preset value based on the element value 0 if it is determined that the length of the vector operator is not an integer multiple of the preset value.

9. The apparatus according to claim 7, characterized in that, The matrix invocation module is configured to invoke a matching operation matrix whose number of elements is an integer multiple of the preset value, including: If the number of elements in the matching operation matrix is ​​not an integer multiple of the preset value, the number of elements in the matching operation matrix is ​​padded to an integer multiple of the preset value based on the element value 0.

10. The apparatus according to claim 7, characterized in that, The matrix invocation module is configured to invoke a matching operation matrix whose number of elements is an integer multiple of the preset value, based on the operation requirements of the vector operator, including: When the operation requirement of the vector operator is summation, an auxiliary matrix of k×k with all elements having a value of 1 is called as the matching operation matrix.

11. The apparatus according to claim 7, characterized in that, The data conversion module is further configured to, after receiving the vector operator, convert the vector operator into a reduce-type vector operator if it is determined that the vector operator is not a reduce-type vector operator.

12. The apparatus according to any one of claims 7-11, characterized in that, The calculation process for the target matrix reuses the register space used for the calculation process of the matrix to be determined.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-6.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1-6.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1-6.

Images