Processor, chip product, computer device, and tensor processing method

Abstract

A processor, a chip product, a computer device, and a tensor processing method, relating to the technical field of processors. A first memory cell stores a first tensor having a plurality of elements arranged in a first ordering sequence; a tensor memory engine is used for reading the plurality of elements from the first memory cell to a second memory cell in a second ordering sequence to obtain a reordered tensor; a tensor compute engine is used for performing a logical operation on the reordered tensor to obtain an operation result; a format conversion unit is used for converting the data format of each element in the operation result into a target data format to obtain a format-converted operation result, and storing the format-converted operation result into a third memory cell, wherein the format-converted operation result comprises respective operation results of the plurality of elements, and a third ordering sequence of the respective operation results of the plurality of elements in the third memory cell is the same as the first ordering sequence. The present application can reduce development costs of processors.

Description

Processors, chip products, computer equipment and tensor processing methods

[0001] This application claims priority to Chinese Patent Application No. 202411997249.4, filed on December 31, 2024, entitled "Processor, Chip Product, Computer Equipment and Tensor Processing Method", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of processor technology, and in particular to a processor, chip product, computer device, and tensor processing method. Background Technology

[0003] Currently, processors can include TME (Tensor Memory Engine) and TCE (Tensor Compute Engine). TME is responsible for accessing memory units to accelerate the access of tensor data, while TCE is responsible for tensor computation.

[0004] TCE (Tencent Processing Unit) performs logical operations on tensor data to obtain the first operation result. This first result includes the operation results of each element within the tensor, distributed across different registers. When the first operation result needs to be used in subsequent operations, the processor's SIMD (Single Instruction Multiple Data) unit performs data format conversion on the operation results of each element and packages the converted operation results from different registers to obtain a packaged first operation result. This packaged first operation result is then used in subsequent operations. However, because the SIMD unit adjusts the order of the converted operation results during the packaging process, the order of the operation results corresponding to each element in the packaged first operation result becomes inconsistent with the order of the operation results corresponding to each element in the first operation result itself. This is detrimental to subsequent operations (i.e., adjustment is required before subsequent operations).

[0005] To address the issue of inconsistent arrangement order, related technologies employ an offline data rearrangement scheme. This involves adding an offline rearrangement process to the tensor data before the TCE performs logical operations on it. This is achieved by adjusting the order of the encoded data (i.e., the encoded data corresponding to each element in the tensor) through software design, resulting in adjusted tensor data. The TCE then performs logical operations on the adjusted tensor data to obtain a second operation result. This second operation result undergoes data format conversion and packaging to obtain a packaged second operation result. During the packaging process, the SIMD unit further adjusts the order of the elements in the second operation result, ensuring that the order of the operation results corresponding to each element in the packaged second operation result is consistent with the order of the operation results corresponding to each element in the aforementioned first operation result. This facilitates subsequent computations (i.e., no further adjustments are needed before subsequent operations).

[0006] However, the development cost of the processor increases because the related technologies involve additional offline reordering processes through software design. Summary of the Invention

[0007] This application provides a processor, a chip product, a computer device, and a tensor processing method. The technical solutions provided by this application include the following:

[0008] According to one aspect of the embodiments of this application, a processor is provided, the processor comprising: a tensor storage engine, a tensor computation engine, a format conversion unit, a first storage unit, a second storage unit, and a third storage unit; the first storage unit stores a first tensor, the first tensor comprising a plurality of elements arranged in a first arrangement order;

[0009] The tensor storage engine is used to read the plurality of elements from the first storage unit into the second storage unit according to a second arrangement order to obtain a rearranged tensor, wherein the second arrangement order is different from the first arrangement order;

[0010] The tensor computation engine is used to perform logical operations on the rearranged tensor to obtain the computation result;

[0011] The format conversion unit is used to convert the data format of each element in the calculation result to the target data format, to obtain the format-converted calculation result, and to store the format-converted calculation result in the third storage unit;

[0012] The calculation result after format conversion includes the calculation result of each of the multiple elements, and the third arrangement order of the calculation result of each of the multiple elements in the third storage unit is the same as the first arrangement order.

[0013] According to one aspect of the embodiments of this application, a chip product is provided, the chip product including the processor described above.

[0014] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including the processor described above.

[0015] According to one aspect of the embodiments of this application, a tensor processing method applied to a processor is provided, the processor including: a tensor storage engine, a tensor calculation engine, a format conversion unit, a first storage unit, a second storage unit, and a third storage unit; the first storage unit stores a first tensor, the first tensor including a plurality of elements arranged in a first arrangement order;

[0016] The method includes:

[0017] The tensor storage engine reads the multiple elements from the first storage unit into the second storage unit according to the second arrangement order to obtain a rearranged tensor, wherein the second arrangement order is different from the first arrangement order;

[0018] The tensor calculation engine performs logical operations on the rearranged tensor to obtain the calculation result;

[0019] The format conversion unit converts the data format of each element in the calculation result to the target data format to obtain the format-converted calculation result, and stores the format-converted calculation result in the third storage unit;

[0020] The calculation result after format conversion includes the calculation result of each of the multiple elements, and the third arrangement order of the calculation result of each of the multiple elements in the third storage unit is the same as the first arrangement order.

[0021] The technical solutions provided in this application embodiment may include the following beneficial effects.

[0022] During the process of the tensor storage engine reading the first tensor from the first storage unit to the second storage unit, the tensor storage engine can automatically rearrange multiple elements in the first tensor so that the third arrangement order of the operation results of multiple elements in the third storage unit is the same as the first arrangement order of multiple elements. This eliminates the need to develop additional software to support the additional rearrangement process, thereby reducing the complexity of the tensor processing flow in the processor and avoiding the development costs brought by software design, thus reducing the development cost of the processor.

[0023] In addition, since this application implements tensor rearrangement at the existing hardware level, it avoids the impact of adding a rearrangement process on the processor's end-to-end performance, thereby improving the processor's end-to-end performance. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 is a schematic diagram of a processor provided in one possible implementation of this application;

[0026] Figure 2 is a schematic diagram of a method for packaging computation results provided in one possible implementation of this application;

[0027] Figure 3 is a schematic diagram of a tensor storage engine provided in one possible implementation of this application;

[0028] Figure 4 is a schematic diagram of an alternating arrangement provided in one possible implementation of this application;

[0029] Figure 5 is a schematic diagram of the rearrangement of the first tensor provided in one possible implementation of this application under staggered arrangement;

[0030] Figure 6 is a schematic diagram of a possible implementation of the present application, showing the filling of preset elements;

[0031] Figure 7 is a schematic diagram of the rearrangement, data format conversion and packaging of the first tensor provided in one possible implementation of this application under non-interleaved arrangement;

[0032] Figure 8 is a schematic diagram of a graphics processor provided in one possible implementation of this application;

[0033] Figure 9 is a schematic diagram of the processor (i.e., the core computing unit) in the graphics processor provided in one possible implementation of this application;

[0034] Figure 10 is a flowchart of a tensor processing method applied to a processor provided in one possible implementation of this application;

[0035] Figure 11 is a simplified structural block diagram of a computer device provided in one possible implementation of this application. Detailed Implementation

[0036] Before describing the embodiments of this application, the relevant terms involved in this application will be explained.

[0037] Tensor: A tensor is a general term and extension of concepts such as scalar, vector, and matrix. It is a fundamental data structure in the field of machine learning and can be understood as a multidimensional array containing at least one element. For example, a 0-dimensional tensor is a single number, a 1-dimensional tensor is equivalent to a vector, a 2-dimensional tensor corresponds to a matrix, and a 3-dimensional tensor is a cube. This application does not limit the dimension of the tensor; for example, the dimension of a tensor can also be 4 dimensions or higher.

[0038] Tensors can store more information about objects, such as the height, color, depth, and width of an image; the frame rate, color, height, and width of a video; and the time domain, frequency domain, and fundamental frequency of audio.

[0039] Tensor computation mainly refers to the computations and processing performed on tensors in the fields of deep learning and machine learning. These computations typically involve operations such as matrix multiplication and convolution. Tensor computation is a key step in the training and inference of machine learning models.

[0040] Floating-point number type: This is a data type used to represent real numbers. It allows the decimal point to float, thus enabling the representation of fractions, decimals, and certain irrational numbers. Real numbers represented using the floating-point number type are called floating-point numbers.

[0041] Floating-point types include at least one of the following: single-precision floating-point (float), double-precision floating-point (double), high-precision floating-point (decimal), and floating-point types such as FP16 (Floating-Point 16-bit), BF16 (Brain Floating-Point 16-bit), FP8 (Floating-Point 8-bit), FP6 (Floating-Point 6-bit), and FP4 (Floating-Point 4-bit).

[0042] Floating-point numbers are typically represented using scientific notation. For example, FP16: half-precision floating-point, a 16-bit floating-point representation, meaning it uses a 16-bit value to represent a real number, including 1 sign bit, 5 exponent bits, and 10 mantissa bits. FP8: uses an 8-bit value to represent a real number, including a 4-bit exponent and a 3-bit exponent (E4M3), or a 5-bit exponent and a 2-bit mantissa (E5M2), designed to improve computational efficiency by reducing the number of bits used to represent the data.

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0044] Please refer to Figure 1, which shows a schematic diagram of a processor provided in one possible implementation of this application. The processor 100 may include: a Tensor Memory Engine (TME) 101, a first storage unit 103, and a second storage unit 104.

[0045] The processor 100 mentioned above can refer to a processor that supports tensor processing, such as processor 100 supporting at least one of the following operations for tensors: transfer, storage, and computation. For example, the tensor storage engine 101 in processor 100 supports tensor transfer, each storage unit in processor 100 supports tensor storage, and the tensor computation engine 102 in processor 100 supports tensor computation. Tensor transfer can refer to the transfer of tensor data corresponding to a tensor, tensor storage can refer to the storage of tensor data corresponding to a tensor, and tensor computation can refer to the computation of tensor data corresponding to a tensor. Tensor data is used to represent tensors. Optionally, processor 100 can be implemented as a physical circuit for processing tensor data.

[0046] In some embodiments, the processor 100 may be implemented as a processing unit (PU) supporting tensor processing, such as a core computing unit. This application embodiment does not limit the processor 100; the processor 100 may be a completely new design or an improvement upon an existing processor. Exemplarily, the processor 100 may be implemented as at least one of the following: a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a tensor processing unit (TPU), a field-programmable gate array (FPGA), a tensor core, or an NPU (Neural Processing Unit).

[0047] In one example, the processor 100 supports processing data formats including at least one of the following: floating-point numbers in data formats such as FP8, FP16, BF16, FP6, FP4, FP32, and TF32 (Tensor-Floating 32-bit, 32-bit mixed floating-point number); fixed-point numbers in data formats such as U8 and S8; and integers in data formats such as INT8 (8-bit integer) and INT16 (16-bit integer).

[0048] For example, processor 100 supports the processing of tensor data in FP8 data format. Tensor data refers to data used to represent a tensor, which can consist of encoded data generated for each element in the tensor, with each element represented by an encoded data. For example, tensor data in FP8 data format includes: encoded data generated for each element in the tensor according to the FP8 data format, where each encoded data can include 8 bits of data, and each 8-bit data can determine the value of a floating-point number (i.e., an element). The order of the encoded data corresponding to each element is consistent with the order of the elements in the tensor.

[0049] Optionally, each encoded data may consist of a set of binary numbers, such as 0s and 1s. For example, in the FP6 data format, each encoded data may include 6 bits of data, such as 6 binary numbers, which indicate the sign, exponent, and mantissa of the floating-point number (i.e., the element). In the FP4 data format, each encoded data may include 4 bits of data, such as 4 binary numbers, which indicate the sign, exponent, and mantissa of the floating-point number (i.e., the element).

[0050] Tensor storage engine 101 refers to an engine in processor 100 used to perform tensor transfer and tensor storage. The engine can be implemented as a hardware system for processing data, such as physical circuitry (i.e., hardware circuitry). For example, tensor storage engine 101 can be implemented as physical circuitry used to perform tensor transfer and tensor storage.

[0051] Optionally, the processor 100 may further include a Tensor Compute Engine (TCE) 102. Exemplarily, the tensor storage engine 101 is used to read tensors (i.e., tensor data) from the first storage unit 103 and store the tensors (i.e., tensor data) in the second storage unit 104, so that the tensor computation engine 102 can read tensors (i.e., tensor data) from the second storage unit 104 and perform logical operations on the tensors (i.e., tensor data). Optionally, a data transmission channel (such as a data bus) is respectively provided between the tensor storage engine 101, the first storage unit 103, and the second storage unit 104, and the tensor storage engine 101 can transmit tensor data through the data transmission channel. For example, the tensor storage engine 101 can be used to transfer tensor data in FP8 data format in the first storage unit 103 to the second storage unit 104 through the data transmission channel.

[0052] The first storage unit 103 is used to store tensor data to be processed by the processor 100, and the second storage unit 104 is used to temporarily store tensor data to be calculated by the tensor calculation engine 102. The tensor calculation engine 102 can directly read tensor data from the second storage unit 104 to perform tensor calculations.

[0053] In one example, the second storage unit 104 can be implemented as an on-chip cache, denoted as local memory, to temporarily store tensor data, thereby improving the speed at which the tensor computation engine 102 retrieves tensor data. Alternatively, the second storage unit 104 can also be implemented as a register, such as a vector register or a vector register file, to temporarily store tensor data, thereby further improving the speed at which the tensor computation engine 102 retrieves tensor data (the data read speed of registers is much faster than the data read speed of memory).

[0054] The first storage unit 103 can be implemented as a shared cache of at least one tensor computing engine 102, referred to as global memory. It can support at least one tensor computing engine 102 (through tensor storage engine 101) to access its data, such as by supporting multiple tensor computing engines 102 to access tensor data, so as to realize the sharing of tensor data among multiple tensor computing engines 102.

[0055] For example, the second storage unit 104 can be implemented as a Level 1 cache in the processor 100, i.e., an L1 cache, which is the cache layer closest to the tensor calculation engine 102. It typically has a small storage space but a very fast data read speed. The first storage unit 103 can be implemented as a Level 2 cache in the processor 100, i.e., an L2 cache. In this case, the first storage unit 104 is located between the L1 cache (i.e., the second storage unit 104) and the main memory of the processor 100 (such as an L3 cache), and is used to store more tensor data to provide more backups in case of L1 cache misses. The storage space of the L2 cache is larger than that of the L1 cache, but the data read speed of the L2 cache is slower than that of the L1 cache.

[0056] Tensor computation engine 102 refers to the computation engine in processor 100 used to perform tensor computations. For example, tensor computation engine 102 can be implemented as a physical circuit for performing tensor computations. Optionally, the logical operations supported by tensor computation engine 102 include at least one of the following: matrix multiplication, convolution, multiply-accumulate, and addition. For example, tensor computation engine 102 can be used to perform multiply-accumulate operations on tensor data to achieve tensor multiply-accumulate operations and obtain the tensor multiply-accumulate operation result. This application embodiment does not limit the number of tensor computation engines 102 included in processor 100; they can be set and adjusted according to actual usage requirements.

[0057] For example, a data transmission channel is provided between the tensor calculation engine 102 and the second storage unit 104, so that the tensor calculation engine 102 can read data from the second storage unit 104. For example, the tensor calculation engine 102 can be used to read tensor data in FP8 data format from the second storage unit 104, and perform logical operations on the tensor data in FP8 data format to obtain the tensor calculation result, so as to realize tensor calculation.

[0058] Optionally, the processor 100 may further include a third storage unit 105. The tensor computation engine 102 may also be used to store the computation results of tensor data into the third storage unit 105. Optionally, the third storage unit 105 may be implemented as a register file to store the computation results corresponding to different threads, different tensor computation engines, or different processing units in the tensor computation engine. This application embodiment does not limit this.

[0059] In some embodiments, the result of tensor operations can also be used as input for subsequent operations. For example, in the field of deep learning technology, for two connected model layers in a neural network model, the output data of the first model layer (i.e., the result of tensor operations) will be used as the input data of the second model layer. Therefore, it is necessary to convert the data format of the output data into the data format supported by the second model layer, and to package the converted output data into tensor computation engine 102 for subsequent operations.

[0060] For example, referring to Figure 2, matrix 201 has a size of M*K, matrix 202 has a size of K*N, and matrix 203 is the result of the operation between matrix 201 and matrix 202. Performing a matrix multiplication operation on matrix 201 and matrix 202 yields matrix 203. Matrix 203 can be used to represent the result of the matrix multiplication operation between matrix 201 and matrix 202, and its size is M*N. Each element in matrix 203 (i.e., the cell corresponding to matrix 203 in Figure 2) corresponds to a multiplication-accumulation operation result, and the data format of the multiplication-accumulation operation result is FP32. For example, the top-left element in matrix 203 is the result of performing a multiplication-accumulation operation on the element in the first row of matrix 201 (denoted as M0) and the element in the first column of matrix 202 (denoted as N0).

[0061] Each element in matrix 203 is stored in a different register. For example, the first four columns of elements in matrix 203 are stored in register R0, the second four columns are stored in register R1, the third four columns are stored in register R2, and the last four columns are stored in register R3.

[0062] T0 to T31 are 32 threads started by the format conversion unit (not shown in Figure 1). These 32 threads are used to perform data format conversion and packing on the encoded data of each element in matrix 203. Data format conversion refers to the process of converting data from one data format to another, and packing refers to the process of packing multiple data together to form continuous data. The format conversion unit can be implemented as at least one of the following: SIMD, SIMT (Single Instruction Multiple Thread), or a vector unit on the NPU.

[0063] Taking thread T0 as an example, thread T0 needs to perform data format conversion and packing on the first element of R0, the first element of R1, the first element of R2, and the first element of R3. For example, thread T0 can convert the data format of each element corresponding to T0 from FP32 to FP8, obtaining four elements in FP8 data format, and then pack the four elements in FP8 data format into a single register in sequence. That is, the four elements in FP32 data format form 32-bit encoded data, which includes four 8-bit encoded data, each 8-bit encoded data corresponding to one element (i.e., the result of the above operation).

[0064] When sorting by the columns of matrix 203, the order of the elements in matrix 203 is: columns N0 to N15. However, after processing by the format conversion unit (i.e., data format conversion and packaging), the order of the elements becomes: N0, N4, N8, N12, N1, N5, N9, N13, N2, N6, N10, N14, N3, N7, N11, and N15. This processed order differs from the original order. If the processed elements are needed for subsequent calculations, their original order must be restored, increasing the complexity of subsequent calculations (e.g., needing to readjust the original order for calculation).

[0065] To address the issue of inconsistent arrangement order, this embodiment of the application supports the tensor storage engine 101 automatically rearranging the arrangement order of tensors in the second storage unit 104 during the process of reading tensors from the first storage unit 103. This ensures that the arrangement order of the calculation results after processing by the format conversion unit is consistent with the arrangement order of the calculation results obtained by normally performing tensor calculations (i.e., without the rearrangement process), thereby reducing the complexity of subsequent calculations. In other words, there is no need to readjust the original arrangement order for calculation.

[0066] For example, the coded data of each element column in matrix 202 is arranged in the first storage unit 103 in the order of coded data of column N0 to coded data of column N15. That is, the coordinate address order of the coded data of each element column is: the coordinate address of the coded data of column N0 to the coordinate address of the coded data of column N15, forming the coordinate address information of tensor data. The tensor storage engine 101 can adjust the coordinate address order of the coded data of each element column to: the coordinate address of the coded data of column N0, the coded data of column N4, and the coordinate address of the coded data of column N15. Based on the coordinate addresses of the N0 column elements, the coordinate addresses of the encoded data of the N8 column elements, ..., the coordinate addresses of the encoded data of the N15 column elements, the adjusted coordinate address information is obtained. The tensor storage engine 101 reads the encoded data of each element column from the first storage unit 103 into the second storage unit 104 in sequence according to the adjusted coordinate address information. The arrangement order of the encoded data of each element column in the second storage unit 104 is: encoded data of the N0 column elements, encoded data of the N4 column elements, encoded data of the N8 column elements, ..., encoded data of the N15 column elements.

[0067] After the tensor computation engine 102 performs logical operations on the encoded data of each element column, the order of the operation results corresponding to each element column is as follows: operation results of the encoded data of the N0 column element, operation results of the encoded data of the N4 column element, operation results of the encoded data of the N8 column element, ..., operation results of the encoded data of the N15 column element (i.e. matrix 203).

[0068] The format conversion unit converts and packages the data format of the operation results corresponding to each element column. Taking thread T0 as an example, thread T0 needs to convert and package the data format of the first element in R0 (corresponding to column N0), the first element in R1 (corresponding to column N1), the first element in R2 (corresponding to column N2), and the first element in R3 (corresponding to column N3). For example, thread T0 can convert the data format of each element corresponding to T0 from FP32 to FP8, obtaining four elements in FP8 data format. Then, it packages the four elements in FP8 data format into a single register in sequence, thus restoring the first element in column N0, column N1, column N2, and column N3 to a continuous arrangement.

[0069] Following this logic, after data format conversion and packaging, the order of the operation results corresponding to each element column changes to: the operation results of column N0 to column N15. This ensures that the order of the operation results after processing by the format conversion unit is consistent with the order of the operation results obtained by normally performing tensor calculations (i.e., without the rearrangement process).

[0070] This application embodiment supports the tensor storage engine 101 to automatically rearrange tensors during tensor transfer without the need to develop additional software design to support the additional rearrangement process. This reduces the complexity of the tensor processing flow in the processor and avoids the development costs associated with software design, thereby reducing the processor's development costs.

[0071] In some embodiments, taking the first tensor as an example, the process of tensor rearrangement implemented by the tensor storage engine 101 is described.

[0072] The aforementioned first tensor can refer to any tensor to be processed by the processor 100. Optionally, the first tensor can be a tensor composed of a subset of elements from a certain tensor. For example, when an M×N tensor (i.e., a matrix) is uniformly divided into k (M / k)×N blocks, the first tensor can refer to a tensor composed of elements corresponding to any block among the k blocks. The first tensor can also be an undivided tensor, such as an M×N tensor. The embodiments of this application do not limit the source of the first tensor.

[0073] The embodiments of this application do not limit the elements in the first tensor. Exemplarily, the first tensor can be used to store model parameters of a neural network model, such as model parameters (e.g., weight parameters, bias parameters) of various model layers (e.g., convolutional layers, fully connected layers, attention mechanism layers, etc.) in a large model. The first tensor can also be used to store feature data in deep learning tasks, such as image feature data, speech feature data, text feature data, video feature data, etc. The first tensor can also be used to store coefficients in functions or formulas.

[0074] Optionally, the first storage unit 103 stores a first tensor, which includes multiple elements arranged in a first permutation order. That is, the first storage unit 103 stores tensor data of the first tensor, and the tensor data includes multiple encoded data, each encoded data representing an element in the first tensor. For ease of understanding and explanation, in this embodiment, "first tensor" refers to the tensor data of the first tensor, and "element" refers to the encoded data of an element.

[0075] Optionally, the first arrangement order of the multiple elements in the first storage unit 103 is the same as the arrangement order of the multiple elements in the first tensor (i.e., no staggered arrangement as described below); the first arrangement order of the multiple elements in the first storage unit 103 may also be different from the arrangement order of the multiple elements in the first tensor, such as staggered arrangement as described below. This application embodiment does not limit the first arrangement order, which can be set and adjusted according to actual usage requirements.

[0076] The tensor data of the first tensor is used to represent the first tensor. The tensor data of the first tensor can be composed of a coded data corresponding to each element in the first tensor. Each coded data can be used to determine the value of an element.

[0077] Optionally, the total number of bits of the encoded data for each element can be determined based on the first data format. For example, the total number of bits of the encoded data for each element is the same as the total number of bits (i.e., the total number of bits) corresponding to the first data format. This application embodiment does not limit the format type of the first data format. The format type of the first data format may include at least one of the following: floating-point type, integer type. For example, the first data format may include at least one of the following: FP8, FP16, BF16, FP6, FP4, FP32, TF32, U8, S8, INT8, INT16. For example, when the first data format is FP4, each element can be represented using 4 bits of data to form encoded data; when the first data format is FP6, each element can be represented using 6 bits of data to form encoded data.

[0078] Please refer to Figure 3. The process of tensor rearrangement implemented by tensor storage engine 101 may include the following: tensor storage engine 101 is used to read multiple elements from the first storage unit 103 into the second storage unit 104 according to the second arrangement order to obtain rearranged tensors.

[0079] The second arrangement order differs from the first arrangement order. The second arrangement order can be used to indicate the reading order of multiple elements in the first storage unit 103, that is, the arrangement order of multiple elements in the second storage unit 104. The rearranged tensor is obtained by rearranging multiple elements in the first tensor according to the second arrangement order.

[0080] In one example, the tensor storage engine 101 can rearrange the elements in the first tensor by rearranging the order of the coordinate addresses corresponding to each element in the first storage unit 103. Therefore, this embodiment may further include the following:

[0081] S1, Tensor storage engine 101 is used to obtain the coordinate address information of the first tensor, which includes the coordinate addresses of multiple elements in the first storage unit 103.

[0082] The coordinate addresses of multiple elements in the first storage unit 103 are essentially the coordinate addresses of the encoded data of each element in the first storage unit 103.

[0083] This application embodiment does not limit the reading granularity of the first tensor, which can be set and adjusted according to actual usage requirements. The reading granularity of the first tensor can refer to the number of elements read by the tensor storage engine 101 each time.

[0084] For example, for the tensor data of the first tensor, the data to be read each time can be determined according to the reading granularity of the first tensor. The data to be read can be the entire tensor data, that is, the data to be read includes the encoded data corresponding to all elements in the first tensor. The data to be read can also refer to a portion of the tensor data, that is, the data to be read includes the encoded data corresponding to a portion of the elements in the first tensor.

[0085] For example, referring to Figure 2, for matrix 202, the encoded data corresponding to the 16 columns of elements can be determined as the data to be read, or the encoded data corresponding to each 8 columns of elements can be determined as the data to be read, or the encoded data corresponding to each row of elements can be determined as the data to be read. This application embodiment does not limit this.

[0086] The coordinate address information of the first tensor can be used to indicate the storage location of the tensor data of the first tensor in the first storage unit 103, and the coordinate address information of the data to be read can be used to indicate the storage location of the data to be read in the first storage unit 103. The coordinate address information of the data to be read can be composed of the coordinate addresses of multiple encoded data in the data to be read. The coordinate address of the encoded data of each element can be used to indicate the storage location of the encoded data of that element in the first storage unit 103.

[0087] The coordinate address can be a physical address identifier to point to a specific storage area in the first storage unit 103. For example, the size of each storage area is usually 1 byte, which means it can store 1 byte (i.e., 8 bits) of data. The physical address is an absolutely fixed address.

[0088] Tensor storage engine 101 can access data in first storage unit 103 via coordinate addresses. For example, coordinate addresses can be used to indicate the storage area of ​​encoded data in first storage unit 103, and tensor storage engine 101 can read the encoded data from that storage area based on the coordinate addresses.

[0089] In one example, the tensor storage engine 101 is used to obtain control instructions for the first tensor; according to the control instructions, it obtains the basic information of the first tensor from the fourth storage unit 106, the basic information including at least one of the following: dimension information, coordinate information, arrangement method, data format; based on the basic information of the first tensor, it obtains the coordinate address information of the first tensor.

[0090] For example, taking the data to be read as an example, the tensor storage engine 101 is used to obtain control instructions for the data to be read; according to the control instructions, it obtains the basic information of the data to be read from the fourth storage unit 106, the basic information including at least one of the following: dimension information, coordinate information, arrangement method, data format; based on the basic information of the data to be read, it obtains the coordinate address information of the data to be read.

[0091] Control instructions for the first tensor instruct the tensor storage engine 101 to read tensor data from the first storage unit 103, and control instructions for the data to be read instruct the tensor storage engine 101 to read the data to be read from the first storage unit 103. For example, the tensor storage engine 101 can be used to parse the control instructions, obtain access information for the data to be read, obtain basic information about the data to be read based on the access information, and determine the coordinate address information of the data to be read in the first storage unit 103 based on the basic information.

[0092] The aforementioned access information is used to indicate the storage unit that the tensor storage engine 101 needs to access, and the data that needs to be accessed within that storage unit. For example, the access information for the data to be read can be used to indicate the basic information that the tensor storage engine 101 needs to access. This basic information includes at least one of the following: dimension information, coordinate information, arrangement method, and data format. Specifically, the dimension information of the data to be read indicates the dimension of the data to be read; the coordinate information of the data to be read indicates the position of the data to be read in the tensor data, such as the coordinates of the first element of the data to be read in the tensor data; the arrangement method of the data to be read indicates the arrangement of each encoded data in the data to be read; and the data format of the data to be read refers to the data format corresponding to each encoded data in the data to be read.

[0093] The storage unit indicated by the access information of the data to be read is the storage unit where the basic information of the data to be read is stored. When the basic information of the data to be read is distributed in different storage units, the access information of the data to be read can instruct the tensor storage engine 101 to access different storage units to read the basic information of the data to be read.

[0094] The fourth storage unit 106 is used to store the basic information of the tensor (including the data to be read). Optionally, the fourth storage unit 106 can be implemented as at least one of the following: a register, a register file, or constant memory. For example, referring to Figure 3, since the basic information of the data to be read can be stored in different storage units, the fourth storage unit 106 can include both a register file and constant memory. The register file can be used to store dimension information (i.e., block_dim), coordinate information (i.e., block_pos), etc., while constant memory can be used to store tensor descriptors (i.e., tensor_info). A tensor descriptor is an object used to describe information such as the arrangement and data format.

[0095] The tensor storage engine 101 can read dimension and coordinate information from the register file and tensor description from constant memory based on the access information corresponding to the data to be read, in order to obtain the arrangement and data format. Then, based on the dimension, coordinate, arrangement, and data format, it calculates the coordinate address information of the data to be read.

[0096] For example, dimension information, arrangement method and data format can be used to determine the calculation method of coordinate address information. Coordinate information can be used to calculate coordinate address information according to the calculation method. For example, the calculation formula of coordinate address information of the data to be read can be as follows: block_global_address=tensor_desc.global_address+block_pos[0]+block_pos[1]*tensor_desc.stride[0]+block_pos[2]*tensor_desc.stride[1]+block_pos[3]*tensor_desc.stride[2]+block_pos[4]*tensor_desc.stride[3];

[0097] Wherein, block_global_address is the coordinate address information of the data to be read in the first storage unit 103, tensor_desc.global_address is the coordinate address information of the tensor data in the first storage unit 103, block_pos[] refers to the coordinate information of the data to be read (such as the coordinates of the data to be read in a certain dimension), and tensor_desc.stride[] refers to the tensor descriptor of the first tensor (such as the stride indicated by the data format, i.e., the number of elements between adjacent indices in the same dimension).

[0098] In this embodiment, the tensor storage engine 101 can determine the coordinate address information of the tensor in the first storage unit 103 based on the basic information of the tensor, avoiding a large amount of address calculation and boundary checks, thereby realizing fast access to the tensor.

[0099] S2, Tensor storage engine 101 is also used to adjust the order of coordinate addresses included in the coordinate address information to obtain the adjusted coordinate address information.

[0100] The coordinate addresses in the coordinate address information are arranged sequentially according to the order of the encoded data corresponding to each coordinate address in the first storage unit 103. For example, referring to Figure 2, when the encoded data of matrix 202 is the data to be read (or tensor data), the coordinate addresses of the encoded data of each element column (N0 to N15) are ordered as follows: coordinate address of the encoded data of the element in column N0, coordinate address of the encoded data of the element in column N1, ..., coordinate address of the encoded data of the element in column N15. The coordinate address of the encoded data of each element in each column is composed of the coordinate addresses of the encoded data of each element in that column.

[0101] Optionally, the sorting order of the coordinate addresses in the adjusted coordinate address information is different from the arrangement order of the coordinate addresses in the coordinate address information. For example, referring to Figure 2, the adjusted coordinate address information can be: the coordinate address of the encoded data of column N0, the coordinate address of the encoded data of column N4, the coordinate address of the encoded data of column N8, ..., the coordinate address of the encoded data of column N15.

[0102] Optionally, the tensor storage engine 101 can read the encoded data (i.e., elements) corresponding to each coordinate address from the first storage unit 103 in the order of the coordinate addresses, so as to rearrange the first tensor during the reading process of the first tensor.

[0103] For example, for the first and second encoded data in the data to be read, if the coordinate address information is the first coordinate address and the second coordinate address, the tensor storage engine 101 can first read the first encoded data from the first storage unit 103 according to the first coordinate address, and then read the second encoded data from the first storage unit 103 according to the second coordinate address. The first coordinate address is used earlier than the second coordinate address.

[0104] When the adjusted coordinate address information is the second coordinate address and the first coordinate address, the tensor storage engine 101 can first read the second encoded data from the first storage unit 103 according to the second coordinate address, and then read the first encoded data from the first storage unit 103 according to the first coordinate address. The second coordinate address is used earlier than the first coordinate address.

[0105] Optionally, the tensor storage engine 101 can sequentially store the encoded data into the second storage unit 104 according to the order in which the encoded data is read, so as to adjust the arrangement order of each encoded data in the second storage unit 104 in the data to be read. After at least one piece of data to be read has been read, the arrangement order of each encoded data in the tensor data in the second storage unit 104 can be adjusted.

[0106] For example, based on the above embodiments, for coordinate address information, the position of the first encoded data in the second storage unit 104 is before the position of the second encoded data in the second storage unit 104; while for the adjusted coordinate address information, the position of the second encoded data in the second storage unit 104 is before the position of the first encoded data in the second storage unit 104.

[0107] In one example, to support the tensor storage engine 101 in rearranging the first tensor (i.e., tensor data), this embodiment adds rearrangement parameters corresponding to the tensor data in the fourth storage unit 106, such as one rearrangement parameter for each piece of data to be read. The rearrangement parameters are used to adjust the order of coordinate addresses. For example, the rearrangement parameters can be used to adjust the order of coordinate addresses of each encoded data in the coordinate address information to obtain the adjusted coordinate address information.

[0108] For example, the tensor storage engine 101 is also used to obtain rearrangement parameters from the fourth storage unit 106, the rearrangement parameters being used to adjust the arrangement order of coordinate addresses; when the rearrangement parameters indicate rearranging coordinate addresses, the arrangement order of coordinate addresses included in the coordinate address information is adjusted according to the rearrangement parameters to obtain the adjusted coordinate address information.

[0109] Optionally, the tensor storage engine 101 is further configured to, upon receiving a control instruction for the first tensor, obtain rearrangement parameters of the first tensor from the fourth storage unit 106. The rearrangement parameters of the first tensor include rearrangement parameters corresponding to at least one piece of data to be read. Alternatively, the tensor storage engine 101 is further configured to obtain the rearrangement parameters of the first tensor from the fourth storage unit 106 according to access information corresponding to the control instruction. That is, the access information is also used to indicate which storage unit the tensor storage engine 101 obtains the rearrangement parameters from, which is not limited in this embodiment.

[0110] Optionally, the rearrangement parameters include a third parameter, which indicates whether the tensor storage engine 101 should rearrange the multiple coordinate addresses corresponding to the first tensor. For example, when the third parameter is a first value, the tensor storage engine 101 rearranges the multiple coordinate addresses corresponding to the first tensor; when the third parameter is a second value, the tensor storage engine 101 does not rearrange the multiple coordinate addresses corresponding to the first tensor. This application embodiment does not limit the first and second values; for example, the first value can be 1 and the second value can be 0. In some embodiments, when the result of the operation on the first tensor needs to be used for subsequent calculations, the third parameter can be set to the first value. For example, when the result of the matrix multiplication operation A*B is used for subsequent calculations, the third parameter corresponding to matrix B can be set to 1, while the third parameter corresponding to matrix A can be set to 0. Rearranging coordinate addresses refers to the process of adjusting the order of coordinate addresses.

[0111] For example, referring to Figure 3, the register file stores the reordering parameters (reorder_stride) of the first tensor. The tensor storage engine 101 is used to read the reordering parameters of the first tensor from the register file when a control instruction for the first tensor is received. If the third parameter in the reordering parameters is the first value, the arrangement order of multiple coordinate addresses corresponding to the first tensor is adjusted according to the reordering parameters to obtain the adjusted coordinate address information of the first tensor. Similarly, the arrangement order of multiple coordinate addresses corresponding to the data to be read is adjusted according to the reordering parameters of the data to be read to obtain the adjusted coordinate address information of the data to be read. After obtaining the adjusted coordinate address information of all the data to be read, the adjusted coordinate address information of the first tensor is obtained.

[0112] Based on the rearrangement parameters, the tensor storage engine 101 can rearrange the coordinate addresses, thereby rearranging multiple elements (i.e., encoded data) corresponding to the first tensor. This simplifies the tensor rearrangement process without performance loss, which helps reduce the development complexity of the processor and thus reduces the development cost of the processor.

[0113] In one example, the rearrangement parameters of the first tensor are related to the target data format to which the operation result corresponding to the first tensor needs to be converted. That is, the correspondence between the rearrangement parameters and the target data format can be set, and the correspondence can be stored in the fourth storage unit 106. Then, the tensor storage engine 101 can also be used to obtain the rearrangement parameters from the fourth storage unit 106 based on the target data format to which the operation result corresponding to the first tensor needs to be converted. For example, the rearrangement parameters corresponding to the target data format are determined as the rearrangement parameters of the first tensor, so as to read the rearrangement parameters corresponding to the first tensor from the fourth storage unit 106.

[0114] This application embodiment does not limit the target data format. For example, the target data format may include at least one of the following: FP16, FP8, and FP4. For instance, when the target data format is FP16, the tensor storage engine 101 uses the rearrangement parameters corresponding to the FP16 data format in the fourth storage unit 106 as the rearrangement parameters of the first tensor. Thus, by using the target data format, the rearrangement parameters can be accurately obtained, avoiding incorrect rearrangement of coordinate addresses. The target data format corresponding to the first tensor can be set and adjusted based on actual usage requirements, and this application embodiment does not limit this.

[0115] In this embodiment, the operation result corresponding to the first tensor can refer to the operation result obtained by the tensor calculation engine 102 performing logical operations on the rearranged first tensor (i.e., the rearranged tensor mentioned above). Performing logical operations on the rearranged tensor is essentially performing logical operations on the tensor data of the rearranged tensor. The tensor data of the rearranged tensor is obtained by combining the encoded data of each element in the rearranged tensor according to the second arrangement order.

[0116] In one example, the rearrangement parameters may also include a first parameter and a second parameter. The first parameter indicates the relationship between the total number of bits in the data format of the operation result and the total number of bits in the target data format, and the second parameter indicates the rearrangement granularity of the first tensor.

[0117] Optionally, the quotient between the total number of bits in the data format of the result and the total number of bits in the target data format can be used as the first parameter. For example, if the data format of the result is FP32 and the target data format is FP16, 32 / 16 = 2 can be used as the first parameter.

[0118] The rearrangement granularity is used to indicate the span involved in a single rearrangement of the operation result in a certain dimension, such as the number of elements, rows, and columns in a certain dimension of the first tensor involved in each rearrangement. For example, referring to Figure 2, for matrix 203, with the data format of the operation result being FP32 and the target data format being FP8, a single packing in the first dimension of matrix 203 (i.e., dim0, where M represents the direction of data change) involves 16 columns of elements. That is, the 16 columns of elements in the first dimension of matrix 202 need to be rearranged so that after packing, the order of the operation results is maintained as follows: the operation result of the encoded data of column N0, the operation result of the encoded data of column N4, the operation result of the encoded data of column N8, ..., the operation result of the encoded data of column N15. This span is 16, so 16 can be determined as the second parameter.

[0119] For example, taking the data format of the calculation result as FP32, the rearrangement parameter settings may include at least one of the following:

[0120] 1. When the target data format is FP4, the first parameter is 8 and the second parameter is 32.

[0121] 2. When the target data format is FP8, the first parameter is 4 and the second parameter is 16.

[0122] 3. When the target data format is FP16, the first parameter is 2 and the second parameter is 8.

[0123] In one example, the process of obtaining the adjusted coordinate address information after obtaining the rearrangement parameters may include the following.

[0124] (1) Tensor storage engine 101 is used to obtain the first intermediate parameter and the second intermediate parameter for each coordinate address in the coordinate address information, based on the arrangement position of the coordinate address in the coordinate address information and the second parameter.

[0125] Optionally, based on the packaging method of the calculation results, during the rearrangement of the first tensor, rearrangement can be performed using individual, entire row, or entire column elements of the first tensor in a certain dimension as units. For example, referring to Figure 2, for matrix 202, since the calculation results are packaged in the first dimension, rearrangement can be performed using entire columns (i.e., N0 to N15) of matrix 202 in the second dimension as units, or rearrangement can be performed using a single row of elements of matrix 202 in the first dimension. This application embodiment does not limit this.

[0126] Therefore, when obtaining the adjusted coordinate address information, if the data to be read is all the encoded data of matrix 202 (i.e., the first tensor), the tensor storage engine 101 can be used to rearrange the coordinate addresses at a rearrangement granularity (e.g., N0 to N15), with the coordinate addresses corresponding to the entire column of elements as the unit. If the data to be read is the encoded data corresponding to a row of elements in matrix 202 (i.e., the first tensor), the tensor storage engine 101 can be used to rearrange the coordinate addresses corresponding to each element in a row at a rearrangement granularity (e.g., N0 to N15).

[0127] For example, referring to Figure 2, the coordinate address information of matrix 202 is: the coordinate address of column N0, the coordinate address of column N1, ..., the coordinate address of column N15. Then the arrangement of each coordinate address in the coordinate address information is as follows: 0, 1, 2, ..., 15.

[0128] The arrangement of coordinate addresses can be used to indicate the order in which the corresponding encoded data is read using coordinate addresses. Therefore, by adjusting the arrangement of coordinate addresses in the coordinate address information, the order in which the encoded data is acquired can be adjusted, thereby adjusting the storage order of the encoded data and thus reordering the encoded data. Therefore, the reordering of tensors can be achieved by rearranging the order of coordinate addresses.

[0129] The aforementioned first and second intermediate parameters can be used to adjust the arrangement of coordinate addresses, thereby adjusting the order of each coordinate address in the coordinate address information. For example, the calculation formula for the first intermediate parameter can be expressed as follows: uint32_t outer = source_pos / outer_stride * outer_stride;

[0130] Wherein, outer is the first intermediate parameter, source_pos is the position of the coordinate address (e.g., 0 to 15 corresponding to matrix 202), outer_stride is the second parameter (e.g., 16 corresponding to matrix 202), and uint32_t represents an unsigned 32-bit integer.

[0131] For example, referring to Figure 2, if the coordinate address of the element in column N0 is 0, then outer = 0 / 16*16 = 0, meaning the position of the element in column N0 belongs to the 0th rearrangement granularity (i.e., N0 to N15); if the coordinate address of the element in column N1 is 1, then outer = 1 / 16*16 = 0, meaning the coordinate address of the element in column N1 also belongs to the 0th rearrangement granularity (i.e., N0 to N15), and so on. It can be determined that the coordinate addresses of the elements in columns N0 to N15 all belong to the 0th rearrangement granularity. In other words, the tensor storage engine 101 needs to arrange the coordinate addresses of the elements in columns N0 to N15 in the same rearrangement process.

[0132] Optionally, the formula for calculating the second intermediate parameter can be expressed as follows: uint32_t inner = source_pos % outer_stride;

[0133] Here, inner is the second intermediate parameter, source_pos is the position of the coordinate address (e.g., 0 to 15 corresponding to matrix 202), and outer_stride is the second intermediate parameter (e.g., 16 corresponding to matrix 202).

[0134] For example, referring to Figure 2, if the coordinate address of the element in column N0 is at position 0, then inner = 0 % 16 = 0, meaning the coordinate address of the element in column N0 is in position 0 in the 0th rearrangement granularity; if the coordinate address of the element in column N1 is at position 1, then inner = 1 % 16 = 1, meaning the coordinate address of the element in column N1 is in position 1 in the 0th rearrangement granularity; if the coordinate address of the element in column N2 is at position 2, then inner = 2 % 16 = 2, meaning the coordinate address of the element in column N2 is in position 2 in the 0th rearrangement granularity, and so on, the position of the coordinate addresses of the elements in columns N0 to N15 in the 0th rearrangement granularity can be determined.

[0135] (2) Tensor storage engine 101 is used to obtain a third intermediate parameter based on the first parameter, the first intermediate parameter, the second intermediate parameter and the second parameter. The third intermediate parameter is used to indicate the arrangement position of the coordinate address in the adjusted coordinate address information. Based on the third intermediate parameter corresponding to each coordinate address included in the coordinate address information, the adjusted coordinate address information is obtained.

[0136] Optionally, for the first coordinate address in the coordinate address information, the third intermediate parameter corresponding to the first coordinate address can be used to indicate the arrangement position of the second coordinate address to which the first coordinate address needs to be replaced. That is, based on the third intermediate parameter, the arrangement position of the second coordinate address can be replaced with the arrangement position of the first coordinate address. The first coordinate address can refer to any coordinate address in the coordinate address information.

[0137] For example, the formula for calculating the third intermediate parameter can be expressed as follows:

[0138] uint32_t target=(inner*inner_stride) / outer_stride+(inner*inner_stride)%outer_stride+outer; where target is the third intermediate parameter.

[0139] For example, referring to Figure 2, if the coordinate address of the element in column N0 is 0, then the third intermediate parameter corresponding to the coordinate address of the element in column N0 is: target = (0*4) / 16 + (0*4)%16 + 0 = 0, meaning the coordinate address of the element in column N0 remains unchanged (i.e., it is read in the 1st position). If the coordinate address of the element in column N1 is 1, then the third intermediate parameter of the coordinate address of the element in column N1 is: target = (1*4) / 16 + (1*4)%16 + 0 = 4, and the coordinate address of the element in column N4 is 4, meaning the coordinate address of the element in column N4 is updated to 1 (i.e., it is read in the 2nd position). If the position of the coordinate address corresponding to the element in column N2 is 2, then the third intermediate parameter corresponding to the coordinate address of the element in column N2 is: target = (2*4) / 16 + (2*4)%16 + 0 = 8. The position of the coordinate address of the element in column N8 is 8, meaning the position of the coordinate address of the element in column N8 is updated to 3 (i.e., it is the 3rd position in the reading order). Similarly, the third parameter is as follows: 0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15. Therefore, the adjusted coordinate address information is: the coordinate address of the element in column N0, the coordinate address of the element in column N4, the coordinate address of the element in column N8, the coordinate address of the element in column N12, ..., the coordinate address of the element in column N15. It should be noted that the third intermediate parameter is calculated based on the position of the coordinate address in the coordinate address information, not the position of the coordinate address in the adjusted coordinate address information.

[0140] The embodiments of this application can avoid affecting the calculation results by rearranging the coordinate addresses according to the single, whole row or whole column elements of the tensor in a certain dimension.

[0141] In one example, the rearrangement of coordinate addresses is related to the arrangement of the first tensor. The arrangement of the first tensor refers to the arrangement of the elements in the first tensor, such as the arrangement of the first tensor including at least one of the following: non-interleave and interleave.

[0142] The main difference between non-interleaved and interleaved arrays lies in how elements are stored and accessed. When the first tensor is non-interleaved, the encoded data of the elements is stored according to their natural order in memory, typically from left to right and top to bottom. When the first tensor is interleaved, the encoded data of the elements is interleaved according to a specific pattern, rather than being ordered naturally, to optimize computational efficiency or adapt to specific hardware architectures.

[0143] For example, when the first tensor is arranged in an interleaved manner, the tensor storage engine 101 adjusts the order of the coordinate addresses included in the coordinate address information according to the rearrangement parameters and the first dimension corresponding to the first tensor to obtain the adjusted coordinate address information.

[0144] Optionally, the first dimension can refer to the dimension where the data change direction is horizontal, denoted as dim0. For example, referring to Figure 4, for matrix 401 (such as the first tensor), the continuous change direction of the K elements in each element row is the data change direction of the first dimension. According to the first dimension, matrix 401 can be divided into multiple element rows. If the arrangement of matrix 401 in the first storage unit 103 is non-interlaced, then dim0 is M (i.e., corresponding to M element rows) and dim1 is K (K element columns). If its arrangement in the first storage unit 103 is interlaced, and the size of the interlaced arrangement X is K / 2, then dim0 is M, dim1 is (k0, k1), and matrix 401 is divided into two sub-matrices (2 M*X) for storage. dim1 as a whole includes two parts, such as storing k0 first and then storing k1.

[0145] For example, referring to Figure 5, matrix 501 is arranged in an alternating pattern. Based on the first dimension dim0, matrix 501 can be divided into 9 element rows, with the coordinate addresses of these 9 rows arranged sequentially from 0 to 8. When the first parameter of matrix 501 is 2 and the second parameter is 8 (i.e., the rearrangement granularity is 8), based on 2 and 8, the coordinate addresses of the first 8 element rows (i.e., the 0th rearrangement granularity) can be adjusted to: 0, 2, 4, 6, 1…, 7. Since element row 8 is the first element in the 1st rearrangement granularity, the coordinate address of element row 8 remains unchanged. That is, the adjusted coordinate address information of matrix 501 is: coordinate address of element row 0, coordinate address of element row 2, coordinate address of element row 4,…, coordinate address of element row 7, and coordinate address of element row 8.

[0146] Optionally, the tensor storage engine 101 is also used to adjust the order of the coordinate addresses included in the coordinate address information according to the rearrangement parameters and the second dimension corresponding to the first tensor when the first tensor is arranged in a non-interleaved manner, so as to obtain the adjusted coordinate address information.

[0147] In this design, the direction of data change for the first dimension differs from that for the second dimension. For example, the directions of data change for the first and second dimensions are perpendicular to each other. Optionally, the second dimension can refer to a dimension where the direction of data change is vertical, denoted as dim1.

[0148] For example, referring to Figure 2, for matrix 202 (such as the data to be read), if its arrangement in the first storage unit 103 is non-interleaved, then dim0 is K (K rows of elements) and dim1 is N (N columns of elements). The elements in its vertical direction (i.e., the N-direction) are consecutive, forming the second dimension. According to this second dimension, matrix 202 can be divided into multiple columns (i.e., columns N0 to N15). Based on the rearrangement parameters, the coordinate addresses of each group of encoded data in matrix 202 can be adjusted using the column as the unit, resulting in the adjusted coordinate address information of matrix 202. The adjusted coordinate address information is: the coordinate address of column N0, the coordinate address of column N4, the coordinate address of column N8, the coordinate address of column N12, ..., the coordinate address of column N15.

[0149] This application embodiment rearranges the coordinate addresses of entire rows or columns of elements in a tensor according to a certain dimension, which can avoid affecting the calculation results and improve the efficiency of coordinate address rearrangement.

[0150] S3. The tensor storage engine 101 is also used to read multiple elements from the first storage unit 103 into the second storage unit 104 in sequence according to the arrangement order of each coordinate address included in the adjusted coordinate address information, so as to obtain a rearranged tensor.

[0151] Optionally, the encoded data included in the tensor data of the first tensor is stored in the second storage unit 104 according to the adjusted coordinate address information to form the rearranged data corresponding to the rearranged tensor. The arrangement order of each encoded data in the rearranged data is consistent with the arrangement order of the coordinate addresses of each encoded data in the adjusted coordinate address information, that is, the second arrangement order mentioned above.

[0152] For example, referring to Figures 2 and 7, for matrix 202, the tensor storage engine 101 can sequentially read the encoded data of column N0, column N4, column N8, column N12, ..., column N15 from the first storage unit 103 based on the adjusted coordinate address information. The tensor storage engine 101, based on the adjusted coordinate address information, sequentially stores the encoded data of each column element into the second storage unit 104, thus obtaining the rearranged matrix 701 (i.e., the rearranged tensor) corresponding to matrix 202. The rearranged data corresponding to the rearranged matrix 701 is: the encoded data of column N0, column N4, column N8, column N12, ..., column N15.

[0153] In one example, the tensor storage engine 101 is also used to fill L preset elements after the multiple elements in the second storage unit 104 when the number of multiple elements is less than the second parameter; where L is the difference between the second parameter and the number of multiple elements, and L is a positive integer.

[0154] That is, if the number of groups of encoded data included in the tensor data of the first tensor is less than the second parameter, after the rearranged data in the second storage unit 104, L preset encoded data (i.e., the encoded data of preset elements) are filled.

[0155] Optionally, this application embodiment does not limit the method of dividing the number of groups for multiple elements (i.e., multiple encoded data). For example, referring to FIG2, when the first tensor includes multiple encoded data corresponding to element row 0 in matrix 202, the number of elements in element row 0 can be determined as the number of groups. When the first tensor includes multiple encoded data corresponding to element rows 0 to K in matrix 202, the number of element rows can be determined as the number of groups, with each element row as the unit. The number of groups can be used to indicate the number of the smallest units participating in a rearrangement process.

[0156] For example, referring to Figure 6, matrix 601 is arranged in an interleaved manner, with dim0 being M and dim1 being (k0, k1, k2). Matrix 601 can be divided into multiple element rows according to the first dimension. When dim1 = 0, 1, or 2, the tensor storage engine 101 needs to rearrange the coordinate addresses of element rows 0 to 6 (i.e., the number of groups is 7). If the first parameter is 2 and the second parameter is 8, then based on 2 and 8, the adjusted coordinate address information can be obtained: the coordinate address of element row 0, the coordinate address of element row 2, ..., the coordinate address of element row 5. However, since the second parameter indicates that the rearrangement granularity of matrix 601 is 8, after moving the encoded data of element rows 0 to 6 to the second storage unit 104, 8-7 = 1 preset encoded data can be added after the encoded data of element row 5 to form element row 7. This ensures that there is enough data to pack in subsequent processes.

[0157] The embodiments of this application do not limit the preset elements, which can be set and adjusted according to actual usage needs, such as 0.

[0158] In one example, the tensor storage engine 101 is also used to update the asynchronous barrier unit after the first tensor read is completed, so as to terminate the first tensor read by triggering an interrupt through the asynchronous barrier unit.

[0159] The asynchronous barrier unit is a unit built based on the asynchronous barrier mechanism. Optionally, after the tensor storage engine 101 finishes reading the last encoded data corresponding to the first tensor, it sends an interrupt signal to the asynchronous barrier unit. Based on the interrupt signal, the asynchronous barrier unit interrupts the tensor storage engine 101 from reading the first tensor from the first storage unit 103. Reading the first tensor essentially means reading the tensor data of the first tensor.

[0160] In summary, the technical solution provided in this application, during the process of the tensor storage engine reading the first tensor from the first storage unit to the second storage unit, supports the tensor storage engine to automatically rearrange multiple elements in the first tensor so that the third arrangement order of the operation results of the multiple elements in the third storage unit is the same as the first arrangement order of the multiple elements, without the need to develop additional software design to support the additional rearrangement process, thereby reducing the complexity of the tensor processing flow in the processor and avoiding the development costs brought by software design, thus reducing the development cost of the processor.

[0161] In addition, since this application implements tensor rearrangement at the existing hardware level, it avoids the impact of adding a rearrangement process on the processor's end-to-end performance, thereby improving the processor's end-to-end performance.

[0162] In some embodiments, after obtaining the rearranged tensor (i.e., rearranged data) of the first tensor, the embodiments of this application may further include the following.

[0163] 1. Tensor calculation unit 102 is used to perform logical operations on rearranged tensors to obtain the calculation results.

[0164] Performing logical operations on a rearranged tensor is essentially performing logical operations on the rearranged data of the rearranged tensor.

[0165] For example, referring to Figure 7, the tensor calculation unit 102 is used to perform matrix multiplication on the encoded data of matrix 201 and the rearranged data of rearranged matrix 701 to obtain the operation result 702 corresponding to rearranged matrix 701. The operation result 702 includes M*N multiplication and accumulation operation results.

[0166] Tensor calculation unit 102 can store the calculation result 702 into third storage unit 105. Optionally, if third storage unit 105 is implemented as a register file, third storage unit 105 may include four registers: registers R0 to registers R3.

[0167] The M*N multiply-accumulate results are stored in registers R0 to R3, with each register storing 8*4 multiply-accumulate results. For example, the 8 multiply-accumulate results corresponding to the elements in column N0, N1, N2, and N3 are stored sequentially in register R0.

[0168] Optionally, the data format of each multiply-accumulate operation result is FP32, that is, each multiply-accumulate operation result corresponds to 32 bits of data.

[0169] 2. The format conversion unit is used to convert the data format of each element in the calculation result to the target data format to obtain the calculation result after format conversion.

[0170] For example, referring to Figure 7, the SIMT unit is used to convert the data format of M*N multiply-accumulate operation results from FP32 to FP16, so as to obtain M*N multiply-accumulate operation results after format conversion.

[0171] In one example, the format conversion unit is used to convert the data format of the result element in the four registers of each thread to FP8 when the data format of the operation result is FP32.

[0172] Optionally, each element in the operation result can be referred to as a result element. For example, referring to Figure 7, the result of M*N multiply-accumulate operations is the result element corresponding to the operation result 702. When the fourth storage unit 105 is implemented as a register file, the third storage unit 105 may include 4 registers.

[0173] For example, referring to Figure 7, the SIMT unit can simultaneously start 32 threads: T0 to T31, and use these 32 threads to convert the data format of M*N multiply-accumulate operation results from FP32 to FP8 in parallel. The 32 threads can perform the data format conversion of 32 multiply-accumulate operation results in parallel each time, that is, each thread performs the data format conversion once for each register, thus completing the data format conversion of M*N multiply-accumulate operation results.

[0174] In one example, the format conversion unit is also used to convert the data format of the result element corresponding to each thread in the four registers to FP16 when the data format of the operation result is FP32.

[0175] For example, referring to Figure 7, the SIMT unit can simultaneously start 32 threads: T0 to T31, and use these 32 threads to convert the data format of M*N multiply-accumulate operation results from FP32 to FP16 in parallel. The 32 threads can perform the data format conversion of 32 multiply-accumulate operation results in parallel each time, that is, each thread performs the data format conversion once for each register, thus completing the data format conversion of M*N multiply-accumulate operation results.

[0176] 3. The format conversion unit is also used to store the converted calculation results into the third storage unit 105.

[0177] Optionally, the format conversion unit stores the format-converted calculation result in the third storage unit 105 according to the storage granularity of the third storage unit 105.

[0178] The embodiments of this application do not limit the storage granularity of the third storage unit 105. Optionally, the storage granularity of the third storage unit 105 includes 32 bits. For example, when the third storage unit 105 is implemented as a register file, the storage granularity of each register is 32 bits.

[0179] In one example, when the storage granularity of the third storage unit 105 is 32 bits and the target data format is FP8, the format conversion unit is used to package the result elements corresponding to each thread in the four registers into compressed data according to the storage granularity of the registers through multiple threads in parallel, so as to store them in any of the four registers in sequence.

[0180] For example, referring to Figure 2, since the data size of the multiply-accumulate operation result after data format conversion is 8 bits, the multiply-accumulate operation results of the four data format conversions can be packaged together to obtain 32-bit compressed data. This 32-bit compressed data can be stored at the register storage granularity. Thus, 8*16 data format conversion multiply-accumulate operation results can be packaged into 32*32-bit compressed data. The 32*32-bit compressed data can be packaged into the same register simultaneously by 32 threads.

[0181] In one example, when the storage granularity of the third storage unit 105 is 32 bits and the target data format is FP16, the format conversion unit is used to package the result elements corresponding to each thread in the two registers in parallel through multiple threads into compressed data according to the storage granularity of the registers, so as to store them in either of the two registers in sequence.

[0182] For example, referring to Figure 7, since the data size of the multiply-accumulate operation result after data format conversion is 16 bits, the two multiply-accumulate operation results after data format conversion can be packaged together to obtain 32-bit compressed data. This 32-bit compressed data can be stored at the register storage granularity. Thus, 8*16 multiply-accumulate operation results after data format conversion can be packaged into 64*32-bit compressed data. The 64*32-bit compressed data can be divided into two 32*32-bit compressed data sets. The 32 threads can package the two 32*32-bit compressed data sets into two registers in two separate steps.

[0183] Optionally, the format-converted operation result includes the operation results of each element in the first tensor, and the third arrangement order of the operation results of each element in the third storage unit 105 is the same as the first arrangement order mentioned above. The operation result of an element can be referred to as the result element in the format-converted operation result. Optionally, the operation result corresponding to each element column in the first tensor is the operation result corresponding to each element in the element column.

[0184] For example, referring to Figure 7, when the storage granularity of the third storage unit 105 is 32 bits and the target data format is FP16, for the first row result element in the rearrangement matrix 701, T0 can pack the result elements of column N0 and column N1 into a 32-bit compressed data, T1 can pack the result elements of column N2 and column N3 into a 32-bit compressed data, T2 can pack the result elements of column N4 and column N5 into a 32-bit compressed data, T3 can pack the result elements of column N6 and column N7 into a 32-bit compressed data, and so on. The third arrangement order of the first row result elements after being packed is: the result elements of column N0, the result elements of column N1, the result elements of column N2, ..., the result elements of column N15. This is the same as the first arrangement order of the encoded data of column N0 to column N15 in the first storage unit 103.

[0185] In summary, the technical solution provided in this application embodiment, by supporting the tensor storage engine 101 to rearrange the tensor according to the rearrangement parameters when reading the tensor, can make the third arrangement order of the packaged result elements corresponding to the tensor consistent with the first arrangement order of the tensor in the first storage unit 103, without the need to develop additional software design to support additional rearrangement process, thereby reducing the complexity of the tensor processing process in the processor and avoiding the development cost brought by software design, thereby reducing the development cost of the processor.

[0186] In some embodiments, referring to FIG8, a graphics processor is used as an example. The graphics processor 800 may include multiple processor clusters (PCs) 801 (e.g., 4), each processor cluster 801 may include multiple processor execution engines (PXs) 802 (e.g., 4), and each processor execution engine 802 may include multiple processors (Ps) 803 (e.g., 2, i.e., core computing units).

[0187] Multiple processor execution engines 802 in each processor cluster 801 share an L2 cache 804 (i.e., the first storage unit 103 mentioned above). The main memory of the graphics processor 800 is implemented as an L3 cache 805.

[0188] Referring to Figure 9, each processor 803 may include: a tensor calculation engine 8031, a tensor storage engine 8032, an L1 cache 8033 (i.e., the second storage unit 104 mentioned above), and a register 8034 (i.e., the third storage unit 105 mentioned above).

[0189] The L2 cache 804 stores multiple encoded data (i.e., tensor data) corresponding to the first tensor. Each encoded data is generated from an element of the first tensor according to the FP8 data format, that is, each encoded data includes 8 bits of data.

[0190] The tensor storage engine 8032 is used to parse the control instruction received for the first tensor, obtain the access information of the first tensor, and determine the coordinate address information of the first tensor in the L2 cache 804 based on the access information.

[0191] Tensor storage engine 8032 is also used to read the rearrangement parameters of the first tensor from register 8035 (the fourth storage unit 106 mentioned above) after receiving a control instruction for the first tensor.

[0192] The Tensor Storage Engine 8032 is also used to rearrange the positions of each coordinate address in the coordinate address information corresponding to the first tensor according to the rearrangement parameters of the first tensor, so as to obtain the adjusted coordinate address information corresponding to the first tensor.

[0193] The tensor storage engine 8032 is also used to read each encoded data corresponding to the first tensor from the L2 cache 804 in sequence according to the arrangement order of each coordinate address included in the adjusted coordinate address information, and to store each encoded data into the L1 cache 8033 in sequence to obtain the rearranged data corresponding to the first tensor.

[0194] The tensor computation engine 8031 ​​is used to read the rearranged data corresponding to the first tensor from the L1 cache 8033, and to perform logical operations on the rearranged data corresponding to the first tensor to obtain the computation result; the computation result is stored in the register 8034.

[0195] The SIMT unit in processor 803 is used to perform data format conversion on the operation result, obtain the operation result after format conversion, and repackage the operation result after format conversion into register 8034 according to the storage granularity of 32 bits to obtain the compressed data corresponding to the operation result.

[0196] The order in which the operation results corresponding to each element in the compressed data are arranged in register 8034 is consistent with the order in which the encoded data of each element is arranged in L2 cache 804. This compressed data can be directly used in subsequent calculations without adjustment.

[0197] In some embodiments, taking a matrix as an example, the instructions involved in the tensor storage engine during tensor rearrangement are shown in Table 1 below.

[0198] Table 1

[0199] In this embodiment, the tensor storage engine can determine the coordinate address of the tensor data of the first tensor in the L2 cache 804 based on block_pos, block_dim and tensor_desc, and rearrange the arrangement of the coordinate addresses corresponding to the first tensor based on reorder_stride, so as to realize the rearrangement of the first tensor in the L1 cache 8033.

[0200] In summary, the technical solution provided by the embodiments of this application, by supporting the tensor storage engine 101 to rearrange the tensor according to the rearrangement parameters when reading the tensor, can make the arrangement order of the packaged result elements corresponding to the tensor consistent with the arrangement order of the tensor in the first storage unit 103, without the need to develop additional software design to support additional rearrangement process, thereby reducing the complexity of the tensor processing process in the processor and avoiding the development cost brought by software design, thereby reducing the development cost of the processor.

[0201] The following are embodiments of the method of this application. For details not disclosed in the embodiments of the method of this application, please refer to the embodiments above.

[0202] Please refer to Figure 10, which shows a flowchart of a tensor processing method applied to a processor provided in one possible implementation of this application. The components of the processor can be found in the description of the embodiments above. As shown in Figure 10, the method may include the following steps.

[0203] Step 1001: The tensor storage engine reads multiple elements from the first storage unit into the second storage unit according to the second permutation order to obtain a rearranged tensor. The second permutation order is different from the first permutation order.

[0204] In one example, the tensor storage engine obtains the coordinate address information of the first tensor, which includes the coordinate addresses of multiple elements in the first storage unit; adjusts the order of the coordinate addresses included in the coordinate address information to obtain the adjusted coordinate address information; according to the order of the coordinate addresses included in the adjusted coordinate address information, the multiple elements are sequentially read from the first storage unit to the second storage unit to obtain the rearranged tensor.

[0205] In one example, the processor also includes a fourth storage unit; the tensor storage engine obtains rearrangement parameters from the fourth storage unit, which are used to adjust the order of coordinate addresses; when the rearrangement parameters indicate that the coordinate addresses should be rearranged, the order of the coordinate addresses included in the coordinate address information is adjusted according to the rearrangement parameters to obtain the adjusted coordinate address information.

[0206] In one example, the rearrangement parameters include a first parameter and a second parameter. The first parameter is used to indicate the relationship between the total number of bits in the data format of the operation result and the total number of bits in the target data format. The second parameter is used to indicate the rearrangement granularity of the first tensor.

[0207] For each coordinate address in the coordinate address information, the tensor storage engine obtains a first intermediate parameter and a second intermediate parameter based on the coordinate address's position in the coordinate address information and the second parameter. The first intermediate parameter indicates the rearrangement granularity of the coordinate address, and the second intermediate parameter indicates the number of bits in the rearrangement granularity of the coordinate address. Based on the first parameter, the first intermediate parameter, the second intermediate parameter, and the second parameter, a third intermediate parameter is obtained. The third intermediate parameter indicates the coordinate address's position in the adjusted coordinate address information. Based on the third intermediate parameter corresponding to each coordinate address included in the coordinate address information, the adjusted coordinate address information is obtained.

[0208] In one example, when the target data format is FP4, the first parameter is 8 and the second parameter is 32; or, when the target data format is FP8, the first parameter is 4 and the second parameter is 16; or, when the target data format is FP16, the first parameter is 2 and the second parameter is 8.

[0209] In one example, the tensor storage engine retrieves the rearrangement parameters from the fourth storage unit based on the target data format to which the computation result needs to be converted.

[0210] In one example, the tensor storage engine is used to adjust the order of coordinate addresses in the coordinate address information according to the first dimension of the first tensor, based on the rearrangement parameters, when the first tensor is arranged in an interleaved manner, to obtain the adjusted coordinate address information; or, when the first tensor is arranged in a non-interleaved manner, it adjusts the order of coordinate addresses in the coordinate address information according to the second dimension of the first tensor, based on the rearrangement parameters, to obtain the adjusted coordinate address information; wherein the data change direction corresponding to the first dimension is different from the data change direction corresponding to the second dimension.

[0211] In one example, when the number of multiple elements in a set is less than the second parameter, the tensor storage engine fills the second storage unit with L preset elements after the multiple elements; where the second parameter is used to indicate the rearrangement granularity of the first tensor, L is the difference between the second parameter and the number of multiple elements in a set, and L is a positive integer.

[0212] In one example, the processor includes a fourth storage unit; the tensor storage engine obtains control instructions for a first tensor; based on the control instructions, it obtains basic information of the first tensor from the fourth storage unit, the basic information including at least one of the following: dimension information, coordinate information, arrangement method, and data format; based on the basic information of the first tensor, it obtains the coordinate address information of the first tensor.

[0213] In one example, after completing the reading of the first tensor, the tensor storage engine updates the asynchronous barrier unit to trigger an interrupt via the asynchronous barrier unit to end the reading of the first tensor.

[0214] Step 1002: The tensor calculation engine performs logical operations on the rearranged tensors to obtain the calculation results.

[0215] Step 1003: The format conversion unit converts the data format of each element in the calculation result to the target data format to obtain the format-converted calculation result, and stores the format-converted calculation result in the third storage unit; wherein, the format-converted calculation result includes the calculation result of each of the multiple elements, and the third arrangement order of the calculation results of the multiple elements in the third storage unit is the same as the first arrangement order.

[0216] In summary, the technical solution provided in this application, during the process of the tensor storage engine reading the first tensor from the first storage unit to the second storage unit, supports the tensor storage engine to automatically rearrange multiple elements in the first tensor so that the third arrangement order of the operation results of the multiple elements in the third storage unit is the same as the first arrangement order of the multiple elements, without the need to develop additional software design to support the additional rearrangement process, thereby reducing the complexity of the tensor processing flow in the processor and avoiding the development costs brought by software design, thus reducing the development cost of the processor.

[0217] In addition, since this application implements tensor rearrangement at the existing hardware level, it avoids the impact of adding a rearrangement process on the processor's end-to-end performance, thereby improving the processor's end-to-end performance.

[0218] Please refer to Figure 11, which shows a simplified structural block diagram of a computer device provided in one embodiment of this application. The computer device 1100 can be any electronic device with data computing, processing, and storage functions.

[0219] Computer device 1100 includes a processor 1101 and a memory 1102.

[0220] Processor 1101 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1101 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1101 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1101 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1101 may also include an AI processor for handling computational operations related to machine learning.

[0221] The memory 1102 may include one or more computer-readable storage media, which may be non-transitory. The memory 1102 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1102 are used to store a computer program configured to be executed by one or more processors.

[0222] Those skilled in the art will understand that the structure shown in FIG11 does not constitute a limitation on the computer device 1100, and may include more or fewer components than shown, or combine certain components, or employ different component arrangements.

[0223] In some embodiments, a chip product is also provided, which includes the processor described above. Optionally, the chip product may be a GPU chip product, where the processor described above is a GPU, and the GPU chip product includes the GPU described above. Optionally, the above chip product may be implemented as a graphics card, which includes the processor described above, such as a GPU.

[0224] In some embodiments, a computer device is also provided, which includes the processor described above. Optionally, the computer device may be a personal computer, workstation, game console, and some mobile devices (such as tablets, smartphones, etc.), or it may be an in-vehicle terminal device, smart home device, smart TV, smart robot, etc., or it may be a server, server cluster, artificial intelligence computing cluster, cloud computing cluster, etc. The artificial intelligence computing cluster may also be simply referred to as an intelligent computing cluster or smart computing cluster, and this application does not limit it in this way.

[0225] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0226] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A processor, the processor comprising: Tensor storage engine, tensor computation engine, format conversion unit, first storage unit, second storage unit, and third storage unit; The first storage unit stores a first tensor, which includes multiple elements arranged in a first order. The tensor storage engine is used to read the plurality of elements from the first storage unit into the second storage unit according to a second arrangement order to obtain a rearranged tensor, wherein the second arrangement order is different from the first arrangement order; The tensor computation engine is used to perform logical operations on the rearranged tensor to obtain the computation result; The format conversion unit is used to convert the data format of each element in the calculation result to the target data format, to obtain the format-converted calculation result, and to store the format-converted calculation result in the third storage unit; The calculation result after format conversion includes the calculation result of each of the multiple elements, and the third arrangement order of the calculation result of each of the multiple elements in the third storage unit is the same as the first arrangement order.

2. The processor according to claim 1, wherein, The tensor storage engine is used for: Obtain the coordinate address information of the first tensor, wherein the coordinate address information includes the coordinate addresses of the plurality of elements in the first storage unit; The order of the coordinate addresses included in the coordinate address information is adjusted to obtain the adjusted coordinate address information; Based on the arrangement order of the coordinate addresses included in the adjusted coordinate address information, the multiple elements are sequentially read from the first storage unit into the second storage unit to obtain the rearranged tensor.

3. The processor according to claim 2, wherein, The processor further includes a fourth storage unit; the tensor storage engine is used for: Rearrangement parameters are obtained from the fourth storage unit, and the rearrangement parameters are used to adjust the arrangement order of the coordinate addresses; When the rearrangement parameter indicates that the coordinate addresses should be rearranged, the order of the coordinate addresses included in the coordinate address information is adjusted according to the rearrangement parameter to obtain the adjusted coordinate address information.

4. The processor according to claim 3, wherein, The rearrangement parameters include a first parameter and a second parameter. The first parameter indicates the relationship between the total number of bits in the data format of the operation result and the total number of bits in the target data format. The second parameter indicates the rearrangement granularity of the first tensor. The tensor storage engine is used for: For each coordinate address in the coordinate address information, a first intermediate parameter and a second intermediate parameter are obtained based on the arrangement position of the coordinate address in the coordinate address information and the second parameter. The first intermediate parameter is used to indicate the rearrangement granularity to which the coordinate address belongs, and the second intermediate parameter is used to indicate the number of bits of the coordinate address in the rearrangement granularity to which the coordinate address belongs. Based on the first parameter, the first intermediate parameter, the second intermediate parameter, and the second parameter, a third intermediate parameter is obtained, which is used to indicate the arrangement position of the coordinate address in the adjusted coordinate address information; The adjusted coordinate address information is obtained based on the third intermediate parameters corresponding to each coordinate address included in the coordinate address information.

5. The processor according to claim 4, wherein, When the target data format is a 4-bit floating-point number FP4, the first parameter is 8, and the second parameter is 32; or... When the target data format is 8-bit floating-point number FP8, the first parameter is 4, and the second parameter is 16; or, When the target data format is 16-bit floating-point number FP16, the first parameter is 2 and the second parameter is 8.

6. The processor according to any one of claims 3 to 5, wherein, The tensor storage engine is used for: Based on the target data format to which the calculation result needs to be converted, the rearrangement parameters are obtained from the fourth storage unit.

7. The processor according to any one of claims 3 to 6, wherein, The tensor storage engine is used for: When the first tensor is arranged in an interleaved manner, according to the rearrangement parameters, the order of the coordinate addresses included in the coordinate address information is adjusted according to the first dimension corresponding to the first tensor to obtain the adjusted coordinate address information; or... When the first tensor is arranged in a non-interleaved manner, according to the rearrangement parameters, the order of the coordinate addresses included in the coordinate address information is adjusted according to the second dimension corresponding to the first tensor to obtain the adjusted coordinate address information; The direction of data change corresponding to the first dimension is different from the direction of data change corresponding to the second dimension.

8. The processor according to any one of claims 1 to 7, wherein, The tensor storage engine is used for: If the number of groups of the plurality of elements is less than the second parameter, L preset elements are filled after the plurality of elements in the second storage unit; Wherein, the second parameter is used to indicate the rearrangement granularity of the first tensor, L is the difference between the second parameter and the number of groups of the plurality of elements, and L is a positive integer.

9. The processor according to any one of claims 1 to 8, wherein, The processor includes a fourth storage unit; the tensor storage engine is used for: Obtain control instructions for the first tensor; The basic information of the first tensor is obtained from the fourth storage unit according to the control instruction. The basic information includes at least one of the following: dimension information, coordinate information, arrangement method, and data format. Based on the basic information of the first tensor, the coordinate address information of the first tensor is obtained.

10. The processor according to any one of claims 1 to 9, wherein, The tensor storage engine is used for: After the first tensor is read, the asynchronous barrier unit is updated so that the reading of the first tensor is terminated by triggering an interrupt through the asynchronous barrier unit.

11. A chip product comprising the processor as described in any one of claims 1 to 10.

12. A computer device comprising a processor as described in any one of claims 1 to 10.

13. A tensor processing method applied to a processor, the processor comprising: Tensor storage engine, tensor computation engine, format conversion unit, first storage unit, second storage unit, and third storage unit; The first storage unit stores a first tensor, which includes multiple elements arranged in a first order. The method includes: The tensor storage engine reads the multiple elements from the first storage unit into the second storage unit according to the second arrangement order to obtain a rearranged tensor, wherein the second arrangement order is different from the first arrangement order; The tensor calculation engine performs logical operations on the rearranged tensor to obtain the calculation result; The format conversion unit converts the data format of each element in the calculation result to the target data format to obtain the format-converted calculation result, and stores the format-converted calculation result in the third storage unit; The calculation result after format conversion includes the calculation result of each of the multiple elements, and the third arrangement order of the calculation result of each of the multiple elements in the third storage unit is the same as the first arrangement order.

Images