Integrated circuit for data processing, processor and computing device

Abstract

The present disclosure relates to an integrated circuit for data processing, a processor and a computing device. The integrated circuit comprises a data storage circuit and a matrix operation circuit. The data storage circuit is configured to store matrix data. The matrix operation circuit comprises: a control logic circuit configured to: on the basis of a specified data format, read a first matrix and a second matrix from the data storage circuit, the specified data format indicating that elements in the first matrix are floating-point data, and elements in the second matrix are integer data; and an operation logic circuit configured to: convert elements in the first matrix and in the second matrix into elements of a target floating-point type on the basis of the data type of the elements indicated by the specified data format and determine, on the basis of the converted elements of the first matrix and the second matrix, a result matrix corresponding to the first matrix and the second matrix.

Description

Integrated circuits, processors, and computing devices for data processing

[0001] This application claims priority to Chinese Patent Application No. 202411548446.8, filed on October 31, 2024, entitled "Integrated Circuit, Processor and Computing Device for Data Processing", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of integrated circuit technology, and more specifically, to an integrated circuit, processor, and computing device for data processing. Background Technology

[0003] In fields such as Artificial Intelligence (AI), a large number of matrix operations are often involved. Taking AI models as an example, the training process involves a large number of operations between feature matrices, weight matrices, and other matrices. As the model structure becomes more complex and larger, on the one hand, the efficiency of matrix operations will limit the model training efficiency; on the other hand, the massive number of weight parameters will occupy a high amount of storage space. Summary of the Invention

[0004] This disclosure provides an integrated circuit, processor, and computing device for data processing.

[0005] According to one aspect of the present disclosure, an integrated circuit for data processing is provided, comprising: a data storage circuit configured to store matrix data; and a matrix operation circuit, comprising: a control logic circuit configured to: read a first matrix and a second matrix from the data storage circuit based on a specified data format, wherein the specified data format indicates that elements in the first matrix are floating-point data and elements in the second matrix are integer data; and an operation logic circuit configured to: convert elements in the first matrix and the second matrix into target floating-point elements according to the data type of the elements indicated by the specified data format, and determine a result matrix corresponding to the first matrix and the second matrix based on the converted elements of the first matrix and the second matrix.

[0006] In some embodiments, the integer elements in the second matrix are closely packed.

[0007] In some embodiments, the arithmetic logic circuit includes: at least one conversion circuit, wherein each conversion circuit is configured to: receive elements in a first matrix or a second matrix and element type data indicating the data type of the received elements, and convert the received elements into target floating-point elements based on the element type data, wherein the element type data is determined based on a specified data format; and an arithmetic circuit, coupled to the conversion circuit, and configured to: perform at least one operation of multiplication and addition based on the elements of the converted first matrix and second matrix to determine a result matrix.

[0008] In some embodiments, each conversion circuit is configured to: in response to an element type data indication that the received element is integer data, determine the target floating-point element corresponding to the element based on a pre-stored mapping table or by invoking preset conversion logic.

[0009] In some embodiments, the conversion circuit is configured to: in response to an element type data indication that the received element is 4-bit integer data, determine the target floating-point element corresponding to the element based on a mapping table from 4-bit integer data to target floating-point data; in response to an element type data indication that the received element is 8-bit integer data, determine the sign value, exponent, and mantissa corresponding to the element according to the data structure of the 8-bit integer data, and determine the target floating-point element corresponding to the element based on the determined sign value, exponent, and mantissa.

[0010] In some embodiments, the data storage circuitry includes a register array, on which integer elements of the second matrix are stored in a close-packed manner according to the matrix structure, wherein the control logic circuitry is configured to read the close-packed integer elements of the second matrix from the register array.

[0011] In some embodiments, the data storage circuit includes a buffer in which integer elements of the second matrix are stored in a closely packed manner in an interleaved pattern. The interleaved pattern matches the data type of the integer elements and the size of the matrix stored in the buffer. In the interleaved pattern, elements in the same column of the row-major second matrix are stored in different storage cells of the buffer, or elements in the same row of the column-major second matrix are stored in different storage cells of the buffer. The control logic circuit is configured to read closely packed integer elements of the second matrix from multiple storage cells of the buffer according to the interleaved pattern.

[0012] In some embodiments, the integrated circuit further includes a data loading circuit configured to load an external matrix including a second matrix to a data storage circuit according to a loading step size, wherein the loading step size matches the size of the external matrix to be loaded and the data type of the integer elements.

[0013] In some embodiments, the control logic circuit is configured to: acquire a read step size, wherein the read step size matches the size of the second matrix and the data type of the integer elements; and read the integer elements of the second matrix from the data storage circuit according to the read step size.

[0014] In some embodiments, the target floating-point type is a non-standard floating-point data type used by the hardware circuit when performing floating-point operations.

[0015] In some embodiments, the arithmetic logic circuit includes a plurality of sub-circuits, wherein each sub-circuit corresponds to an element in the result matrix, and includes: a conversion module configured to: receive elements from a first matrix and a second matrix corresponding to elements in the result matrix, and convert the received elements into target floating-point elements based on the data type of the received elements; an arithmetic module coupled to the conversion module and configured to: perform at least one operation of multiplication and addition based on the elements of the converted first matrix and second matrix to obtain target floating-point elements corresponding to elements in the result matrix; and a regularization module coupled to the arithmetic module and configured to: convert the obtained target floating-point elements into desired floating-point elements, wherein the desired floating-point type is a standard floating-point data type different from the target floating-point type.

[0016] In some embodiments, the conversion circuit includes: a floating-point branch circuit configured to: in response to an element type data indication that the received element is a floating-point element, determine the sign value, exponent, and mantissa corresponding to the element according to the floating-point type data structure indicated by the element type data, and determine the target floating-point element corresponding to the element based on the determined sign value, exponent, and mantissa; and an integer branch circuit configured to: in response to an element type data indication that the received element is an integer element, determine the target floating-point element corresponding to the element based on a pre-stored mapping table or by calling preset conversion logic.

[0017] According to another aspect of the present disclosure, a processor is provided, including the integrated circuit described in various embodiments of the foregoing aspects.

[0018] According to another aspect of the present disclosure, a computing device is provided, including the processor described in the foregoing aspects.

[0019] In the integrated circuit for data processing provided in this embodiment, the matrix operation circuit can directly read a first matrix and a second matrix that respectively contain floating-point elements and integer elements, and convert them into the same target floating-point elements through internal arithmetic logic circuits, thereby performing operations to obtain the corresponding result matrix. Therefore, the matching of matrix element data formats can be implemented in hardware within the matrix operation circuit, which helps to further improve matrix operation efficiency and reduce storage space occupation and data transmission volume.

[0020] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0021] Further details, features, and advantages of this disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0022] Figure 1 schematically illustrates an example input layout for a tensor kernel;

[0023] Figures 2A, 2B, and 2C schematically illustrate example block diagrams of integrated circuits for data processing according to some embodiments of the present disclosure;

[0024] Figure 3 schematically illustrates an exemplary circuit diagram of the internal circuitry of an operational logic unit according to some embodiments of the present disclosure;

[0025] Figure 4 schematically illustrates an exemplary circuit diagram of a conversion circuit according to some embodiments of the present disclosure;

[0026] Figures 5A, 5B, 5C, and 5D exemplarily illustrate schematic diagrams of the storage of matrix data on a register array according to some embodiments of the present disclosure;

[0027] Figures 6A, 6B, 6C, 6D, 6E, and 6F exemplarily illustrate storage diagrams when matrix data is read into a buffer in an integrated circuit according to some embodiments of the present disclosure;

[0028] Figures 7A and 7B schematically illustrate exemplary block diagrams of processors according to some embodiments of the present disclosure;

[0029] Figure 8 schematically illustrates an exemplary block diagram of a computing device according to some embodiments of the present disclosure. Detailed Implementation

[0030] Traditional processors typically employ SIMD (Single Instruction Multiple Data) or SIMT (Single Instruction Multiple Threads) architectures, relying on FMA (Fused Multiply-Accumulate operation) to perform matrix operations. However, in these implementations, data sharing between different threads is limited, requiring multiple reads of repetitive data, resulting in poor performance and energy consumption. To improve the speed of matrix operations, some processors incorporate Tensor Cores as matrix acceleration units. The processing function of a Tensor Core is D = A × B + C, where A, B, and C are input matrices, and D is the output matrix. The data formats of the input matrices supported by the Tensor Core are shown in the table below:

[0031] It is evident that input matrices A and B need to have the same data format and the same number of bits.

[0032] In some application scenarios, there may be a large demand for matrix operations, resulting in a significant need to store matrix elements. Taking large model training as an example, the training process of large models often involves numerous convolutional operations and a massive number of weight parameters. If stored in floating-point format, these weight parameters would occupy a large amount of storage space. For instance, the GLM-130B model has 130 bytes of weight parameters; if stored in FP16 format, it would require 260GB of storage space. Furthermore, transmitting the weight parameters would consume significant bandwidth. Since processors such as GPUs (Graphics Processing Units) generally have limited storage space and bandwidth, this often necessitates the use of a large portion of the GPU for model training, making it difficult to fully utilize the computational power of tensor kernels, leading to low tensor kernel utilization. To address this issue, some researchers have proposed large model quantization schemes. For activation values, the FP16 format is still used, while for weights, they are quantized into 8-bit or 4-bit integer data. This significantly saves storage space and bandwidth. Taking the quantization of GLM-130B weights from FP16 format to INT4 format as an example, 75% of storage space and bandwidth can be saved. The 130B weight parameters can be stored in 65GB of space, which significantly reduces the demand for GPUs. This reduces the number of GPUs required during the training process.

[0033] However, under the aforementioned quantization scheme, activation values ​​can be in FP16 format, while weights are in INT4 or INT8 format, which clearly does not meet the data format requirements of the tensor kernel mentioned earlier. To resolve this conflict, some researchers in related technologies have adopted software algorithms to decompress the weights before feeding the decompressed weights into the tensor kernel for computation. Specifically, for the weight matrix to be processed, each pair of INT4 weights can first be treated as an 8-bit data and loaded into a register in the GPU, then converted into two INT8 data, and then each INT8 data is converted into the corresponding FP16 data and stored in a register according to the input layout of the tensor kernel, ready to be transmitted to the tensor kernel for computation. For example, Figure 1 schematically shows an example input layout of the tensor kernel, which is an 8×8 matrix, where each thread (T) contains two FP16 data. It should be understood that this layout is merely exemplary; in practice, a 16×16 matrix or other input layouts can also be used. When combining the aforementioned software algorithm-based pre-decompression of weights with the quantization scheme, while the requirements for storage space and bandwidth can be reduced, several drawbacks arise. First, additional registers are needed to process intermediate formats; for example, after decompressing the weight data, additional registers are required to store the decompressed weight data. Second, this software decompression algorithm involves numerous data conversion and data transfer operations, which require the support of other computational circuits besides the tensor kernel, as well as data transfer circuits, and consume additional processing cycles. Furthermore, if the tensor kernel has a high processing speed, the aforementioned weight decompression process may not be able to keep up with its speed, leading to a waste of tensor kernel resources and negatively impacting the overall computational efficiency.

[0034] Based on the above considerations, embodiments of this disclosure propose a novel integrated circuit for data processing that helps to eliminate or at least alleviate the aforementioned problems.

[0035] Figures 2A, 2B, and 2C schematically illustrate example block diagrams of integrated circuits for data processing according to some embodiments of the present disclosure.

[0036] For example, FIG2A shows a schematic block diagram of an integrated circuit 200 for data processing according to some embodiments of the present disclosure. As shown in FIG2A, the integrated circuit 200 may include a data storage circuit 210 and a matrix operation circuit 220.

[0037] The data storage circuit 210 can be configured to store matrix data. Optionally, the data storage circuit 210 can be a combination of one or more of a buffer, register, or other circuit structures that can be used for data storage. The matrix data stored in the data storage circuit 210 can include various types of matrix data used to participate in matrix multiplication and / or addition operations. Taking an AI model training scenario as an example, the stored matrix data can include weight parameter matrices, activation value matrices, etc., wherein the weight parameter matrix can be obtained from external memory, such as the processor's global memory (e.g., video memory), and the activation value matrix can be obtained through previous calculations.

[0038] Matrix operation circuit 220 can be configured to perform matrix operations based on matrix data in data storage circuit 210. Optionally, matrix operation circuit 220 can be the tensor kernel or other matrix acceleration unit mentioned above, or it can be other types of circuits for performing matrix operations. As shown in FIG2A, matrix operation circuit 220 may include control logic circuit 221 and arithmetic logic circuit 222. Control logic circuit 221 can be configured to read a first matrix and a second matrix from data storage circuit 210 based on a specified data format, wherein the specified data format indicates that the elements in the first matrix are floating-point data and the elements in the second matrix are integer data. Arithmetic logic circuit 222 can be configured to convert the elements in the first matrix and the second matrix into target floating-point elements according to the data type of the elements indicated by the specified data format, and determine the result matrix corresponding to the first matrix and the second matrix based on the converted elements of the first matrix and the second matrix.

[0039] In the embodiments of this disclosure, the terms "first matrix" and "second matrix" are used merely to distinguish between the two matrices and do not indicate or imply the order in which the two matrices participate in the operation. Taking matrices A and B participating in the matrix multiplication operation A×B as an example, the first matrix can refer to either matrix A or matrix B, and correspondingly, the second matrix can refer to the other matrix. In the embodiments of this disclosure, floating-point data should be understood to encompass various types of floating-point data, such as standard floating-point data types like FP16, BF16, FP32, and TF32, as well as other self-defined or non-standard floating-point data types used in specific scenarios; integer data should be understood to encompass various types of integer data, such as standard integer data types like INT4, INT8, and UINT8, as well as other self-defined or non-standard integer data types used in specific scenarios. In embodiments of this disclosure, the target floating-point type may refer to a non-standard floating-point data type used when performing floating-point operations within hardware circuitry such as matrix operation circuit 220. For example, it may refer to a hardware intermediate format of floating-point elements, such as a hardware intermediate format that includes a sign bit, a 10-bit exponent, and 24-30 bits of mantissa. Alternatively, the target floating-point type may also be selected from various standard floating-point data types.

[0040] In embodiments of this disclosure, the result matrix corresponding to the first matrix and the second matrix can be the result of an operation performed on the first matrix and the second matrix, such as one or more combinations of multiplication, addition, etc. Optionally, the elements in the result matrix can remain as elements of the target floating-point type. Alternatively, the elements in the result matrix can also be elements of other desired floating-point types, such as FP32, FP16, or other floating-point data types. Exemplarily, after performing an operation based on the elements of the converted first matrix and the second matrix, the arithmetic logic circuit 222 can convert the target floating-point operation result into FP32 format to obtain an FP32 type result matrix. Optionally, the result matrix can be further quantized, for example, by performing scaling, biasing, etc., to obtain a matrix in a lower bit floating-point format, such as FP16, or even quantized into an integer format matrix such as INT4 or INT8. Further exemplarily, this quantization process can be implemented by a conventional programmable SIMD processor.

[0041] For example, the specified data format can be pre-configured according to requirements, such as by the user based on the data format requirements of the application scenario. For instance, for matrices A and B participating in the multiplication operation A×B, the data format can be defined in a manner similar to "data type A_data type B", where one of data type A and data type B can be an integer data type and the other can be a floating-point data type. For example, the following data formats can be defined: data format "bf16_int4" indicates that the elements in matrix A are BF16 data and the elements in matrix B are INT4 data; data format "bf16_int8" indicates that the elements in matrix A are BF16 data and the elements in matrix B are INT8 data; data format "int4_bf16" indicates that the elements in matrix A are INT4 data and the elements in matrix B are BF16 data; data format "int8_bf16" indicates that the elements in matrix A are INT8 data and the elements in matrix B are BF16 data. Similarly, other data formats such as "fp16_int4", "fp16_int8", "int4_fp16", and "int8_fp16" can also be defined. Optionally, the newly defined data formats can be added to existing instruction sets used for tensor kernels or other types of matrix operation circuits.

[0042] For example, the first and second matrices read from the data storage circuit 210 can contain the same number of elements, such as 8×8, 16×16, or other numbers of elements. The integer elements in the second matrix can be closely packed. In embodiments of this disclosure, close packing can be understood as each element conforming to the integer data type indicated by the specified data format, and there are no padding bits or free bits between adjacent elements. For example, assuming the specified data format indicates that the elements in the second matrix are integer data of type INT4, then each element in the acquired second matrix is ​​INT4 data occupying 4 bits, and there are no free bits between adjacent elements, nor are there bits filled with other values. This close packing of integer data can save storage space and reduce data interaction, thereby reducing bandwidth requirements and saving power. Taking INT4 and FP16 data formats as examples, when the size of the stored or read matrices is the same (i.e., the number of elements in the matrices is the same), the storage space occupied or the amount of data to be read when using the INT4 format can be reduced to one-quarter of that when using the FP16 format.

[0043] Compared to the previously mentioned scheme of pre-compressing integer matrix elements using software algorithms, the integrated circuit 200 enables matrix operation circuits such as matrix cores to natively support matrix operations on the first and second matrices, which contain floating-point and integer elements respectively. This eliminates the format conversion and related data transfer processes involved in the aforementioned software decompression. On the one hand, this reduces the burden on software developers, and on the other hand, it helps improve circuit operation efficiency, reduce the occupation of local storage space, and reduce the amount of data transmission.

[0044] In some embodiments, as shown in FIG2B, the arithmetic logic circuit 222 may include at least one conversion circuit 2221 and an arithmetic circuit 2222 coupled to the at least one conversion circuit 2221. Each conversion circuit 2221 may be configured to: receive elements in a first matrix or a second matrix and element type data indicating the data type of the received elements, and convert the received elements into target floating-point elements based on the element type data, wherein the element type data is determined based on a specified data format. The arithmetic circuit 2222 may be configured to: perform at least one operation of multiplication and addition based on the elements of the converted first matrix and second matrix to determine a result matrix. For example, assuming the specified data format is INT4_FP16 and the matrix operation performed is "A×B", then for elements in matrix A, the element type data may be determined to be INT4, and for elements in matrix B, the element type data may be determined to be FP16. For example, a conversion circuit array can be configured, wherein each conversion circuit can receive an element and its corresponding element type data from a first matrix or a second matrix, and convert the received element; alternatively, each conversion circuit can receive more than one element and its corresponding element type data, and convert the received elements sequentially. Optionally, the arithmetic circuit 2222 can perform multiplication, multiplication-addition, etc., based on the elements in the converted first and second matrices. When performing multiplication-addition, the arithmetic circuit 2222 can also receive elements from a third matrix, which can also be elements that have been converted to the target floating-point format.

[0045] In some embodiments, each conversion circuit can be configured to: in response to an element type data indication that the received element is integer data, determine the target floating-point element corresponding to the element based on a pre-stored mapping table or by invoking preset conversion logic. Optionally, the received integer element can be directly converted to the corresponding target floating-point element based on a pre-stored mapping table or by invoking preset conversion logic, such as a hardware intermediate format for floating-point elements. This helps to achieve data format conversion more efficiently and improve hardware processing efficiency. Alternatively, it can be first converted to other floating-point elements, such as the floating-point data format originally supported by tensor kernels or other matrix operation circuits, such as the floating-point data format of the elements in the first matrix, and then further converted to the target floating-point element. This can be more easily adapted to existing circuit structures. More specifically, compared to the latter approach, the former approach can save a lot of unnecessary shift operations, thereby helping to improve overall processing efficiency. For example, in the regularization process of converting integer elements to the floating-point data format of the elements in the first matrix, many additional shift operations are involved, which consume additional hardware resources and time costs.

[0046] For example, in response to an element type data indication that the received element is a 4-bit integer, the corresponding target floating-point element can be determined based on a mapping table from 4-bit integer data to target floating-point data. Taking INT4 as an example, there are 16 types of input data. The mapping table from INT4 to target floating-point data can be pre-stored. When the conversion circuit receives an INT4 element, it can directly obtain the corresponding target floating-point data by looking up the table. Alternatively, a mapping table from INT4 to FP16 or other floating-point data can be pre-stored. When the conversion circuit receives an INT4 element, it can obtain the corresponding FP16 data by looking up the table, and then obtain the corresponding target floating-point data based on the conversion logic from FP16 to target floating-point data.

[0047] For example, in response to an element type data indication that the received element is an 8-bit integer, the sign value, exponent, and mantissa corresponding to the element can be determined according to the data structure of the 8-bit integer, and the target floating-point element corresponding to the element can be determined based on the determined sign value, exponent, and mantissa. Taking INT8 as an example, the data format conversion can be performed using the following example pseudocode:

[0048] The pseudocode above demonstrates how to convert a received INT8 element into a corresponding FP16 element. Specifically, it first checks if the input element is 0; if so, it returns 0 directly. If not, it checks if the input element is negative, for example, by checking if its first sign bit is 1. If so, the sign bit is assigned a value of 1, and the two's complement of the input element is obtained. If not, the sign bit remains initially assigned a value of 0. Then, a circular shift can be used to determine the position of the highest 1 bit of the input element's value. Furthermore, the exponent and mantissa of the input element can be determined based on the determined position of the highest 1 bit. The third-to-last line of the pseudocode illustrates the operation of obtaining the corresponding FP16 element based on the determined sign bit (sign), exponent (exp), and mantissa (mant). However, to reduce hardware load, this operation can be omitted; that is, instead of obtaining the corresponding FP16 element, the target floating-point element can be determined directly based on the sign bit, exponent, and mantissa, such as a non-standard floating-point format containing a sign bit, a 10-bit exponent, and 24-30 mantissa bits. Alternatively, for 8-bit integer elements, a lookup table method can be used to determine the corresponding target floating-point element. This process is similar to that for 4-bit integer elements and will not be elaborated here.

[0049] In some embodiments, the arithmetic logic circuit 222 may include multiple sub-circuits, such as multiple ALUs (Arithmetic and Logic Units), where each sub-circuit may correspond to an element in the result matrix and includes a conversion module, an arithmetic module, and a regularization module. The conversion module may be configured to receive elements from the first and second matrices corresponding to the element in the result matrix and perform data format conversion. The arithmetic module may perform at least one operation, either multiplication or addition, based on the converted elements to obtain a target floating-point element corresponding to the element in the result matrix. The regularization module may convert the obtained target floating-point element into a desired floating-point element, where the desired floating-point type is a standard floating-point data type different from the target floating-point type. In such embodiments, each element in the second matrix read by the control logic circuit 221 may be assigned to a corresponding sub-circuit.

[0050] Figure 3 schematically illustrates an exemplary circuit diagram of the internal circuitry of an operational logic unit according to some embodiments of the present disclosure.

[0051] For example, Figure 3 shows a schematic circuit diagram of a sub-circuit 300 that can be used in the arithmetic logic circuit 222. The sub-circuit 300 shown in Figure 3 can be used to perform multiplication and addition operations on 8×8 matrices, such as the matrix operation "A×B+C". As shown in Figure 3, the sub-circuit 300 includes circuit structures of levels 0 to 9. The multiple conv modules of level 0 can be regarded as transformation modules and correspond to the functions of at least one of the aforementioned transformation circuits. For example, each conv module can be regarded as a transformation circuit. The circuits of levels 1 to 7 can be regarded as arithmetic modules and correspond to the functions of the aforementioned arithmetic circuits. The circuits of levels 8 to 9 can be regarded as regularization circuits, which can perform regularization and data format conversion on the operation results of the arithmetic circuits. Specifically, taking the matrix operation "A×B+C" as an example, where A, B, and C are all 8×8 matrices, at level 0, the elements in matrix A and matrix B can be received through the conv modules. Assuming sub-circuit 300 is used to determine the element in row i and column j of the result matrix, level 0 can receive elements from the i-th row of matrix A, elements from the j-th column of matrix B, and elements from the i-th row and column j of matrix C. Each conv module can receive two elements directly multiplied from matrices A and B, and module c can receive elements from matrix C. As described in the preceding embodiments, the conv module can convert the received elements into target floating-point elements based on the element type data. The target floating-point type can be a hardware intermediate format for floating-point elements, such as the floating-point format mentioned in the preceding embodiments, which includes a sign bit, 10 exponent bits, and 24-30 mantissa bits. In level 1, the mul module can perform multiplication operations on the target floating-point elements from the corresponding conv module. In level 2, the max.exp module can determine the maximum exponent in the output elements of each mul module. In level 3, the align module can perform exponent alignment based on the maximum exponent output by max.exp. In levels 4 and 5, the `add` module performs an accumulation operation based on the aligned data to obtain the product of the element in the i-th row of matrix A and the element in the j-th column of matrix B. In level 6, this product can be aligned with the exponent of the corresponding element in matrix C. The corresponding element in matrix C can be pre-processed using the `conv&max` module, which converts the element to a target floating-point number and determines the maximum exponent between it and the product of the element in the i-th row of matrix A and the element in the j-th column of matrix B. Finally, in level 7, the aligned elements can be added together to obtain intermediate result elements corresponding to the elements in the final matrix.The intermediate result element can then be processed by the Norm&conv module to obtain the corresponding element in the result matrix. The Norm&conv module can perform regularization and data format conversion on the element to convert it into an element in the desired data format, such as FP32 or other standard floating-point data types. It should be understood that the circuit shown in Figure 3 is merely exemplary; in practice, other circuit structures can be designed according to specific application requirements.

[0052] Figure 4 schematically illustrates an exemplary circuit diagram of a conversion circuit according to some embodiments of the present disclosure.

[0053] Further exemplarily, the conv module in Figure 3 can have the circuit structure shown in Figure 4. As shown in Figure 4, circuit 400 can receive input data data_in and convert it into corresponding output data data_out. Here, data_in can be 32-bit data of various data types, and data_out can be target floating-point data, such as 37-bit data as shown in the figure, which includes 1 bit sign bit S, 10 bits exponent bits E[9:0], and 26 bits mantissa bits Sig[25:0]. In addition, the circuit structure can also receive type[2:0], which can be element type data indicating the data type of the input data, such as INT4, INT8, FP16, BF16, and other possible data formats. Furthermore, the circuit structure can also output tag[2:0], which can indicate some special values, such as nan (not a number), inf (infinity, including positive and negative infinity), and 0 (zero value). As shown in Figure 4, the circuit structure contains two processing branches, as indicated by the dashed boxes. The left branch is the floating-point branch circuit 410, which can be used to process floating-point input data, and the right branch is the integer branch circuit 420, which can be used to process integer input data. For the floating-point branch circuit 410, in response to the type indicating that data_in is floating-point, the sign bit S, exponent E, and mantissa M of data_in can be obtained according to the data structure of the specific floating-point type indicated by type, and the element of the target floating-point type corresponding to data_in can be determined based on them. For the sign bit S, it can be directly mapped to the sign bit S of data_out. For the exponent E, the corresponding exponent E[9:0] in data_out can be determined according to the label reflecting the value of data_in. The label can be determined based on the exponent E and mantissa M of data_in, and can be used to indicate that data_in is normal, denormal, infinity, not a number, or zero. For regularized data, E can be added to the constant cons0 to obtain E[9:0] in the corresponding data_out. For irregularized data, infinity / non-numeric values, and zero values, E[9:0] can be determined as the constants cons1, cons2, and 0, respectively. Similarly, for the mantissa M, the corresponding Sig[25:0] in data_out can be determined based on the above labels. For regularized data, M can be shifted to obtain M', and 1 can be added to the first bit to obtain the corresponding Sig[25:0]. For irregularized data, infinity, and non-numeric values, only the shift operation can be performed to obtain M', which serves as the corresponding Sig[25:0]. For zero values, Sig[25:0] can be directly determined as 0.For the integer branch circuit 420, in response to the type indicator indicating that data_in is an integer, it can be converted to the corresponding target floating-point type data_out, as described in the various embodiments above, based on a pre-stored mapping table or by calling preset conversion logic. It should be understood that the circuit structure shown in Figure 4 is merely exemplary; in practice, other forms of circuit structures can be designed according to specific application requirements.

[0054] In some embodiments, the data storage circuit 210 may include a register array, on which the integer elements of the second matrix may be stored in a closely packed manner according to the matrix structure. In this embodiment, the control logic circuit 221 may be configured to read the closely packed integer elements of the second matrix from the register array. Optionally, the matrix data (including the aforementioned second matrix) stored in the register array may come from external memory, such as the processor's global memory, and the matrix data loaded into the register array each time may include one or more matrices; for example, in an AI model training scenario, one or more weight parameter matrices may be loaded into the register array simultaneously. The control logic circuit 221 may read the integer elements of the second matrix from the register array according to the matrix structure of the second matrix to be read.

[0055] Figures 5A, 5B, 5C, and 5D exemplarily illustrate schematic diagrams of matrix data stored on register arrays according to some embodiments of the present disclosure. Specifically, Figure 5A shows an exemplary compact arrangement of row-major matrix data containing 8×32 4-bit integer elements on a register array; Figure 5B shows an exemplary compact arrangement of column-major matrix data containing 32×8 4-bit integer elements on a register array; Figure 5C shows an exemplary compact arrangement of row-major matrix data containing 8×16 8-bit integer elements on a register array; and Figure 5D shows an exemplary compact arrangement of column-major matrix data containing 16×8 8-bit integer elements on a register array. As shown, in the 4-bit element scenario, each thread (T) can correspond to 8 4-bit elements, and in the 8-bit element scenario, each thread (T) can correspond to 4 8-bit elements. In a row-major matrix, elements of the matrix data are stored sequentially in row-major order into the registers corresponding to each thread. After one row of elements is stored, the next row is stored. In a column-major matrix, elements of the matrix data are stored sequentially in column-major order into the registers corresponding to each thread. After one column of elements is stored, the next column is stored. Comparing this to the input layout of FP16 shown in Figure 1, it can be seen that when using data formats with lower bit counts, such as 4-bit or 8-bit integers, the same register array can store more elements, and the same number of elements can occupy fewer registers and fewer threads.

[0056] In the layout shown in Figures 5A and 5B, taking the matrix data of the first matrix stored therein as an example, assuming that the control logic circuit 221 needs to read the second 8×8 matrix, it can read the data corresponding to threads T0, T4, T8, T12, T16, T20, T24, and T28, where each thread corresponds to a row / column element, and each thread corresponds to a different row / column; assuming that the control logic circuit 221 needs to read the second 16×16 matrix, it can read the data corresponding to threads T0, T1, T4, T5, T8, T9, T12, T13, T16, T17, T20, T21, T24, T25, T28, and T29 (the remaining data can be read from the matrix data not shown), where each group of adjacent threads (e.g., T0, T1) corresponds to a row / column element, and each group of adjacent threads corresponds to a different row / column. In the layout shown in Figures 5C and 5D, taking the matrix data of the first matrix stored therein as an example, assuming that the control logic circuit 221 needs to read the second 8×8 matrix, it can read the data corresponding to threads T0, T1, T4, T5, T8, T9, T12, T13, T16, T17, T20, T21, T24, T25, T28, and T29. Each group of adjacent threads (e.g., T0, T1) corresponds to a row / column element, and each group of adjacent threads corresponds to different rows / columns. Assuming that the control logic circuit 221 needs to read the second 16×16 matrix, it can read the data corresponding to threads T0 to T31 (the remaining data can be read from the matrix data not shown). Each group of threads (e.g., T0, T1, T2, T3) corresponds to a row / column element, and each group of threads corresponds to different rows / columns.

[0057] In some embodiments, the data storage circuit 210 may include a cache in which integer elements of the second matrix are stored in a close-packed, interleaved manner. Optionally, the cache may be local memory of the integrated circuit 210. The interleaving pattern may be matched to the data type of the integer elements and the size of the matrix stored in the cache, and in the interleaving pattern, for a row-major second matrix, different elements in the same column may be stored in different memory bodies of the cache, or for a column-major second matrix, different elements in the same row may be stored in different memory bodies of the cache. In this embodiment, the control logic circuit 221 may be configured to read the close-packed integer elements of the second matrix from multiple memory bodies of the cache according to the interleaving pattern. Exemplarily, the cache may contain multiple blocks that may form multiple storage lines and multiple storage banks. When reading data from such a cache, data may be read from only one storage line at a time for each storage bank. By using the above interleaving mode, it can be ensured that the control logic circuit 221 can read the required matrix data synchronously without causing bank conflict, thereby helping to improve data interaction efficiency. Bank conflict is caused by the inability to read data from different storage rows of the same storage bank at the same time.

[0058] Assuming the second matrix is ​​a 16×16 matrix (with a total of 256 elements), Figures 6A, 6B, 6C, 6D, 6E, and 6F exemplarily illustrate storage diagrams when matrix data (including multiple 16×16 matrices) is read into a buffer in an integrated circuit according to some embodiments of this disclosure. The original data rows correspond to the originally read matrix data, different interleaved rows correspond to different storage rows in the buffer, and two or more blocks aligned with each other along the column direction in different storage rows correspond to the same memory bank. Each block can store 16 bytes (128 bits) of data, i.e., 32 4-bit elements or 16 8-bit elements. An identifier within each block indicates the row (M) and column (K) sequence number of the first element stored in that block. Specifically, Figure 6A shows the interleaving pattern when 4-bit row-major matrix data of M×K=64×64 is stored in the buffer; Figure 6B shows the interleaving pattern when 4-bit column-major matrix data of K×M=64×64 is stored in the buffer; Figure 6C shows the interleaving pattern when 4-bit column-major matrix data of K×M=64×128 is stored in the buffer; Figure 6D shows the interleaving pattern when 4-bit column-major matrix data of K×M=64×258 is stored in the buffer; Figure 6E shows the interleaving pattern when 8-bit row-major matrix data of M×K=128×128 is stored in the buffer; and Figure 6F shows the interleaving pattern when 8-bit column-major matrix data of K×M=128×128 is stored in the buffer. As can be seen from these figures, for row-major elements, such as Figures 6A and 6E, for each 16×16 matrix to be read, each row of data is stored in a different memory location. For example, a 16×16 matrix composed of elements from M0K0, M1K0, M2K0, M3K0, M4K0, M5K0, M6K0, M7K0, M8K0, M9K0, M10K0, M11K0, M12K0, M13K0, M14K0, and M15K0, etc. When using a 4-bit data format, only half of the data in each block can be read at a time, while when using an 8-bit data format, all the data in each block can be read at a time. For column-major elements, such as those in Figures 6B, 6C, 6D, and 6F, for each 16×16 matrix to be read, each column of data is stored in a different memory block. For example, a 16×16 matrix composed of elements from K0M0, K1M0, K2M0, K3M0, K4M0, K5M0, K6M0, K7M0, K8M0, K9M0, K10M0, K11M0, K12M0, K13M0, K14M0, and K15M0, etc. When using a 4-bit data format, only half of the data in each block can be read at a time. When using an 8-bit data format, all the data in each block can be read at a time.

[0059] It should be understood that the interleaving patterns shown in Figures 6A to 6F are merely exemplary and illustrative. In practice, other interleaving patterns can be designed according to specific application requirements, as long as the following conditions are met: for a row-major second matrix, different elements in the same column can be stored in different storage bodies of the cache; or, for a column-major second matrix, different elements in the same row can be stored in different storage bodies of the cache. For example, different interleaving patterns can be pre-designed for various data types and various sizes of the matrices to be stored in the cache. In subsequent use, the corresponding interleaving pattern can be selected to store the matrix data loaded into the cache based on the data type of the elements in the second matrix indicated by the specified data format and the size of the matrix data to be loaded into the cache.

[0060] In some embodiments, as shown in FIG2C, integrated circuit 200 may further include data loading circuit 230. Data loading circuit 230 may be configured to load an external matrix, including the second matrix, into a data storage circuit according to a loading stride, wherein the loading stride matches the size of the external matrix to be loaded and the data type of the integer elements. For example, for a row-major matrix, the loading stride may correspond to the number of rows of data in the matrix to be loaded, which can be determined based on the number of elements per row of the matrix to be loaded and the number of bits corresponding to the element data type. When loading the external matrix, the next row of data can be loaded after each loading stride's worth of rows has been loaded. For a column-major matrix, the loading stride may be the number of columns of data in the matrix to be loaded, which can be determined based on the number of elements per column of the matrix to be loaded and the number of bits corresponding to the element data type. When loading the external matrix, the next column of data can be loaded after each loading stride's worth of columns has been loaded. Alternatively, the loading step size can be measured in bytes or other units. For example, it can be measured by the number of 16-bit data blocks contained in the rows or columns of the external matrix to be loaded, and so on. Taking the matrix data shown in Figures 5A to 5D and Figures 6A to 6F as examples, for Figure 5A, the size of the row-major external matrix loaded into the register array is 8×32, with each element occupying 4 bits, so the loading step size can be determined as 16 bytes (i.e., 32×4 bits); for Figure 5B, the size of the column-major external matrix loaded into the register array is 32×8, with each element occupying 4 bits, so the loading step size can be determined as 16 bytes (i.e., 32×4 bits); for Figure 5C, the size of the row-major external matrix loaded into the register array is 8×16, with each element occupying 8 bits, so the loading step size can be determined as 16 bytes (i.e., 16×8 bits); for Figure 5D, the size of the column-major external matrix loaded into the register array is 16×8, with each element occupying 8 bits, so the loading step size can be determined as 16 bytes. (i.e., 16×8 bits); For Figures 6A and 6B, the size of the external matrix loaded into the buffer is 64×64, with each element occupying 4 bits, so the loading step size can be determined as 32 bytes (i.e., 64×4 bits); For Figure 6C, the size of the column-major external matrix loaded into the buffer is 64×128, with each element occupying 4 bits, so the loading step size can be determined as 64 bytes (i.e., 128×4 bits); For Figure 6D, the size of the column-major external matrix loaded into the buffer is 64×256, with each element occupying 4 bits, so the loading step size can be determined as 128 bytes (i.e., 256×4 bits); For Figures 6E and 6F, the size of the external matrix loaded into the buffer is 128×128, with each element occupying 8 bits, so the loading step size can be determined as 128 bytes (i.e., 128×8 bits).By configuring the loading step size, the data loading circuit 230 can load the data of the external matrix without needing to consider the data type and size of the elements of the external matrix to be loaded. It only needs to load the data of the external matrix according to the configured loading step size. In other words, for each row or column in the matrix, only the corresponding number of data blocks (e.g., bytes) need to be loaded according to the configured loading step size. This simplifies the data loading operation of the data loading circuit 230. Furthermore, by way of example, when loading the external matrix into the buffer, an interleaving mode can be configured to reduce potential memory conflicts and improve data interaction efficiency. The interleaving mode can be configured according to the various embodiments described above.

[0061] In some embodiments, the control logic circuit 221 may read the second matrix from the data storage circuit 210 by the following steps: obtaining a read step size, wherein the read step size matches the size of the second matrix and the data type of the integer elements; and reading the integer elements of the second matrix from the data storage circuit according to the read step size. For example, assuming the size of the second matrix is ​​16×16, the read step size can be configured to 8 bytes (i.e., 16×4 bits) for 4-bit integer elements and 16 bytes (i.e., 16×8 bits) for 8-bit integer elements. Furthermore, similar to the aforementioned load step size, the read step size can also be measured in units or methods other than bytes.

[0062] For example, when starting matrix operations, an instruction set can be configured, which may include configuring a specified data format, such as configuring it as one of the aforementioned data formats such as "fp16_int4", "fp16_int8", "int4_fp16", "int8_fp16", "bf16_int4", "bf16_int8", "int4_bf16", and "int8_bf16". It may also include configuring storage information, such as the read step size and starting address of the second matrix, so that, according to the configured storage information, the elements of the second matrix can be read from the register array or buffer described in the various embodiments above.

[0063] Alternatively, the integrated circuit 200 may also include a non-matrix operation circuit, which can be used to perform operations other than matrix operations, such as quantizing the elements in the result matrix obtained by the matrix operation circuit 220 and converting them into elements in other data formats, etc.

[0064] Figures 7A and 7B schematically illustrate exemplary block diagrams of processors according to some embodiments of the present disclosure.

[0065] Referring to Figure 7A, embodiments of this disclosure also propose a processor 700, which may include the integrated circuit 200 described according to the various embodiments above. The processor 700 may be various types of processors, such as GPUs (Graphics Processing Units), GPGPUs (General-Purpose computing on Graphics Processing Units), NPUs (Neural-network Processing Units), CPUs (Central Processing Units), etc. Furthermore, the processor 700 may have the same or similar embodiments and advantages as the integrated circuit 200 described according to the various embodiments above.

[0066] For example, FIG7B schematically illustrates an example architecture of a processor according to some embodiments of the present disclosure. This example architecture may include an integrated circuit 710 for data processing and a global memory 720 for processing. The external matrix data mentioned in the foregoing embodiments may be stored in the global memory 720. The integrated circuit 710 may include a matrix operation circuit 711, which may further include control logic circuitry 7111 and arithmetic logic circuitry 7112. The matrix operation circuitry 711 may read matrix data and perform matrix operations according to the various embodiments described above. The integrated circuit 710 may also include a buffer 712 and a register array 714. As described in the various embodiments described above, external matrix data containing a second matrix may be loaded from the global memory 720 and stored in the buffer 712 or the register array 714, and this loading process may be performed, for example, by a data loading circuit 713. For example, the register array 714 can be divided into four parts 714-1, 714-2, 714-3, and 714-4, which can correspond to different thread groups 715-1, 715-2, 715-3, and 715-4, respectively, where each thread group can contain, for example, 32 threads. For example, the matrix data can be stored in a layout similar to that shown in Figures 5A-5D on the register array corresponding to each thread group. It should be understood that the architecture shown in Figure 7B is merely exemplary; in practice, different architectures can be designed according to application requirements.

[0067] In addition, each thread group can optionally have a corresponding non-matrix operation circuit. Taking thread group 715-1 as an example, it can have an integer pipeline 7151, a floating-point pipeline 7152, and conversion logic 7153, etc. The integer pipeline 7151 can handle operations related to integer data, the floating-point pipeline 7152 can perform operations related to floating-point data, and the conversion logic 7153 can perform conversion operations between integer and floating-point types. In related technologies, the operation of pre-decompressing an integer element matrix into a floating-point element matrix can be implemented by this circuit structure of integer pipeline 7151, floating-point pipeline 7152, and conversion logic 7153. Obviously, this decompression operation involves additional data interaction, and the decompressed data will occupy additional storage space. Furthermore, when the operation speed of the aforementioned non-matrix operation circuit is lower than that of the matrix operation circuit 711, the decompression process cannot match the matrix operations performed by the matrix operation circuit 711, thus adversely affecting the overall processing efficiency. However, by using the technical solutions proposed in the embodiments of this disclosure, the decompression operation can be built into the matrix operation circuit 711, which can effectively reduce the amount of data interaction and data storage, and can give full play to the computing power of the matrix operation circuit and improve the overall processing efficiency.

[0068] Figure 8 schematically illustrates an exemplary block diagram of a computing device according to some embodiments of the present disclosure.

[0069] Referring to Figure 8, embodiments of this disclosure also propose a computing device 800, which may include a processor 700 (or integrated circuit 200) as described in the various embodiments above. This computing device 800 may have the same or similar embodiments and advantages as the processor 700 or integrated circuit 200 described in the various embodiments above, which will not be elaborated further here.

[0070] Without violating logic, different embodiments of this disclosure can be combined with each other. The descriptions of different embodiments have different focuses, and the parts not described in a particular focus can be referred to the descriptions of other embodiments.

[0071] By studying the accompanying drawings, the disclosure, and the appended claims, those skilled in the art will be able to understand and implement variations of the disclosed embodiments in practicing the claimed subject matter. In this disclosure, the word "comprising" does not exclude other elements or steps, and "a" or "an" does not exclude a plurality. The mere fact that certain measures are described in mutually different implementations does not imply that combinations of these measures cannot be used for profit.

Claims

1. An integrated circuit for data processing, comprising: The data storage circuit is configured to store matrix data; as well as, Matrix operation circuit, including: The control logic circuit is configured to: read a first matrix and a second matrix from the data storage circuit based on a specified data format, wherein the specified data format indicates that the elements in the first matrix are floating-point data and the elements in the second matrix are integer data; and, The arithmetic logic circuit is configured to: convert elements in the first matrix and the second matrix into target floating-point elements according to the data type of the elements indicated by the specified data format, and determine a result matrix corresponding to the first matrix and the second matrix based on the converted elements of the first matrix and the second matrix.

2. The integrated circuit according to claim 1, wherein, The operational logic circuit includes: At least one conversion circuit, wherein each conversion circuit is configured to: receive elements from the first matrix or the second matrix and element type data indicating the data type of the received elements; and convert the received elements into elements of the target floating-point type based on the element type data, wherein the element type data is determined based on the specified data format; and, An arithmetic circuit, coupled to the conversion circuit, is configured to perform at least one of multiplication and addition operations based on the elements of the converted first matrix and second matrix to determine the result matrix.

3. The integrated circuit according to claim 2, wherein, Each conversion circuit is configured as follows: In response to the element type data indicating that the received element is integer data, the target floating-point element corresponding to the element is determined based on a pre-stored mapping table or by calling a preset conversion logic.

4. The integrated circuit according to claim 2 or 3, wherein, The conversion circuit is configured as follows: In response to the element type data indicating that the received element is 4-bit integer data, the element corresponding to the target floating-point data is determined based on the mapping table from 4-bit integer data to target floating-point data; In response to the element type data indication that the received element is 8-bit integer data, the sign value, exponent, and mantissa corresponding to the element are determined according to the data structure of 8-bit integer data, and the target floating-point element corresponding to the element is determined based on the determined sign value, exponent, and mantissa.

5. The integrated circuit according to any one of claims 1 to 4, wherein, The integer elements in the second matrix are closely packed.

6. The integrated circuit according to claim 5, wherein, The data storage circuit includes a register array, on which the integer elements of the second matrix are stored in a compact arrangement according to the matrix structure. The control logic circuit is configured to read closely packed integer elements of the second matrix from the register array.

7. The integrated circuit according to claim 5, wherein, The data storage circuit includes a buffer in which integer elements of the second matrix are stored in an interleaved pattern in a compact arrangement. The interleaving pattern matches the data type of the integer elements and the size of the matrix stored in the buffer. Furthermore, in the interleaving pattern, row-major elements of the second matrix in the same column are stored in different memory locations within the buffer, or column-major elements of the second matrix in the same row are stored in different memory locations within the buffer. The control logic circuit is configured to read closely packed integer elements of the second matrix from multiple storage bodies of the cache according to the interleaving mode.

8. The integrated circuit according to any one of claims 1 to 7, further comprising: A data loading circuit is configured to load an external matrix, including the second matrix, into the data storage circuit according to a loading step size, wherein the loading step size matches the size of the external matrix to be loaded and the data type of the integer elements.

9. The integrated circuit according to any one of claims 1 to 8, wherein, The control logic circuit is configured as follows: Obtain the read step size, wherein the read step size is matched with the size of the second matrix and the data type of the integer elements; The integer elements of the second matrix are read from the data storage circuit according to the read step size.

10. The integrated circuit according to any one of claims 1 to 9, wherein, The target floating-point type is a non-standard floating-point data type used by the hardware circuit when performing floating-point operations.

11. The integrated circuit according to any one of claims 2 to 10, wherein, The arithmetic logic circuit includes multiple sub-circuits, wherein each sub-circuit corresponds to an element in the result matrix and includes: The conversion module is configured to: receive elements from the first matrix and the second matrix that correspond to the elements in the result matrix, and convert the received elements into the target floating-point elements based on the data type of the received elements; The arithmetic module, coupled to the conversion module, is configured to: perform at least one operation of multiplication and addition based on the elements of the converted first matrix and second matrix to obtain the target floating-point element corresponding to the element in the result matrix; The regularization module, coupled to the operation module, is configured to convert the obtained target floating-point element into a desired floating-point element, wherein the desired floating-point type is a standard floating-point data type different from the target floating-point type.

12. The integrated circuit according to any one of claims 2 to 11, wherein, The conversion circuit includes: A floating-point branch circuit is configured to: in response to the element type data indicating that the received element is a floating-point element, determine the sign value, exponent, and mantissa corresponding to the element according to the floating-point type data structure indicated by the element type data, and determine the target floating-point element corresponding to the element based on the determined sign value, exponent, and mantissa; The integer branch circuit is configured to: in response to the element type data indicating that the received element is an integer element, determine the target floating-point element corresponding to the element based on a pre-stored mapping table or by calling a preset conversion logic.

13. A processor comprising an integrated circuit according to any one of claims 1-12.

14. A computing device comprising the processor of claim 13.

Images